Apple's self-driving vehicles disengage for safety more frequently by design

Posted:
in General Discussion edited February 2019
The California Department of Motor Vehicles has released a letter it received from Apple concerning the "Project Titan" self-driving vehicle project's disengagement rates, with the letter advising how Apple is collecting data on incidents where a driver takes over for the automated driving system, and why the number is so high.

Apple's self-driving car testbed spotted in California.
Apple's self-driving car testbed spotted in California.


Data compiled on Tuesday revealed Apple had what could be the worst-performing self-driving system undergoing testing in California, with its fleet of heavily modified Lexus RX450h SUVs recording an average of 871.65 disengagements for every 1,000 miles travelled, lagging far behind competing systems tested by Google's Waymo, GM Cruise, and Zoox.

The letter from Apple, published to the DMV website on Wednesday, was originally sent by Apple's Apple's senior director of Autonomous Systems Engineering Jamie Waydo on December 21, 2018. The letter explains it is the first disengagement report from Apple since receiving its autonomous vehicle testing permit, and covers a period between April 14, 2017 until November 30, 2018.

"Apple's approach to disengagements is conservative, because our system is not yet designed to operate in all conditions and situations," writes Waydo. "To support this approach, our public road testing policies require drivers to proactively take manual control of the vehicle any time the system encounters a scenario beyond our currently proven abilities."

"In addition, our software self-monitors and returns control back to the driver when it encounters any errors or issues during operation" the letter continues, advising this allows Apple to "prioritize the safe operation of our vehicles above all else at all times."

For recording the disengagements, Apple puts such events into two classes: Manual Takeovers and Software Disengagements.

Manual Takeovers are times when the drivers take control from the self-driving system, adhering to "conservative operational constraints" adhered to in testing. Apple's examples include instances when there is a sign of an approaching emergency vehicle, construction zones, and unexpected objects in and around the roadway, with drivers and operators also instructed to assume control "any time they deem necessary."

Software Disengagements are when an issue is detected with inputs, outputs, or the decision-making ability of the driving system itself. It is further broken down into four subcategories, with Controls covering the actuator systems, Motion Planning being a failure to competently plan a path, Perception failing to sufficiently localize, detect, track, or classify objects in the environment, and Communications being issues where the driving system fails to send messages between processes in a timely manner or fails completely.

Starting from July, Apple started to categorize in more detail, introducing a new listing for Important Disengagements. According to the company, these are flagged for situations "that might have resulted in a safety-related event or a violation of the rules of the road."

The letter mentions an attached but unpublished report that describes the methodologies of identifying the disengagements, before and after the change in reporting. It is noted that the period until June 2018 is "over-inclusive when disclosing disengagements," whereas afterward only Important Disengagements are disclosed.

Between April 2017 and June 2018, Apple's test vehicles travelled 24,604 miles autonomously, experiencing 40,198 Manual Takeovers and 36,359 Software Disengagements during that time. For the post-July period, the cars are said to have travelled 56,135 miles autonomously, and to have experienced just 28 Important Disengagements, with two highlighted to be incidents where other motorists struck Apple's vehicles.

The figures provided in the letter certainly explain how Apple could be considered the worst-performing self-driving vehicle system on numbers alone, but taking the later period on its own, Apple's performance improves to a more credible 2,004 miles on average between Important Disengagements.

"Protecting the safety of all road users will always remain our primary focus," signs off Waydo. "As we continue to improve our capabilities and tools, we expect our future reporting will focus on Important Disengagements."
«1

Comments

  • Reply 1 of 24
    That supports my theory (in the other thread) that Apple's approach to reporting disengagements is radically conservative, leading to an overreporting of disengagements by a factor of 100 or more.  Apple reported 69,510 disengagements; the next highest number of reports was 1,194 by Mercedes Benz; and the third most was 526.  This reflects a different approach to testing and reporting more than it does the safety or maturity of their systems.   https://thelastdriverlicenseholder.com/2019/02/12/disengagement-reports-2018-preliminary-results/

    Kirke
    SpamSandwichSolilongpathlollivertmayjbdragonjony0cgWerksradarthekatStrangeDays
  • Reply 2 of 24
    One would hope that ALL companies testing autonomous vehicles on public roads would be this conservative.  

    The metric to watch going forward is the rate of decline in disengagements, as this will inform progression of the systems.
    lolliverMisterKitnetmagemuthuk_vanalingamradarthekat
  • Reply 3 of 24
    hentaiboyhentaiboy Posts: 1,252member
    ""Apple's approach to disengagements is conservative, because our system is not yet designed to operate in all conditions and situations," writes Waydo."

    Yet apparently Google's is. Perhaps Waydo should go and work for Waymo 
    :D 
    edited February 2019
  • Reply 4 of 24
    AppleExposedAppleExposed Posts: 1,805unconfirmed, member
    Basically another Apple doom story debunked.
    lolliverStrangeDays
  • Reply 5 of 24
    I would like to see the disengagement report of humans driving every day. I think people would be surprise how often people become distracted and are no longer engaged in driving. I see people taking their hands off the wheel and looking down or somewhere else in the car all the time.
    ravnorodomclaire1radarthekatStrangeDayspatchythepirate
  • Reply 6 of 24
    DAalsethDAalseth Posts: 2,783member
    Between April 2017 and June 2018, Apple's test vehicles travelled 24,604 miles autonomously, experiencing 40,198 Manual Takeovers and 36,359 Software Disengagements during that time. For the post-July period, the cars are said to have traveled 56,135 miles autonomously, and to have experienced just 28 Important Disengagements

    April 17-June 18: 24,604 miles with 76, 557 disengagements
    After July 18: 56,135 miles with just 28 disengagements.

    Sounds to me this includes an early experimental testing period when yes the SW would be buggy as heck.
    The latter figure with over twice as many miles and less than thirty issues is more endemic of a maturing system.
    edited February 2019 tmayslprescottclaire1tobianradarthekat
  • Reply 7 of 24
    That supports my theory (in the other thread) that Apple's approach to reporting disengagements is radically conservative, leading to an overreporting of disengagements by a factor of 100 or more.  Apple reported 69,510 disengagements; the next highest number of reports was 1,194 by Mercedes Benz; and the third most was 526.  This reflects a different approach to testing and reporting more than it does the safety or maturity of their systems.   https://thelastdriverlicenseholder.com/2019/02/12/disengagement-reports-2018-preliminary-results/

    Kirke

    To your point, it looks like Apple is reporting every error they are getting, I wonder if other companies see the same error and attempt to recover and it they do they keep moving forward.

    I worked with a guy who worked on intelligent systems back in the 80"s for the federal government (could not say what they did) these were things which were analyzing data and making some decisions about the data, nothing like today AI. But he said these systems had 4 processing system, and before a decision was made all 4 systems had to agree otherwise it would not excuse the next step. If one system did not agree with the other 3 that system was taken off line. If two disagree then it would not take action and the system would shut down or go into stand by until it was understood why all the systems did not agree, they all look at the data differently.

    I wonder if the brainyacts working on these self driving cars are doing something similar or are they relying on one system doing the analysis and decision process of what to do next based on the inputs. Won't you want something change the work of the system making the decision what to do next.
    claire1
  • Reply 8 of 24
    maestro64 said:
    I would like to see the disengagement report of humans driving every day. I think people would be surprise how often people become distracted and are no longer engaged in driving. I see people taking their hands off the wheel and looking down or somewhere else in the car all the time.
    I suspect the "disengagement rate" for humans is 98%+

    The physical act of driving itself is mostly muscle memory - it's the cognitive part that is hard.  We just do it all the time and think we are in control when we text, email & phone while driving.  Police forces, and other first responders, who have to clean up the mess of our total driving inadequacy call us Lemmings for our idiocy.

    Good for you Apple - security, privacy, customer focus - keep those AI drivers throttled back until they even begin to approach the capabilities of the everyday driver.

    Dead serious - no satire here tonight.
    jbdragonclaire1muthuk_vanalingambeowulfschmidt
  • Reply 9 of 24
    I’m sure a wealth of valuable knowledge is going to come from all this autonomous driving research, but I really don’t see it having a practical application unless our concept of transportation was restarted from scratch. As fascinating as the topic is, I still can’t get past a trying to pound a round peg into a square hole scenario.


    muthuk_vanalingam
  • Reply 10 of 24
    MplsPMplsP Posts: 3,911member

    "Apple's approach to disengagements is conservative, because our system is not yet designed to operate in all conditions and situations,"

    Perfect - exactly what one would hope a company would do. Far better this way than to do an ‘Uber.’
    bonobobclaire1cgWerks
  • Reply 11 of 24
    MplsPMplsP Posts: 3,911member
    Basically another Apple doom story debunked.
    The other story wasn’t a doom and gloom story; it simply reported the known facts. People interpreted them in various ways. 
    muthuk_vanalingam
  • Reply 12 of 24
    lkrupplkrupp Posts: 10,557member
    So the original article implied, no, reported the utter failure of Apple’s Project Titan. Now the backpedalling and explanations have started.
    randominternetperson
  • Reply 13 of 24
    jbdragonjbdragon Posts: 2,305member
    hentaiboy said:
    ""Apple's approach to disengagements is conservative, because our system is not yet designed to operate in all conditions and situations," writes Waydo."

    Yet apparently Google's is. Perhaps Waydo should go and work for Waymo 
    :D 
    Really? I don't see any Google cars running around in snow for example. So not all conditions. Waymo has already killed someone. You know what happens when the driver just sits there and does nothing? Stops paying attention as it gets boring fast when you're just sitting there and pulls out their phone and start playing around with that.

    I don't see Apple in a rush at all. They really haven't said anything about this program unlike the others out there. If Apple could keep it more under the radar they would.
    edited February 2019 claire1cgWerks
  • Reply 14 of 24
    DAalseth said:
    Between April 2017 and June 2018, Apple's test vehicles travelled 24,604 miles autonomously, experiencing 40,198 Manual Takeovers and 36,359 Software Disengagements during that time. For the post-July period, the cars are said to have traveled 56,135 miles autonomously, and to have experienced just 28 Important Disengagements

    April 17-June 18: 24,604 miles with 76, 557 disengagements
    After July 18: 56,135 miles with just 28 disengagements.

    Sounds to me this includes an early experimental testing period when yes the SW would be buggy as heck.
    The latter figure with over twice as many miles and less than thirty issues is more endemic of a maturing system.
    If you read again what you quoted you will see that, from the data given, we can't come to the conclusion you came to. 
    The latter figure only proves what they said, that they changed the types of disengagements that they reported. 

    I also assume they are improving but there's no proof of that from the data above.
    claire1randominternetpersonnetmage
  • Reply 15 of 24
    gatorguygatorguy Posts: 24,176member
    jbdragon said:
    hentaiboy said:
    ""Apple's approach to disengagements is conservative, because our system is not yet designed to operate in all conditions and situations," writes Waydo."

    Yet apparently Google's is. Perhaps Waydo should go and work for Waymo  :D 
    Really? I don't see any Google cars running around in snow for example. So not all conditions. Waymo has already killed someone. You know what happens when the driver just sits there and does nothing? Stops paying attention as it gets boring fast when you're just sitting there and pulls out their phone and start playing around with that.

    I don't see Apple in a rush at all. They really haven't said anything about this program unlike the others out there. If Apple could keep it more under the radar they would.
    When did Waymo kill someone? Never. You're either terribly confused or simply making stuff up, one or the other. 

    And Waymo is testing in snowy Michigan.
    Geesh and someone actually gave you a like for misleading them.  
    edited February 2019 muthuk_vanalingam
  • Reply 16 of 24
    MisterKit said:
    I’m sure a wealth of valuable knowledge is going to come from all this autonomous driving research, but I really don’t see it having a practical application unless our concept of transportation was restarted from scratch. As fascinating as the topic is, I still can’t get past a trying to pound a round peg into a square hole scenario.
    Uh, what about a shift to driverless rideshares and cargo trucks isn't a change in concept of transportation? Jeez.
    StrangeDays
  • Reply 17 of 24
    k2kwk2kw Posts: 2,075member
    lkrupp said:
    So the original article implied, no, reported the utter failure of Apple’s Project Titan. Now the backpedalling and explanations have started.
    Only the Apple Board of Directors know how much money Apple threw into the Titan hole.  

    Considering Waymo received Apprval for a driverless Car in California and drove 1.2 million miles on California roads last year I would say that Apple is far behind.  I figure two or three years behind.   57000 miles is nothing.  That’s my commute for 3.5 years.   That said Waymo’s rate of one disengagement per 11,000 miles really isn’t good enough. I wouldn’t feel safe unless they can go on average atleast 60,000miles between Disengagements.  Hopefully they get there soon.  Waymo has ordered tens of thousands of Chrysler Pacificas hybrids  and Jaguar F-Pace EVs. 

    of course if Apple can come up with a much smaller, cheaper Lidar system Apple could have a hit.  If they can find a customer for it.
  • Reply 18 of 24
    cgWerkscgWerks Posts: 2,952member
    gatorguy said:
    jbdragon said:
    hentaiboy said:
    ""Apple's approach to disengagements is conservative, because our system is not yet designed to operate in all conditions and situations," writes Waydo."

    Yet apparently Google's is. Perhaps Waydo should go and work for Waymo  :D 
    Really? I don't see any Google cars running around in snow for example. So not all conditions. Waymo has already killed someone. You know what happens when the driver just sits there and does nothing? Stops paying attention as it gets boring fast when you're just sitting there and pulls out their phone and start playing around with that.

    I don't see Apple in a rush at all. They really haven't said anything about this program unlike the others out there. If Apple could keep it more under the radar they would.
    When did Waymo kill someone? Never. You're either terribly confused or simply making stuff up, one or the other. 

    And Waymo is testing in snowy Michigan.
    Geesh and someone actually gave you a like for misleading them.  
    Well, certainly not all conditions. But, maybe they are confusing Waymo with Uber?
  • Reply 19 of 24
    MplsPMplsP Posts: 3,911member
    lkrupp said:
    So the original article implied, no, reported the utter failure of Apple’s Project Titan. Now the backpedalling and explanations have started.
    Did you even read the other article? All it did was report the disengagement rate as reported to California and compared it to the reported rates of other manufacturers. You seem to be adding your own editorial spin. (again)
  • Reply 20 of 24
    k2kw said:
    lkrupp said:
    So the original article implied, no, reported the utter failure of Apple’s Project Titan. Now the backpedalling and explanations have started.
    Only the Apple Board of Directors know how much money Apple threw into the Titan hole.  

    Considering Waymo received Apprval for a driverless Car in California and drove 1.2 million miles on California roads last year I would say that Apple is far behind.  I figure two or three years behind.  
    Oh nos! “Apple is behind!” Where have we heard that one before? Oh, almost every single time. It’s like you guys have no memory of previous Apple launches and the chorus of identical comments. 

    Not something you want to rush. 
    fastasleep
Sign In or Register to comment.