Apple greatly reduced autonomous vehicle testing in 2019, fleet of 23 cars drove only 7,50...

Posted:
in General Discussion edited February 2020
Apple's autonomous vehicle testing program saw a significant drawdown in 2019, with its fleet driving 72,201 miles less than it did in 2018. The company also appears to have changed its disengagement reporting protocol, information commonly used to gauge system efficacy.

Apple Car
Apple's self-driving car testbed in California.


According to numbers submitted by Apple to California's Department of Motor Vehicles, 23 of the company's 69 registered self-driving testbeds were active between December 2018 and November 2019. Collectively, the fleet of modified Lexus RX450h SUVs logged 7,544 miles during the reporting period, down from 79,754 miles in 2018.

Apple reported 64 disengagements across the 7,544 miles driven, which equates to 8.48 disengagements per 1,000 miles. This compares to a whopping 69,510 disengagements, or 871.65 disengagements per 1,000 miles, recorded in 2018. At the time, Apple's system was reported, on a disengagements-by-mile basis, as the worst-performing test platform in California. By contrast, then-front runner Waymo managed 0.09 disengagements per 1,000 miles over the same evaluation period.

While the improvement seems impressive, it should be noted that disengagement reporting is not a hard science. Typically, disengagements are defined as instances in which a self-driving vehicle's autonomous systems are unable to process current conditions, forcing it to pass control back to the human driver. Human interventions also count as disengagements.

As explained by Apple in a letter to California's DMV last year, the company in 2018 took a "conservative" approach to disengagements that required "drivers to proactively take manual control of the vehicle any time the system encounters a scenario beyond our currently proven abilities." Further, the company said its "software self-monitors and returns control back to the driver when it encounters any errors or issues during operation."

The letter implies Apple's built-in threshold for failure is, or at least was, much less tolerant than competing systems, a thesis was borne out in total disengagement figures recorded during the period. From December 2017 through November 2018, the 47 other firms testing automated vehicles on California roads clocked a cumulative 3,040 disengagements over nearly 2 million logged miles.

That said, Apple appears to have narrowed its reporting methodology in 2019.

Of the 64 disengagements, 52 were initiated by the autonomous vehicle system, while human pilots overrode AV decisions 12 times. The most common reason for disengagement was a vague "controls discrepancy" issue.

According to numbers compiled by CNET, Baidu led the pack in 2019 with 0.06 disengagements per 1,000 miles, followed by Waymo and Cruise, both of which managed 0.08 disengagements per 1,000 miles. Toyota reported the highest rate of disengagements with 1,620 per 1,000 miles, though the Japanese carmaker fielded six cars that traveled 1,817 miles.

Apple's self-driving car initiative began under the "Project Titan" banner, with the program initially tasked with creating a branded car from whole cloth. Apple scaled back operations in late 2016 after hitting a number of snags and the Titan team has since refocused attention to autonomous vehicle subsystems. Whether an "Apple Car" will one day make a public debut is unknown.

Comments

  • Reply 1 of 18
    At the time, Apple's system was the worst-performing test platform in California.”. 

    Based on what evidence? Human intervention counts as a disengagement. It’s entirely possible that Apple’s testers are directed to intervene more often out of an abundance of caution. I doubt they want to be all over the news for being involved in a collision, unlike some of the other folks. 
    StrangeDays
  • Reply 2 of 18
    Yeah it's hard to compare disengagements across brands, but you can certainly get insight from inside each project to see how quickly they're resolving perceived issues.

    It's also difficult to read into reduced road time, it could mean anything, such as time spent in R&D, time spent testing new software on private tracks before moving into public spaces.

    Since Apple now have roads in continental USA 3D mapped as part of their Maps overhaul, it seems probable that this data can also be used in some virtual testing scenarios for the car project.
    lolliver
  • Reply 3 of 18
    avon b7avon b7 Posts: 5,901member
    mr lizard said:
    “At the time, Apple's system was the worst-performing test platform in California.”. 

    Based on what evidence? Human intervention counts as a disengagement. It’s entirely possible that Apple’s testers are directed to intervene more often out of an abundance of caution. I doubt they want to be all over the news for being involved in a collision, unlike some of the other folks. 
    Most of what you have mentioned, was covered in the article.

    The evidence is there but the caveats and changes since then are also explained in the article.

    Until two different cars are put to exactly the same road conditions at the same time, it will be difficult to truly compare how autonomous they really are and how they perform in other areas. 
    muthuk_vanalingam
  • Reply 4 of 18
    I don't see how a Post-Jobs Apple could hope to compete against Musk.  

    The success of both was largely / mostly due to an overwhelming and obsessive drive to accomplish a lofty goal.   Musk has it.   Steve did too.
    flyingdpBeatschemengin1
  • Reply 5 of 18
    No one should read much of anything into this article. While the article mentions three variables  — disengagements, miles driven, and company — there’s likely hundreds of variables that would go into understanding the status of self-driving vehicle systems. 

    Those in AI suggest general such systems are more than ten years out. Until then, we will need to be content with bots vacuuming our carpets, delivering pizzas to college dormitories on research campuses. 
  • Reply 6 of 18
    I don't see how a Post-Jobs Apple could hope to compete against Musk.  

    The success of both was largely / mostly due to an overwhelming and obsessive drive to accomplish a lofty goal.   Musk has it.   Steve did too.
    Tesla was definitely the trendsetter with probably a 5 year lead over other automakers in technology and the only company with anywhere near an adequate battery supply for current demand.  I'm not sure where Apple fits in, but other automakers are still struggling with just building the software and hardware around a basic human driven electric car.  Apple isn't going to be a first mover in electric cars, but it is possible they could gain an edge in self-driving.  This certainly feels riskier than other successes where Apple had a longer head start.  If the plan is to license to automakers it is probably more likely to be successful.  Apple is not normally one to license, but not licensing just because it didn't work for Macs is somewhat shortsighted.  I don't have any confidence traditional non-vertically integrated automakers will produce a decent self driving car without help from a company like Apple.  I think the main reason this project still exists is because of the parity with machine learning and augmented reality.  Even if the project is a complete failure, it probably contributed to those other efforts.
    edited February 2020 GeorgeBMacStrangeDays
  • Reply 7 of 18
    mjtomlinmjtomlin Posts: 2,446member
    mr lizard said:
    “At the time, Apple's system was the worst-performing test platform in California.”. 

    Based on what evidence? Human intervention counts as a disengagement. It’s entirely possible that Apple’s testers are directed to intervene more often out of an abundance of caution. I doubt they want to be all over the news for being involved in a collision, unlike some of the other folks. 
    It was mentioned in the article that Apple was much more conservative when reporting numbers.

    The main thing to take into consideration is that it seems each company is allowed to determine what counts as a "disengagement". So trying to compare one company's numbers with another's, is pointless. And as someone else pointed out, no one knows the differences in driving conditions during those logged miles.

    Having such a sharp drop in miles driven makes me think that larger number was just random driving to collect as much data as they could. And then last year they drove and tested (gathered data) for very specific situations.
    muthuk_vanalingamlolliver
  • Reply 8 of 18
    The issue here is not the disengagement metrics.   It is the fact that Apple has severely curtailed testing.   It is another red flag to me that there is no go-to-market plan or timeline for Titan despite the huge investment in people and money
    darkvader
  • Reply 9 of 18
    I don't see how a Post-Jobs Apple could hope to compete against Musk.  

    The success of both was largely / mostly due to an overwhelming and obsessive drive to accomplish a lofty goal.   Musk has it.   Steve did too.
    Tim Cook has it too just in other areas.  Whenever you hear him speak, he always seems bullish with respect to Apple's health & AR efforts.
    Beatslolliver
  • Reply 10 of 18
    BeatsBeats Posts: 2,531member
    I don't see how a Post-Jobs Apple could hope to compete against Musk.  

    The success of both was largely / mostly due to an overwhelming and obsessive drive to accomplish a lofty goal.   Musk has it.   Steve did too.

    Wasn't Apple gonna go out of business when Jobs died? Why is Apple bigger now and why did some of the most successful products in Apple history release post-Jobs?

    Are we gonna hang onto the "riding on Jobs momentum" theory forever? It's gonna be 10 years for crying out loud!
    jony0StrangeDayslolliverchasm
  • Reply 11 of 18
    "the Titan team has since refocused attention to autonomous vehicle subsystems."  

    Please correct this as it is unsubstantiated rumor and speculation.  Apple has never, nor has anyone confirmed this guess/speculation about what Apple is up to with Project Titan.  Indeed, this particular guess is generally thought of as one the least likely avenues Apple would have any reason to go down regarding Project Titan.
    StrangeDayslolliver
  • Reply 12 of 18

    I don't see how a Post-Jobs Apple could hope to compete against Musk.  

    The success of both was largely / mostly due to an overwhelming and obsessive drive to accomplish a lofty goal.   Musk has it.   Steve did too.
    Your claim than one man was responsible for success is at odds with reality. It’s also at odds with what that man said himself — Jobs had said at events before that credit for their success is not his alone, that it takes an entire team, teams of people, to realize these goals. By your logic Apple was DOOMed after Jobs’ death, but we have seen conclusively that this was not the case. 
    lolliver
  • Reply 13 of 18
    mpantonempantone Posts: 1,609member
    red oak said:
    The issue here is not the disengagement metrics.   It is the fact that Apple has severely curtailed testing.   It is another red flag to me that there is no go-to-market plan or timeline for Titan despite the huge investment in people and money
    This is correct.

    Moreover, Apple did zero autonomous vehicle testing between December 2018 and May 2019. None of their cars logged a single mile during this six-month timeframe which points to a possible suspension of the entire program.

    And when they did resume in June 2019, only a fraction of their registered vehicles resumed very light testing.
    gatorguydarkvader
  • Reply 14 of 18
    I presume Apple still has in place their agreement with the “Chinese Uber”, Didi? And I understood they were collecting driver information there which would be shared with Apple.
  • Reply 15 of 18
    cgWerkscgWerks Posts: 2,720member
    I wish they'd have gone the other direction and tried their hand at making an EV. I'm glad they seem to be scaling back, but this autonomous vehicle stuff is a waste/distraction from doing something useful.
  • Reply 16 of 18
    chasmchasm Posts: 2,391member
    There's a far simpler possibility that the article only mentions at the very end (which seemingly none of the previous commenters got to) as to why miles driven in test cars has dropped so dramatically: Apple's not building a car.

    I concur with the last lines of the article that Apple has focused on subsystems and is interested mostly in improving safety and accuracy in a system users will be comfortable with because its based on something they already know -- how iOS devices work -- that can be licensed to manufacturers as the hugely successful CarPlay is today.
  • Reply 17 of 18
    darkvaderdarkvader Posts: 594member
    Yeah, Apple isn't going to build a car.  Apple isn't going to build a system to license either.

    They're just too far behind at this point.  

    I'm not sure why they're still testing at all, maybe it's for something to do with another project.  But it's pretty clear that they're barely testing, and when they do their system just isn't very good.
  • Reply 18 of 18
    cgWerkscgWerks Posts: 2,720member
    darkvader said:
    ... They're just too far behind at this point.  
    ... and when they do their system just isn't very good.
    What is there to be behind, though? None of the systems are very good.
Sign In or Register to comment.