Developers of self-driving cars are ramping up criticism of a California reporting requirement on test data, saying the data could mislead, as the state prepares to release the latest results for 2019.
Companies such as General Motors’ Cruise and startup Aurora have said the metric, called disengagements, is not an accurate or relevant way to measure their technical progress, even though it is widely used to do just that.
The debate is taking on more importance amid delays in the rollout of self-driving vehicles and concerns over a lack of regulation and the prospects for profitability for the companies that make such vehicles.
The focus on disengagements — when a human driver must take manual control from a self-driving system — and the backlash from self-driving companies have been growing since the California Department of Motor Vehicles began releasing annual disengagement reports five years ago.
California requires all companies testing self-driving vehicles on public roads to submit an annual report on disengagements and what caused them, “written in plain language.”
In 2018, the companies with the most miles between disengagements were Google affiliate Waymo and Cruise. Companies with the greatest number of disengagements were Apple Inc. and Uber Technologies Inc.
In a February 2019 blog post, Waymo wrote that “the key to self-driving technology safely improving and scaling is through a robust breadth of experience and scenario testing, represented by a wider array of data points beyond disengagement alone.”
In the same post, Waymo noted its disengagement rate in 2018 dropped 50 percent from the previous year, while miles between disengagements nearly doubled.
Self-driving companies say the disengagement data can draw unfair comparisons between companies and their self-driving technology.
Aurora co-founder Chris Urmson, who previously headed Waymo’s self-driving program, wrote last month that “these numbers mean little when there’s no clear definition of what constitutes a disengagement.”
Cruise co-founder Kyle Vogt expressed similar concerns in a blog post last month, adding:
“The general public and regulators deserve hard, empirical evidence that an (autonomous vehicle) has performance that is super-human” — that is, better than the average human driver — if the deployment of that technology is to have “a positive overall impact on automotive safety and public health.”