A Look at How Waymo's Self-Driving Test Fleet Safely Traveled 2.7 Million Miles in San Francisco Last Year

author: Eric Walz   

A Waymo robotaxi arrives to pick up passenger in San Francisco. (Photo: by Jessica Christian/The San Francisco Chronicle)

Autonomous driving developer Waymo, which spun out of Google Self-driving car project, is widely considered to be one of the world's leading developers of self-driving technology. With the backing of parent company Alphabet Inc over the years, the company has been working on autonomous driving since 2009 as part of Google's self-driving car project. 

Since then, Waymo's fleet of self-driving vehicles traveled over 25 million miles on public roads to perfect its driverless technology. The company aims to launch an app-based robotaxi service soon in U.S. cities that's similar to Uber and Lyft called "Waymo One '', which will use fully autonomous vehicles with no safety drivers onboard. 

One of the first cities that Waymo plans to launch its ride-hailing service is in San Francisco. In preparation for a safe launch of the service, Waymo's fleet has traveled 2.7 million miles on city streets last year, mapping the city and collecting valuable data to improve its autonomous driving technology stack called the "Waymo Driver".

In a new blog post, Waymo shared a more detailed look of how its managed to travel nearly three million miles in one of the most challenging environments for self-driving vehicles to safely operate in. 

San Francisco's hilly terrain and busy streets are crowded with bicyclists and pedestrians pose a challenge for developers of self-driving vehicles. But Waymo's advanced autonomous driving technology is capable enough to handle it, even without a safety driver onboard. 

Waymo continuously evaluates three main components of its Waymo Driver – hardware, software, and operations – against a safety framework the company created and unveiled in Oct 2020. So here's a closer look at them.

The Hardware Platform

The hardware platform includes the integration of the Waymo Driver's sensors and compute platform with the vehicle platform, which in this case are Jaguar I-Pace electric SUVs. Waymo now has hundreds of I-Pace vehicles in its San Francisco fleet. The hardware and sensor suite includes lidar, radars and cameras. From here, Waymo throughly validates how all of these systems work together in tandem in a variety of real world conditions. 

This includes rigorous evaluation of the performance of vehicle sensors and the compute platform in a range of situations the self-driving vehicles can expect to encounter in San Francisco, including weather. The weather performance includes testing the Waymo Driver in fog, powerful gusts of winds and sands blowing off Ocean beach near San Francisco's Pacific Coast shoreline. 

This evaluation process also includes monitoring the vehicle's steering and speed control systems and how they perform while operating on the city's hilly and narrow streets.

The robust compute platform enables the Waymo Driver to react quickly in critical situations, such as approaching emergency vehicles or a driver running a red light at a busy intersection, which can happen often in the real world. 

Waymo also evaluatues the robustness of its autonomous driving system as a whole, such as the ability of the Waymo Driver to come to a safe stop and the readiness of backup systems, as well as redundancies to handle any unexpected situations, such as a single sensor failing.

Waymo uses a range of methods to safely operate its self-driving vehicles in the city, including computer simulations, structured testing on closed courses, then driving on public roads to validate the performance of the Waymo Driver, which is now its fifth generation. 

After conducting simulation and closed course testing, Waymo then drove almost three million miles autonomously in the city to further validate the performance of its Waymo Driver.

Waymo says the millions of driving miles in San Francisco are critical for training its system, ensuring simulation realism, evaluating the Waymo Driver's performance and continuously improving its safety.

Last year, Waymo installed the fifth generation of the Waymo Driver on the Jaguar I-PACE vehicle platform in San Francisco. The vehicles, which collected data, were launched with trained autonomous specialists behind the wheel, who could disengage the vehicle's autonomous mode and take over the driving tasks manually when they felt it was appropriate.

As Waymo accumulated more test miles, the company evaluated the performance of its Waymo Driver in increasingly more challenging driving situations both day and night, in various weather conditions, including dense fog, which San Francisco is famous for. For fog in particular, Waymo updated its fifth generation hardware and perception system to help the Waymo Driver handle it more effectively. 

Waymo looks at these and other weather challenges across its hardware, software and operations then compares it with the Waymo Driver's performance to predetermined targets and metrics. This means that the Waymo Driver can behave differently based on real-time weather conditions, for example, driving more slowly on slippery roads or proceeding more cautiously at intersections in dense fog. 

The Software

Like other developers of self-driving vehicles, Waymo uses a comprehensive set of methodologies to evaluate the performance of its driving software, this includes testing how it completes ride-hailing trips autonomously while adhering to applicable road rules and safely, while avoiding conflicts with other road users.

Dense urban environments like San Francisco present a unique set of challenges for the software that powers the Waymo Driver. These include crowded and complex intersections, narrow streets, as well as social interactions and cues from other drivers, cyclists and pedestrians that human drivers rely on. The latter requires the Waymo Driver to understand other road users' intentions and accurately predict their next moves, according to Waymo.

To address this problem, Waymo created an open dataset that it also shares with others. The dataset is used to train machine learning models to better predict the behavior and intentions of other road uses. For a self-driving vehicle, correctly predicting the behavior of other road users is essential in order to mitigate crashes and operate with a high level of safety.

Last year, Waymo added a "motion dataset", which is one of the largest of its kind ever released for research, the company says. The data includes motion forecasting for autonomous driving.

Waymo ensured that its new dataset includes interesting examples of how its self-driving vehicles interact with other road users, such as cyclists and vehicles sharing the roadway, cars quickly passing through intersections or groups of pedestrians clustering on the sidewalk. 

At 20 seconds long, each segment in the motion dataset is sufficient enough to train machine learning models to capture complex behaviors encountered in urban environments. Waymo said it purposefully tried to make this data as useful as possible for others, in order to help third parties developers to better create their own motion forecasting models. 

Operations

The third part of the Waymo Driver is operations, which includes evaluating the hardware and software and how the two platforms interact with the base vehicle platform. Waymo said that serving passengers requires assessing the performance of its fleet and entire operations, particularly in San Francisco.

Part of the operations area is Waymo's Risk Management Program, which identifies and resolves potential safety issues before any new or updated features or software is used on public roads in vehicles carrying passengers.

Waymo also has a Field Safety Program that works with the Risk Management Program by collecting, assessing, and resolving potential safety concerns that can originate from riders, employees, or the public. The two programs help Waymo proactively address new potential risks associated with operating in a new or changing environment like San Francisco.

Waymo said its made a number of advancements last year ahead of its vehicle deployments in San Francisco. For one, the company fine-tuned its fleet operations to account for the many confusing rules around pick-ups and drop-off locations in the city that Uber and other app-based mobility providers drivers must deal with.

Waymo also refined other features of Waymo One service based on feedback from early riders. An example of one of these is a feature that allows riders to add multiple stops to their trip and provide clearer directions to pick-up points to meet requests from people with accessibility challenges.

After fully validating the performance and safety of its technology stack, Waymo began offering trial rides last year to members of the community by offering hundreds of San Francisco residents a chance to ride in one of Waymo's self-driving vehicles. These early riders provide Waymo with valuable feedback that helps it to continuously refine and advance its technology.  

The response from the public was so great that Waymo created a wait list. Waymo's co-CEO Tekedra Mawakana, said in December that "tens of thousands" of San Francisco residents are currently on the waitlist.

The successful commercial launch of Waymo One requires the trust of the public, which is one of the reasons for Waymo's comprehensive validation process for all of the technology that powers the Waymo One service, and its working. 

A recent survey conducted by Waymo found that 94% of riders in San Francisco are satisfied with the experience and 97% expressed trust in the company's technology.

Waymo said its will gradually extend the program to include even more riders in 2022 as it gets ready for a more widespread launch of the Waymo One robotaxi service in San Francisco, then in other major U.S. cities.

Eric Walz

Originally hailing from New Jersey, Eric is a automotive & technology reporter covering the high-tech industry here in Silicon Valley. He has over 15 years of automotive experience and a bachelors degree in computer science. These skills, combined with technical writing and news reporting, allows him to fully understand and identify new and innovative technologies in the auto industry and beyond. He has worked at Uber on self-driving cars and as a technical writer, helping people to understand and work with technology.