FEATURED COVERAGE: Testing Autonomous Vehicles to Save Lives in the Wake of Recent Accidents

author: Peter Els   

In what could be described as the darkest hour in the very young life of the autonomous vehicle (AV), two people have lost their lives in less than a week in accidents involving vehicles operating in self-driving mode.

On March 18th, in Arizona a pedestrian lost her life after being struck by an Uber owned Volvo XC90 operating in self-driving mode, while on March 23rd a Tesla Model X with its Autopilot engaged plowed into a concrete barrier on the 101 Freeway in Mountain View, California, killing the driver.

In the Model X accident, the vehicle logs indicate the driver did not have his hands on the steering wheel and was not responding to warnings to retake control.

In the Uber tragedy, video footage clearly shows that the driver was not focused on the road leading up to the accident.

Image Source: Autoblog

It would be expedient to point to the drivers' lack of engagement with the vehicles in both these accidents (Which was also highlighted by the National Transportation Safety Board as the overriding cause of the 2016 accident in which a Tesla Model S driver lost his life), but the industry can ill-afford to ignore questions consumers are raising around self-driving vehicle safety: To do so could destroy much-needed consumer confidence.

When will society accept self-driving technology as safe?

Mobility researchers at the University of Michigan believe that for consumers to accept driverless vehicles, tests will need to prove with 80 percent confidence that they're 90 percent safer than human drivers.

To get to that confidence level, test vehicles would need to be driven for 17.7 billion kilometers (~11 billion miles). But it would take nearly a decade of round-the-clock testing to reach just 3.2 million kilometers (~2 billion miles) in typical urban conditions.

According to Susan M. Paddock, senior statistician at RAND and co-author of the 2016 Rand Corporation study into self-driving transportation, although the data gathered so far is important, it does not come close to the level of driving that is needed to calculate safety rates—even if autonomous vehicle fleets accumulated 16 million kilometers, it still would not be possible to draw statistical conclusions about safety and reliability.

For example, assuming a self-driving vehicle fleet had a 20 percent lower fatality rate than that of human drivers. Substantiating this with 95 percent confidence would necessitate driving 8 billion kilometers. This would be equivalent to driving every road in Texas nearly 16,000 times over. It would take a fleet of 100 vehicles, driving 24/7, around 225 years to deliver these results!

Arguably the best-in-class automated driving company, Waymo, just passed the four million mile mark in November 2017 while testing in 20 different U.S. cities, adding a further 4 billion simulated kilometers using virtual environments that reflect real-world conditions.

These statistics make it clear that road-testing alone will never deliver the required results in a reasonable timeframe.

To address the challenges, motor manufacturers have embarked on a two-pronged approach to the problem:

    — Intensive virtual testing to evaluate various scenarios

    — Simulating traffic and road conditions, as well as predicting vehicle response to system changes       before updating test vehicles 

    — On-road driving to log miles under real driving conditions and verify the results obtained through virtual testing

Safely racking up miles in the virtual world

While most self-driving car development teams acknowledge that actual road testing is crucial, virtual simulation is equally important.

Speaking about the efficacy of road testing Mark Hakim, managing director at Qosmotec Software Solutions commented: "Field tests leave a high degree of uncertainty. You need to construct and reproduce a wide variety of traffic situations to ensure they are actually tested".

Even if all cars were driving autonomously and communicating with each other, problems could still occur. Intended to evaluate the impact of these disruptions, Qosmotec's virtual system allows engineers to detect potential problems and emulate real-life traffic scenarios in the lab.

Once a problem has been identified the ‘fix' would usually first be tested in a virtual environment using thousands of permutations before integrating the update into the road-test vehicles. This virtual simulation allows engineers to assess the new software in a realistic computer-generated environment, testing how tweaks to their behavior might play out in reality.

To create an even more realistic virtual setting, NVIDIA recently introduced NVIDIA DRIVE Constellation, a cloud-based system that uses a computing platform built on two different servers to perform photorealistic simulations.

The first server runs NVIDIA DRIVE Sim software to simulate a self-driving vehicle's sensors, such as cameras, lidar and radar. The second contains a powerful NVIDIA DRIVE Pegasus AI car computer that runs the complete autonomous vehicle software stack and processes the simulated data as if it were coming from the sensors of a car driving on the road.

The DRIVE Sim software generates photoreal data streams to create a massive array of different testing environments. It can simulate different weather conditions such as rainstorms and snowstorms; blinding glare at different times of the day, or limited vision at night; and different types of road surfaces and terrain.

Dangerous situations can also be scripted in simulation to test the autonomous car's ability to react, without ever endangering road users.

However, driving culture and norms vary considerably around the world. Therefore self-driving companies need to collect a lot more infrastructure and digital data about other global regions before the technology can be rolled out with confidence.

To solve this problem agent-based modeling such as the Ricardo Agent Drive simulation software can be used to create real-world driving scenarios to test complex driving situations for autonomous vehicles. These scenarios revolve around the agents and are therefore independent of specific geographical locations.

This form of testing places agents (vehicles, people or infrastructure) with specific driving features such as selfish, aggressive or defensive in a defined environment consisting of cities, freeways or test tracks, to understand the complex interactions that occur in real-world situations.

Employing this approach OEMs and their suppliers are able to cut product development cycles, reduce costs related to test-vehicle damage and lower the risk of harming road users during testing.

Notwithstanding the importance of these virtual simulation tests, the response of the imperfect human can never be fully replicated. By nature, humans have varying and inconsistent reactions to situations, and therefore reliable testing has to include the human driver.

The human in-the-loop simulator delivers the best of both worlds

Active safety systems need to perform their role in a user-friendly manner and continue to function smoothly despite a panicking driver‘s instinctive reactions to unfolding events during an emergency maneuver.

And because traffic safety has been a primary component of the promotion of automated driving, the unpredictable human intervention in an autonomous system could seriously compromise this objective.

One step away from actually road testing a vehicle in the real world, a human-in-the-loop (HIL) simulator lets the developer bridge the gap between subjective targets and objective measures during the early development phases by measuring the driver's reaction to random environmental inputs.

One such driving simulator at the University of Warwick, in Britain, has been specifically designed to test "intelligent" vehicles. It can thus interact with the sensors of an autonomous car and put such a vehicle through its virtual paces without ever seeing a road.

The car to be tested sits in the middle of the simulator, which projects a 360° high-definition image of the vehicle's virtual surroundings, constructed from digital maps of 48 km of roads in and around the nearby city of Coventry, together with infrastructure and scenery.

The simulator can depict virtual traffic, cyclists, pedestrians and even dogs crossing the road, all of which programmers can control. It also features surround-sound and actuators that move the vehicle as it would when accelerating, braking or cornering. Even the thump of a virtual pothole can be created.

Highly visual camera-based systems typically use a form of artificial intelligence, known as machine vision, to analyze the shapes of objects. But in extreme conditions, such as when cameras succumb to a condition known as "washout", typically caused by the glare of bright ambient light which occurs at sunrise and sunset, the system may incorrectly identify road hazards.

Unlike the real world, hundreds of sunrises and sunsets can be created in the HIL simulator every day, thereby reducing the development time of antiglare systems.

Other visible hazards that might be hard for self-driving cars to manage, such as streets crowded with pedestrians, cars jumping red lights and joggers suddenly running into the road, can also be created at will in the simulator - without endangering anyone.

Once the simulations show that the vehicle is capable of responding appropriately to the most challenging virtual situations it's time to confirm the results in the real world.

Putting it all together

In the interests of public safety any testing carried out on public roads is highly regulated.

Local governments first need to give self-driving pilot programs permission to operate, which is why, for example, Uber decided to test its self-driving taxis in Arizona rather than its hometown of San Francisco following a very public regulatory dispute with the California DMV.

Now, Bloomberg Philanthropies and the Aspen Institute have released the Global Atlas of Autonomous Vehicles in Cities that shows where all self-driving car development projects are being tested around the world.

The guide is specifically focused on urban areas where local governments are playing a significant role in the roll out of the technology. The primary objective of the Atlas is to organize information about the various projects to help cities learn from other efforts currently underway.

The Atlas profiles 35 "Piloting" cities, which are either hosting or have announced plans to host AV test programs on public roads. Each point on the map lists more specific details about the city's priorities, partners, and exactly what its pilot program is focused on developing.  

Image Source: Bloomberg Philanthropies

A shorter list of 18 locales are tabbed as "Preparing" cities, which haven't committed to any specific pilot programs but have publicly begun to examine the impact AVs might have.

Of concern however, is the fact that 30 cities have described multiple obstacles to expanding their programs even further, which can be ascribed to a lack of understanding or, in some cases, a lack of funding of the growing technologies.

So while AVs will ultimately make our world a safer place it's important that the technology is thoroughly tested before mass deployment, and even then it would be prudent to guard against thinking that the roll out will happen without glitches.

In this event it might be useful to view these mishaps as minor ‘detours' that have low probabilities but significant impacts. Unlike outright failures, these corrections have relatively transient effects—they are setbacks at worst, or even small windfall opportunities.

These detours alter plans rather than destroying them. They are the everyday nuisances that often delay ambitious projects beyond their original schedules. They may be tragic, as accidental death always is, but we know they will occur, if not their exact form and timing.

Peter Els

Peter Els has been involved in the automotive industry since 1979 when he joined Nissan South Africa’s product development team as a student engineer. On qualifying, he explored all facets of the industry holding, positions in engineering, after-sales service, and marketing. Peter has held positions with OEM’s such as Daimler, Fiat, Toyota, Nissan and Beijing Automotive Works. He’s also been responsible for OE sales and exports at Robert Bosch South Africa. After years in an industry driven by information gathering and communication Peter began writing technical articles on various aspects of the industry and the cars it produces. Since then he has produced a broad spectrum of automotive analysis work.