Toyota's New ‘Intelligent Assistant' Learns Voice Commands and Gets Smarter Over Time Using Machine Learning

author: Eric Walz   

Once a driver asks for help, Toyota's Intelligent Assistant listens and responds to natural language conversation.

As vehicles come packed with more technology and features, using some of it on the road can become a distraction. So Toyota developed a voice-activated assistant to lend a hand as part of the "Toyota Connected" vehicle services. 

Toyota's new in-car "Intelligent Assistant" is a highly advanced voice activated assistant that can learn a driver's commands and get smarter over time. It's available for both the Toyota Audio Multimedia and Lexus Interface, including the 2022 like the Toyota Tundra and Lexus NX. 

Toyota offers Connected Services trials for each vehicle, which includes the services Drive Connect, which offers Intelligent Assistant, Cloud Navigation and Destination Assist.

Toyota says that the rollout of these intelligent connected services will transform it from an automotive company to a mobility company.

Toyota's Intelligent Assist can be activated with the phrase "Hey,Toyota" and supplements the onboard Voice Assistant that's embedded in Toyota vehicles, according to Ryan Oehler, product owner at Toyota Connected, an independent software and innovation company, who leads the team responsible for Toyota's new Intelligent Assistant.

Once a driver asks for help, the Intelligent Assistant listens and responds to natural language conversation. For example, if a driver says "I would like a coffee", the system will bring up a list of nearby coffee shops on the vehicle's infotainment screen, then ask the driver if they want directions to one of them for a midday break.

Suppose a driver is looking for a nearby Starbuck and the assistant brings up multiple listings, a driver can then say "take me to the third one" and the navigation system will launch the turn-by-turn directions.

To make navigation easier, the map can be brought up on the infotainment screen at any time by saying, "Hey, Toyota, show the map." Drivers can also ask the assistant to zoom in on the map for easier reading.

Each request is fulfilled using a complex web of commands, and behind the scenes processing on the vehicle itself and the cloud to complete it.

Oehler said there has been a steady ramp of the Intelligent Assistant to meet both mainstream American customers as well as tech-savvy "power users" with intuitive features for drivers. He also said that even more advanced features are in the pipeline.

Not only does Intelligent Assist listen to a driver's voice using automatic speech recognition (ASR), it also looks at waveforms in the audio. When the driver pushes the talk button in the vehicle, it transcribes that speech to text both in the vehicle's embedded computer and over the cloud.

By analyzing the audio waveforms, "the system is able to understand, phonetically, what that translates to and formulate transcriptions based on that audio input," Oehler said.

The technology is even able to recognize commands for different accents, dialects and even different pitches, to recognize what to do next. Since Intelligent Assistant is available in the U.S., Canada and Mexico,  it's designed to recognize commands in English, Spanish and French.

Once a command is transcribed into text, its processed by Toyota Connected machine learning models in the cloud. The machine learning models can determine the intent of the words being used, from audio commands to windshield wipers or even finding a five-star-rated restaurant nearby. 

From there, it's fed back to the vehicle where notes are compared between the embedded and cloud voice assistants to ensure the car is most accurately executing the command the user actually wants.

"This is where the Toyota Connected magic comes in," Oehler said.

The system first determines "top-line intent", which includes common vehicle controls such as temperature, audio and navigation. The next step is to determine "sub-intent", such as a specific temperature, a song title or frequently visited cafe. 

If a drivers requests a certain song while already listening to their favorite streaming platform such as Spotify, the head unit will attempt to find the song on that specific platform. If it's not found, the head unit will switch to the next best option, such as satellite radio. Drivers can also ask Intelligent Assist to play specific genres of music by saying "Hey Toyota, play rock music." 

The system gets smarter by looking at accuracy and tweaking commands to add more ways to ask for specific functions, according to Toyota. Drivers can also ask the system to send a text message to a stored contact just by saying the person's name then using their voice to send a message. 

"Certain audiences expect to interact in completely different ways," Oehler said. "Younger audiences that are more familiar with their modern voice assistants tend to operate in a more fluid manner, whereas people that are less familiar would say, ‘Navigation' and then ‘Texas' in a series of steps," for instance. To help the user, we've added more contextual things, such as prompts, to help guide them through this new user experience. One of the other important things that we're learning is what our customers want to use on a daily basis."

Intelligent Assistant is already available in the Lexus NX, LX, and Toyota Tundra pickup for the 2022 model year, but will be introduced in more Toyota models for 2023 in North America. As it rolls out to more models, the speed at which the machine learning system gets smarter will only accelerate, according to Toyota.

Among the new features planned is asking the Intelligent Assistant to tell a joke. The sense of humor was added to the Virtual Assistant's repertoire after the team of engineers observed customers asking for it.

"There's a lot of opportunity," Oehler said. "There's value with a voice assistant because it adds depth to the experience, is smart, and can and likely will strengthen consumer confidence as more people use it and evolve in future iterations."

Using Intelligent Assistant is also optional on Toyota vehicles. For drivers that would rather not use it, the vehicle's embedded Voice Assistant can be used instead for more basic voice functions, such as adjusting the interior temperature.

Eric Walz

Originally hailing from New Jersey, Eric is a automotive & technology reporter covering the high-tech industry here in Silicon Valley. He has over 15 years of automotive experience and a bachelors degree in computer science. These skills, combined with technical writing and news reporting, allows him to fully understand and identify new and innovative technologies in the auto industry and beyond. He has worked at Uber on self-driving cars and as a technical writer, helping people to understand and work with technology.