Teleoperation steps in when an autonomous vehicle does not know what to do

In the not-too-distant future, autonomous, networked vehicles will be able to handle most tasks independently. However, they are not yet capable of dealing with all traffic situations. If they encounter a problem that they do not know how to solve, safety comes first. Most of the time this would mean pulling over to the side of the road and stopping. The German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt; DLR) is conducting research into the teleoperation of these vehicles in order to resolve such situations safely and quickly in the future. In unclear situations, the autonomous vehicle can request teleoperation support from a technical supervisor.

Teleoperation – a technological bridge and a legal requirement

"Teleoperation of autonomous vehicles serves as a bridge to bring existing technologies in the field of automation rapidly into use and onto the roads," says Michael Oehl of the DLR Institute of Transportation Systems in Braunschweig. His team’s work focuses on autonomous driving at SAE Level 4, where the automation takes over all of the driving tasks. The occupants do not have to intervene, even though this may remain possible using control elements such as steering wheels.

Experts describe the different degrees of automation as SAE levels, based on the classification of systems for autonomous driving devised by SAE International (formerly the Society of Automotive Engineers), the organisation for mobility technology. This classification ranges from Level 0 to Level 5. At Level 0, the vehicle has no automation whatsoever. Level 5 equates to full automation, where the vehicle is completely autonomous and must be able to determine the overall driving behaviour under all circumstances. The occupants are simply passengers. Humans no longer have to intervene, not even by means of teleoperation. SAE Level 4 systems are not yet ready for everyday use, but the legal framework for using them does exist in a number of places, including Germany. The 'Autonomous Driving Act' came into force in July 2021 and the rules were specified in greater detail in the 'Autonomous Driving Regulations'. The law allows autonomous driving at SAE Level 4 on special routes or within defined areas such as airports, logistics centres and motorways. This opens up possibilities including transporting people or goods with driverless shuttles, for example. For operation and liability, teleoperation or technical supervision – as defined in the law – plays a decisive role.

Remote Assistance: A Key Enabler for Highly Automated Driving (SAE 4)

Highly automated vehicles (SAE 4) are becoming ever more sophisticated. However, due to the sheer number of potential situations the vehicle may encounter, its automation is not yet capable of dealing seamlessly with all of them. Remote Assistance as a type of remote operation is a way to take care...Credit: © DLR. All rights reserved

Focus on humans and automation working together

"We are at the beginning of development and have many ideas and concepts for making vehicle teleoperation safe and efficient," says Oehl. "For now, we are concentrating primarily on the interaction between humans and automation, and on the design of the interface that this requires – the human-machine interface, or HMI for short. For example, we are looking at the form that a teleoperation workstation might take, what the job requires of people and how it has to be designed from the point of view of work and engineering psychology."

Teleoperation workstation. Credit: © DLR. All rights reserved

The human factor – requirements and workplace design for teleoperation

In future, several teleoperators equipped with monitors, user interfaces and headsets could sit together in a control centre, guiding autonomous vehicles at SAE Level 4 through difficult situations. Imagine a double-parked delivery vehicle, for instance. In such a scenario, a car will not drive over a solid line of its own accord but remain where it is. Similar situations can arise on construction sites that have not yet been incorporated into the system, so are not known to the automation system.

Before they can grapple with these numerous scenarios, however, DLR's researchers still have to address a number of fundamental questions. What is the best way to map the vehicles for monitoring, and what information is needed? How many vehicles can a single person monitor and oversee on screen at the same time? What kind of live images can cameras in highly automated vehicles provide to help the technical supervisors make decisions? How long does it take to process a request for assistance on average? And how long does it take for operators to put themselves in the position of the vehicle and make a decision?

"Vigilance is particularly important in such monitoring tasks. In research, we use this term to describe a state of constant alertness. Even if nothing happens for a long time, you have to be able to react quickly when it does. Teleoperation workstations have to be specially designed in order to prevent a situation where people get distracted, switch off or start experiencing fatigue. Just like train drivers or flight controllers, these operators cannot be constantly overworked, but they must not be underchallenged either," says Oehl. In order to investigate these questions more closely in the context of the teleoperation of vehicles, the team has constructed such a workstation as a prototype. In this way, the researchers are investigating in a very application-oriented way what information teleoperators will require at their workstation in the future in order to be able to support autonomous vehicles with SAE Level 4 quickly and safely in situations that are unclear for the automation system.