Thinking and Learning
Autonomous cars employ advanced algorithms, machine learning, and artificial intelligence to "think" and "learn." They gather data from various sensors like cameras, radar, and LiDAR, and then process and interpret this data to understand their environment. Machine learning enables these vehicles to improve over time, adapting to new situations and optimizing their responses.
The decision-making process is real-time, with the onboard computer systems controlling navigation and obstacle avoidance. Additionally, network connectivity and cloud computing play a role, allowing the vehicles to access broader data and computational resources for enhanced learning and decision-making. Download the full report below.
Frontiers of AI Learning Approaches for AVs
Multi-modal learning allows AVs to glean insights from a range of data sources, including visual inputs, RADAR data, and LiDAR readings. Over the past three years, the integration of multi-modality 3D object detection, for example, has emerged as a promising strategy to bolster the accuracy and resilience of perception tasks in autonomous driving. These advancements encompass diverse methodologies, such as employing sophisticated cross-modality attention-based feature fusion, crafting more dependable homogeneous representations across distinct modalities, and formulating intricate and resilient unified frameworks. Examples range from 3D detection based on LiDAR data and camera-LiDAR fusion to the prediction of multimodal trajectories within autonomous driving systems.38,39
Multi-Modal Sensor Fusion-Based Deep Neural Network for End-to-End Autonomous Driving
Deep Reinforcement Learning empowers control logic to make optimal real-time decisions. This adaptability makes it particularly well-suited for the ever-shifting and uncertain circumstances inherent in AVs. In the last three years, reinforcement learning has been employed in autonomous driving to optimize controllers, refine path planning and optimize trajectory, enhance motion planning and dynamic path planning, formulate high-level driving policies for intricate navigation challenges, and implement scenario-based policy learning for diverse scenarios. Moreover, it can also be employed for reward learning through inverse reinforcement learning from expert data, aiding in intent prediction for traffic actors like pedestrians and vehicles.39-41
Shift from task-specific to task-agnostic AI represents another frontier of AI learning approaches for AVs in the last three years. Traditional AI systems require training on millions of examples within a specific domain. For instance, an image-recognition system needed extensive data to identify animal species. However, recent developments have led to large foundation models that can be trained on general data using self-supervised learning. These models can grasp general concepts with few examples or prompts, significantly improving their adaptability to new scenarios and improving technology performance and safety.
Innovation in generative AI technology, it's the most advanced, high-fidelity closed-loop simulator to date, crucial for enabling autonomous driving. Generative AI could be efficiently used to create highly accurate digital replicas of the real world from raw sensor data. It can modify these replicas to simulate endless scenarios for training and testing AVs.
This includes adding or removing other vehicles, simulating emergencies, accidents, and more. This technology creates both typical and critical driving situations automatically and on a large scale. This reduces the need for real-world test driving, making autonomous driving development safer and more cost-effective. The combination of generative AI-powered simulation with an AI model tailored for physical interaction promises faster, safer, and more scalable deployment of autonomous technology worldwide.
Autonomous driving systems with traffic and driving simulations empowered by generative AI.
NLP and GANs Reshaping Autonomous Driving
Natural Language Processing (NLP) enables machines to understand and generate human language, facilitating effective communication between humans and machines. The synergy between NLP and AVs has introduced novel dimensions to human-machine interaction and AV safety.
Incorporating NLP into AV software encompasses diverse methodologies. One approach involves adopting a rule-based system, entailing the creation of rules that govern the understanding of natural language commands. On the other hand, an approach involving ML can be employed, entailing a model's training on a dataset comprising natural language commands and their corresponding actions.
The choice of approach is contingent on the specific application. Rule-based systems are well-suited for simpler scenarios like navigating a small delivery robot, whereas ML systems are better equipped to handle more complex tasks, such as orchestrating the actions of an autonomous vehicle.
The incorporation of NLP-driven human-vehicle interaction provides numerous advantages, including:
Enhanced safety by allowing drivers and passengers to engage with the vehicle without diverting their attention from the road.
Diverse linguistic preferences, which enable a broader user base to communicate effectively with the AV.
Increased AV efficiency. Travelers can leverage natural language queries to access details about their environment, which can help to plan trips more effectively.
Personalized touch that offers tailored responses based on individual preferences and contextual understanding.
In general, NLP technology equips vehicles with the capability to process, comprehend, and respond to human language inputs, thereby creating an intuitive interface that fosters seamless interaction. The integration of NLP into AVs operates through intricate mechanisms that facilitate effective communication between passengers and the vehicle's AI system.
For example, Cruise employs NLP for voice commands, enhancing the user experience by understanding complex queries. The company's commitment to interactivity is reflected in the advanced NLP capabilities. Additionally, Cruise utilizes AI in its Continuous Learning Machine, automating data processes to enhance driving system accuracy and safety over time, making the vehicles more adept at handling real-world driving scenarios.
When talking about GANs, their capacity to generate highly authentic images and videos has established them as pivotal networks for cutting-edge AV development. This distinctive capability positions GANs as valuable tools for training AVs to effectively recognize various objects.
In 2020, Uber's Advanced Technologies Group (ATG) has introduced an innovative AI technique aimed at enhancing the prediction accuracy of autonomous vehicles' traffic movements. This method, applicable to Uber's own driverless technologies, utilizes a generative adversarial network named SC-GAN (scene-compliant GAN).
Unlike simpler architectures, SC-GAN incorporates high-definition maps and detection/tracking systems informed by LiDAR, radar, and camera sensors to create trajectories that adhere to scene constraints. This novel approach is expected to significantly improve the precision of predictions, addressing critical issues for autonomous vehicles, such as the ability to detect and anticipate surrounding cars' trajectories.42
“This is the most exciting time in the transportation industry. Every aspect of the auto industry is being transformed by artificial intelligence and by the industrial metaverse.” - Danny Shapiro - VP of Automotive at NVIDIA
Harnessing the Power of LLMs for AV Applications
Following the release of ChatGPT there has been a surge of interest in Large Language Models (LLMs). In the context of AVs, LLMs can be seen as a more specialized version of NLP that can support more general and more interactive applications. Leveraging these characteristics, the use of LLMs is considered for a variety of AV use cases, including:
Integrating language and reasoning capabilities into autonomous vehicles.
Supporting high-level decisions through chain-of-thought.
Implementing generative driver agent simulators that can provide, perceive and analyze complex traffic scenarios towards improving the navigation features of the AV.
Personalizing the driver’s experience based on verbal feedback from LLMs.
Companies Developing AI Algorithms for AV Applications
Leadership Interviews
Interview with Mouser
Interview with Murata
Interview with MacroFab
Interview with Nexperia
Interview with SAE International
Interview with Autoware Foundation
Interview with NVIDIA
References
1. SceneScan Pro User Story: 3D Stereo Vision Cameras for Autonomous Racing Cars. https://nerian.com/news/scenescan-pro-user-story-3d-stereo-vision-cameras-for-autonomous-racing-cars/ (2022).
2. Foresight. 3D Stereo Vision for the Autonomous Vehicle Industry. Foresight https://www.foresightauto.com/3d-stereo-vision-for-the-autonomous-vehicle-industry/ (2023).
3. Bhadoriya, A. S., Vegamoor, V. & Rathinam, S. Vehicle Detection and Tracking Using Thermal Cameras in Adverse Visibility Conditions. Sensors 22, (2022).
4. Computer vision challenges in autonomous vehicles: The future of AI. https://www.superannotate.com/blog/computer-vision-in-autonomous-vehicles.
5. O’Neill, M. A Visionary Leap: Enhancing Computer Vision for Autonomous Vehicles and Cyborgs. SciTechDaily https://scitechdaily.com/a-visionary-leap-enhancing-computer-vision-for-autonomous-vehicles-and-cyborgs/ (2023).
6. AI-Enhanced night vision: A breakthrough for autonomous cars. IO https://innovationorigins.com/en/ai-enhanced-night-vision-a-breakthrough-for-autonomous-cars/ (2023).
7. Cvijetic, N. Panoptic Segmentation Helps Autonomous Vehicles See Outside the Box. NVIDIA Blog https://blogs.nvidia.com/blog/2019/10/23/drive-labs-panoptic-segmentation/ (2019).
8. Bridging to the Autonomous Future with Mobileye SuperVisionTM. Mobileye https://www.mobileye.com/blog/mobileye-supervision-bridge-to-consumer-autonomous-vehicles/.
9. LIDAR Sensor Integration in Autonomous Vehicles Platforms. https://www.mrlcg.com https://www.mrlcg.com/latest-media/lidar-sensor-integration-in-autonomous-vehicles-platforms-300849/ (2023).
10. Innovations of Autonomous Vehicles: Solid-State LiDAR Sensors. EqualOcean https://equalocean.com/analysis/2022031917148.
11. Home. Aeva https://www.aeva.com/ (2023).
12. Rahn, S. Technology. Blickfeld https://www.blickfeld.com/technology/ (2021).
13. Velodyne Lidar lanza un centro de diseño en Bangalore. https://www.businesswire.com/news/home/20210624006043/es/ (2021).
14. Innoviz Technologies. Innoviz Technologies to Present at Upcoming Investor Conferences. The Victoria Advocate https://www.victoriaadvocate.com/innoviz-technologies-to-present-at-upcoming-investor-conferences/article_69c829b2-d474-52b0-825b-7f3591907377.html (2023).
15. Luminar. Luminar (Nasdaq: LAZR). Luminar https://www.luminartech.com/.
16. Home. Aeva https://www.aeva.com/ (2023).
17. Quanergy filing for bankruptcy, going up for sale. Security Info Watch https://www.securityinfowatch.com/perimeter-security/physical-hardening/perimeter-security-sensors/press-release/21290282/quanergy-quanergy-filing-for-bankruptcy-going-up-for-sale (2022).
18. Quanergy Announces Industry-first 3D LiDAR Movement-Based False Alarm Reduction Solution. Quanergy Solutions, Inc. | LiDAR Sensors and Smart Perception Solutions https://quanergy.com/pressrel/quanergy-announces-industry-first-3d-lidar-movement-based-false-alarm-reduction-solution/.
19. [No title]. https://static.mobileye.com/website/corporate/media/radar-lidar-fact-sheet.pdf.
20. Nellis, S. Mobileye looks to build its own lidar to drive down self-driving costs. Reuters (2020).
21. High Resolution 3D Flash LiDARTM. Continental Automotive http://www.continental-automotive.com/en-gl/Passenger-Cars/Autonomous-Mobility/Enablers/Lidars/3D-Flash-Lidar.
22. Continental Aftermarket. http://www.continental-aftermarket.com/us-en/press/press-releases/2021/2021-11-08-continental-releases-hfl110-3d-flash-lidar-to-series-production.
23. Carman, A. Blickfeld Unveils First Smart LiDAR Sensor with Built-in Software Stack. All About Circuits https://www.allaboutcircuits.com/news/blickfeld-unveils-first-smart-lidar-sensor-with-built-in-software-stack/ (2022).
24. Blickfeld. Blickfeld launches Qb2; the world’s first Smart LiDAR sensor, enabling capturing and processing of 3D. Blickfeld https://www.blickfeld.com/blickfeld-launches-qb2/ (2022).
25. Difference between solid state radar and magnetron. https://lidarradar.com/info/difference-between-solid-state-radar-and-magnetron.
26. Hsu, W.-T. & Lin, S.-L. Using FMCW in Autonomous Cars to Accurately Estimate the Distance of the Preceding Vehicle. Int. J. Automot. Technol. Manage. 23, 1755–1762 (2023).
27. Global, B. Bosch Research Blog. Bosch Global https://www.bosch.com/stories/synthetic-aperture-radar/ (2022).
28. Manzoni, M. et al. Automotive SAR imaging: potentials, challenges, and performances. International Journal of Microwave and Wireless Technologies 1–10 (2023).
29. Website. https://www.researchgate.net/publication/334633392_Synthetic_Aperture_Radar_Towards_Automotive_Applications.
30. [No title]. https://arxiv.org/pdf/2306.09784.pdf.
31. Website. https://omdia.tech.informa.com/blogs/2023/sensors-for-safe-autonomous-driving.
32. IEEE Xplore Full-Text PDF: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9528970.
33. Cvijetic, N. How AI Improves Radar Perception for Autonomous Vehicles. NVIDIA Blog https://blogs.nvidia.com/blog/2021/04/28/drive-labs-ai-radar-perception-autonomous-vehicles/ (2021).
34. Autonomous Vehicles. Navtech Radar https://navtechradar.com/explore/autonomous-vehicles/ (2021).
35. The Importance of Imaging Radar. http://www.nxp.com/company/blog/the-importance-of-imaging-radar:BL-THE-IMPORTANCE-OF-IMAGING-RADAR.
36. eVehicle Technology. New 4D Imaging Radar Sensor to Revolutionise Automotive Safety. eVehicle Technology https://www.evehicletechnology.com/news/4d-imaging-radar-sensor-to-revolutionise-automotive-safety/ (2020).
37. Vayyar. Vayyar Becomes First And Only Company In World To Offer Full-cabin Monitoring With One Radar-on-chip. PR Newswire https://www.prnewswire.com/news-releases/vayyar-becomes-first-and-only-company-in-world-to-offer-full-cabin-monitoring-with-one-radar-on-chip-301251217.html (2021).
38. Maanpää, J. et al. Multimodal End-to-End Learning for Autonomous Steering in Adverse Road and Weather Conditions. https://ieeexplore.ieee.org/document/9413109.
39. Multi-modality 3D object detection in autonomous driving: A review. Neurocomputing 553, 126587 (2023).
40. [No title]. https://arxiv.org/pdf/2002.00444.pdf.
41. Ravi Kiran, B. et al. Deep Reinforcement Learning for Autonomous Driving: A Survey. https://ieeexplore.ieee.org/document/9351818.
42. Wiggers, K. Uber claims its AI enables driverless cars to predict traffic movement with high accuracy. VentureBeat https://venturebeat.com/ai/uber-claims-its-ai-enables-driverless-cars-to-predict-traffic-movement-with-high-accuracy/ (2020).
43. Edge AI Computing Advancements Driving Autonomous Vehicle Potential. Global Semiconductor Alliance https://www.gsaglobal.org/forums/edge-ai-computing-advancements-driving-autonomous-vehicle-potential/ (2021).
44. Snapdragon Ride SDK: a premium platform for developing customizable ADAS applications. https://www.qualcomm.com/news/onq/2022/01/snapdragon-ride-sdk-premium-solution-developing-customizable-adas-and-autonomous.
45. Edge AI Computing for Autonomous Driving System - Lanner Electronics. https://www.lannerinc.com/applications/transportation/edge-ai-computing-for-autonomous-driving-system.
46. AI Accelerator Chips Overview and Comparison. HardwareBee https://hardwarebee.com/ai-accelerator-chips-overview-and-comparison/ (2023).
47. Lardinois, F. Google Cloud announces the 5th generation of its custom TPUs. TechCrunch https://techcrunch.com/2023/08/29/google-cloud-announces-the-5th-generation-of-its-custom-tpus/ (2023).
48. Francisco, A. Intel Brings New Chips, Autonomous Driving, and Artificial Intelligence to CES 2020. Showmetech https://www.showmetech.com.br/en/intel-brings-chips-and-news-to-ces-2020/ (2020).
49. Hajela, R. Looking at 11th Generation Intel® Processor Performance on Intel® DevCloud for Edge Workloads. OpenVINO-toolkit https://medium.com/openvino-toolkit/looking-at-11th-generation-intel-processor-performance-on-intel-devcloud-for-the-edge-213a413fb5e1 (2022).
50. Riley, D. Intel makes a splash at CES with AI, autonomous driving tech and Tiger Lake chips. SiliconANGLE https://siliconangle.com/2020/01/06/intel-makes-splash-ces-autonomous-driving-tech-ai-tiger-lake/ (2020).
51. Self-Driving Cars Technology & Solutions from NVIDIA Automotive. NVIDIA https://www.nvidia.com/en-us/self-driving-cars/.
52. Volvo Cars, Zoox, SAIC and More Join Growing Range of Autonomous Vehicle Makers Using New NVIDIA DRIVE Solutions. NVIDIA Newsroom http://nvidianews.nvidia.com/news/volvo-cars-zoox-saic-and-more-join-growing-range-of-autonomous-vehicle-makers-using-new-nvidia-drive-solutions.
53. Korosec, K. Qualcomm unveils its Snapdragon Ride platform for all levels of automated driving. TechCrunch https://techcrunch.com/2020/01/06/qualcomm-unveils-its-snapdragon-ride-platform-for-all-levels-of-automated-driving/ (2020).
54. Website. https://europe.autonews.com/automakers/vw-use-qualcomms-snapdragon-ride-platform-autonomous-driving-push.
55. Automotive RTOS. https://blackberry.qnx.com/en/ultimate-guides/software-defined-vehicle/automotive-rtos.
56. Saxena, A. Understanding AUTOSAR and its Applications in the Automotive Industry. https://www.einfochips.com/blog/autosar-in-automotive-industry/ (2020).
57. Apex.AI. Apex.AI https://www.apex.ai/blog.
58. Cognata. Cognata Autonomous and ADAS Simulation https://www.cognata.com/ (2022).
59. Ford China deploys C-V2X service in Changchun, Jilin Province. https://autonews.gasgoo.com/m/70024138.html.
60. Qualcomm Introduces Car-to-Cloud Service for Over-the-Air Vehicle Updates and On-Demand Services & Features. https://www.qualcomm.com/news/releases/2020/01/qualcomm-introduces-car-cloud-service-over-air-vehicle-updates-and-demand.
61. Vehicle Telematics, Geofencing & Fleet Tracking Systems for the Connected Car. (2006).
62. Cisco 5G Network Architecture. (2020).
63. Infoshare Systems, Inc. How cyber security threats are impacting the automotive industry. https://www.linkedin.com/pulse/how-cyber-security-threats-impacting-automotive-industry-1f (1685456083000).
64. Website. https://www.sciencedirect.com/science/ article/pii/S209580991930503X.
65. [No title]. https://iopscience.iop.org/article/10.1088/1742-6596/1694/1/012024/pdf.
66. ESCRYPT Intrusion detection and prevention solution. https://www.etas.com/en/products/intrusion-detection-and-prevention-solution.php.
67. TRANSFORMING SECURITY FROM A LIMITATION TO A BUSINESS VALUE MULTIPLIER. C2A Security - the Only Mobility Centric DevSecOps Platform https://c2a-sec.com/ (2022).
68. C2A. C2A Security Partners with Valeo to Enhance Cybersecurity for the Software Defined Vehicle. PR Newswire https://www.prnewswire.com/news-releases/c2a-security-partners-with-valeo-to-enhance-cybersecurity-for-the-software-defined-vehicle-301713633.html (2023).
69. Dave, P. Dashcam Footage Shows Driverless Cars Clogging San Francisco. WIRED https://www.wired.com/story/dashcam-footage-shows-driverless-cars-cruise-waymo-clogging-san-francisco/ (2023).
70. Waymo Driver. Waymo https://waymo.com/waymo-driver/.
71. How Autonomous Vehicles Work. https://ltad.com/about/how-autonomous-vehicles-work.html.
72. Merchant, B. Column: Self-driving Waymo cars might worsen L.A. traffic. Los Angeles Times (2023).
73. Waymo Driver. Waymo https://waymo.com/waymo-driver/.
74. Rangaiah, M. How Waymo is using AI for autonomous driving? https://www.analyticssteps.com/blogs/how-waymo-using-ai-autonomous-driving.
75. Website. https://www.uber.com/blog/autonomous-vehicle-safety-guidelines/.
76. Reuters. Uber partners with Alphabet’s Waymo to offer driverless rides. Reuters (2023).
77. Jennewine, T. 6 Ways Tesla’s Autonomous Driving Technology Is Evolving. The Motley Fool https://www.fool.com/investing/2022/01/27/6-ways-teslas-autonomous-driving-technology-is-evo/ (2022).
78. Tesla Vision Update: Replacing Ultrasonic Sensors with Tesla Vision. Tesla https://www.tesla.com/support/transitioning-tesla-vision.
79. Moon, S. How Tesla made autonomous possible without LIDAR. Datahunt - Quality Data with Quality AI https://www.thedatahunt.com/en-insight/how-tesla-autonomous-driving-possible-without-lidar (2023).
80. Autopilot. https://www.tesla.com/autopilot.
81. Fox, E. Tesla 2020.4.11 Software OTA Update To Boost Ranges Of Model S/X. TESMANIAN https://www.tesmanian.com/blogs/tesmanian-blog/tesla-2020-4-11-software-ota-update-to-boost-ranges-of-model-s-x (2020).
82. Cardello, S. What Are Over-the-Air Updates For Cars? MUO https://www.makeuseof.com/what-are-over-the-air-updates-for-cars/ (2022).
83. Hawkins, A. J. Here’s our first full look at the Cadillac Celestiq ultra-luxury electric sedan. The Verge https://www.theverge.com/23273937/gm-cadillac-celestiq-ev-sedan-photos-specs-reveal (2022).
84. [No title]. https://www.volvocars.com/intl/news/technology/Your-Volvo-car-will-understand-you/.
85. Pollock, D. General Motors Applies For Decentralized Blockchain Map Patent. Forbes https://www.forbes.com/sites/darrynpollock/2020/04/03/general-motors-applies-for-decentralized-blockchain-map-patent/ (2020).
86. Schoppa, C. Top Autonomous Vehicles Companies to Watch in 2023. AI Time Journal - Artificial Intelligence, Automation, Work and Business https://aitimejournal.com/autonomous-vehicles-companies-to-watch/ (2022).
87. Ammann, D. The Cruise Origin Story - Cruise - Medium. Cruise https://medium.com/cruise/the-cruise-origin-story-b6e9ad4b47e5 (2020).
88. Vara, V. Leading vehicle manufacturing companies in the autonomous vehicles theme. Just Auto http://www.just-auto.com/data-insights/top-ranked-vehicle-manufacturing-companies-in-autonomous-vehicles/ (2022).
89. Autonomous Vehicle Technology. https://getcruise.com/technology/.
90. Pavlik, T. When Volvo will have its first autonomous vehicle might surprise you. MotorBiscuit https://www.motorbiscuit.com/volvo-first-autonomous-vehicle-might-surprise/ (2022).
91. Dyer, N. Volvo’s Ride Pilot autonomous driving tech allows unsupervised driving. CARHP https://www.carhp.com/news/volvo-s-ride-pilot-autonomous-driving-tech-allows-unsupervised-driving (2023).
92. Rodríguez, J., Jr. Volvo EX90 comes with self-driving hardware, accessible by a subscription. Jalopnik https://jalopnik.com/volvo-ex90-subscription-based-self-driving-tech-1850197959 (2023).
93. Kirby, C. & Flear, K. Volvo’s First OTA Update: More Range for Some Models. https://getjerry.com/insights/volvos-first-ota-update-range-models (2022).
