2023 Autonomous Vehicle Report Interview: Insights from NVIDIA about Artifical Intelligence in AVs
Danny Shapiro, the Vice President of Automotive at NVIDIA discusses the challenges, breakthroughs, and vision that are propelling autonomous vehicles into the next era.
29 Nov, 2023. 14 min read
Danny Shapiro, the Vice President of Automotive at NVIDIA, generously sat down with Samir Jaber to discuss how NVIDIA see the future of autonomy.
In your position as a VP at NVIDIA, what do you think about the current state of the autonomous vehicle industry?
Danny Shapiro: This is an exceptionally dynamic era within the transportation sector, marked by the pervasive influence of artificial intelligence (AI) and the emergence of the industrial metaverse. What we have termed the "Ominiverse" represents our comprehensive solution in this transformative landscape. From the earliest conceptualization and stylistic considerations to the various stages of design, engineering, manufacturing, autonomous vehicle (AV) development, and even marketing and sales within the retail domain, every facet of the automotive industry is undergoing profound metamorphosis.
The forthcoming wave of generative AI promises to assume a pivotal role in each of these specialized domains. Moreover, the advent of digital twins is proving instrumental in enhancing operations across the board, facilitating their seamless integration. In this context, the Ominiverse provides a collaborative platform for designers and professionals worldwide, transcending geographical constraints. Virtual environments enable disparate experts to collectively contribute to the vehicle's composition. Design reviews unfold in a virtual realm, allowing real-time interaction and immediate visualization of alterations.
Yet, the integration extends beyond design to encompass engineers in distinct silos and the production facility, fostering a comprehensive approach. Facilitating this integration is the Universal Scene Descriptor (USD), a novel standard that harmonizes the collaboration of various departments. A design modification automatically reflects in the engineering sphere and synchronizes with the production floor.
Consequently, factory planners can meticulously construct a physics simulation of the manufacturing facility before its physical realization. An array of cutting-edge technologies, coupled with AI, plays an instrumental role across this spectrum. Generative AI optimizes factory layouts, while AI, which has been under development for an extended period, is now experiencing an inflection point, rendering its accessibility to a broader audience. For example, ChatGPT exemplifies how virtually anyone can assume the role of a programmer, possessing an AI copilot to enhance their professional endeavors.
The impact of AI and related technologies is felt throughout the entire product lifecycle. Customers can engage in virtual test drives, configure and personalize vehicles through immersive VR experiences, and make informed decisions prior to purchase. Furthermore, the realm of maintenance and repair is witnessing substantial enhancements, particularly through predictive maintenance practices and the utilization of augmented reality for training purposes.
Autonomous vehicle development, while a prominent aspect, represents only one facet of our extensive involvement within the automotive industry. The confluence of AI, virtualization, and cutting-edge technologies is reshaping the landscape and driving innovation across all fronts.
Can you provide a little bit of internal details of how you guys have been approaching the development of AV technology in the last few years?
Danny Shapiro: We are a comprehensive, full-stack company, engaged in diverse aspects of automotive technology. Our endeavors encompass the development of proprietary processors, the creation of an entire in-car platform, the design and implementation of full-stack software, and the orchestration of an operating system. Within this framework, we proudly offer Drive OS, a solution trusted by numerous automakers, truck manufacturers, and robo-taxi enterprises. Additionally, our portfolio features DriveWork software, a middleware solution housing a myriad of algorithms and deep neural networks, meticulously tailored for vehicular applications.
The software components encompass a multitude of deep neural networks (DNNs) optimized for distinct purposes within the vehicle. These DNNs are designed to perform intricate functions such as pedestrian detection, lane detection, and sign recognition. Furthermore, they are meticulously customized to align with the specific sensory input from LiDAR, RADAR, and camera systems. Notably, our approach extends to a comprehensive "free space" DNN, dedicated to discerning the absence of objects, thereby identifying the open road for safe navigation. The amalgamation of these neural networks and algorithms offers redundancy and diversity, ensuring paramount safety, which stands as our foremost priority.
Our distinctive approach involves the collaborative development of automotive systems with partners. We do not undertake the vehicle manufacturing process, but we are committed to creating the essential "brain" and the software stack. This comprehensive approach spans the entire spectrum, from the uppermost software layers to the fundamental hardware. Our clientele varies in terms of in-house technology capabilities and staffing, permitting them the flexibility to either adopt our complete stack, sensor suite, and computer system or selectively integrate specific components, aligning with their internal resources.
Consequently, a multitude of customers opt to incorporate our hardware "brain" while selectively implementing portions of our software stack, allowing them to craft their proprietary applications. Our partnerships are diverse and encompass a wide array of automotive manufacturers, including carmakers, truck manufacturers, robo-taxi operators, and shuttle service providers. Each engagement is distinctive, recognizing the inherent complexity of automotive technology. No single entity possesses the capability to address all aspects comprehensively.
Our collaborations, such as the one with Mercedes-Benz, exemplify close cooperation between our engineers and those of our partners. We provide substantial software resources, and the eventual product customization rests with our collaborators, who tailor the solution to meet their unique brand identity, specific use cases, and desired features. Our collaborative network spans globally, extending to esteemed brands like Jaguar, Land Rover, Volvo, Polestar, and numerous enterprises in China. These entities leverage our foundational platform while retaining the ability to fine-tune it to suit their individual requirements.
A pivotal development in the automotive industry is the paradigm shift toward the concept of a software-defined vehicle. Central to this model is a high-performance onboard computer that is amenable to over-the-air updates, facilitating the seamless addition of new features and capabilities throughout the vehicle's operational lifespan.
How is NVIDIA advancing in AI?
Danny Shapiro: We possess a distinctive advantage predicated on several key attributes. While we are deeply immersed in the automotive sector, it is pivotal to note that our core identity remains that of an accelerated computing company. This strategic positioning empowers us to direct substantial resources toward ongoing research and development endeavors in the realm of artificial intelligence, transcending the confines of the automotive domain. The resultant innovations find application across an expansive spectrum of industries, underpinning our unique position in the market.
One illustrative case is our substantial involvement in the healthcare industry, particularly in the domain of cancer detection. Within this context, AI technology plays a pivotal role by aiding radiologists in the analysis of various medical scans, facilitating diagnostic processes, and contributing to disease mitigation. Remarkably, the very technology designed for cancer cell detection can be readily repurposed for pedestrian detection. Though the datasets and training processes differ, the underlying algorithms exhibit striking similarities. Our capacity to leverage insights and methodologies from diverse industries and seamlessly integrate them into the automotive sector represents a distinctive and invaluable capability.
Notably, our commitment extends beyond in-car technologies. An equally significant facet of our strategy encompasses the development of cutting-edge computer systems and AI infrastructure within the data center and cloud environment. What sets us apart is the uniformity of architecture employed across both realms, synchronizing the design and functionality of data center components with their in-car counterparts. This alignment empowers developers with a profound advantage in the AI development process, encompassing the training phase and the real-time inference stage. The seamless integration of these critical components represents a pivotal stride in AI development.
Furthermore, it is imperative to acknowledge the perpetual nature of AI development in the automotive sector. This iterative process entails continuous cycles of training, testing, deployment, data collection, and further refinement. The software within vehicles remains in a state of constant evolution, mirroring the update model familiar to smartphone users. The expectation for modern vehicles is rapidly aligning with this paradigm, with consumers increasingly anticipating ongoing software enhancements and updates. The ability to provide such continuous improvement is swiftly becoming a decisive factor in the market. In essence, the vehicle ownership experience is transitioning toward a model akin to that of contemporary smartphones, wherein software updates and enhancements are integral to user satisfaction and functionality.
Is NVIDIA developing AV safety mechanisms?
Danny Shapiro: Certainly. The intricacies of system development are contingent upon the specific level of autonomy the system aims to achieve. A critical consideration is whether the system is designed for full autonomy or driver assistance, as these determinations govern the requisite fail-operational mechanisms. In driver assistance systems, the presence of a human operator behind the wheel serves as a backup. Conversely, in the context of autonomous vehicles, such as robo-taxis, the absence of a steering wheel and pedals necessitates the implementation of robust fail-operational systems.
These fail-operational systems encompass a core computer, complemented by a backup computer. Although the backup may not replicate the full functionality of the primary unit, it possesses the capability to safely guide the vehicle to the side of the road, initiate a controlled stop, and request assistance in the event of a primary system failure. The implementation of such systems necessitates a fusion of diverse techniques at the chip level, including the incorporation of redundancy and diversity in sensor types, overlapping sensor deployments, and a repertoire of algorithms, including various deep neural networks.
Redundancy extends to the software domain as well. For instance, multiple algorithms may concurrently execute, each performing similar calculations to cross-verify results. This multi-pronged approach is fundamental to the paramount consideration of safety, encompassing all aspects from chip architecture, software components, to sensor boards.
One prevailing challenge that has significantly complicated the timeline for the widespread adoption of self-driving vehicles pertains to the complexity of the problem. Initial estimates and expectations, as far back as 2015, failed to account for the intricacies and unforeseen challenges encountered in this domain. The foremost concern in the pursuit of autonomous vehicles is the assurance of safety, prompting rigorous safety measures and validation procedures.
A pivotal facet of our approach revolves around simulation. Our product, Drive Sim, is instrumental in creating a digital twin of urban environments within the Ominiverse framework. This digital replica encompasses road infrastructure, signage, traffic flow, other vehicles, pedestrians, cyclists, and an array of scenarios. These scenarios may encompass rare, challenging, or potentially dangerous situations that are impractical or unsafe to replicate in the real world. Simulation empowers us to execute millions of miles of virtual testing, including scenarios involving variable weather conditions and lighting effects, such as blinding glare during sunset. This augmented approach supplements real-world testing, significantly enhancing the rigor of our validation processes.
Our simulation techniques encompass both software in the loop and hardware in the loop. Notably, our Constellation product serves as a simulator equipped with an array of GPUs to generate synthetic data, simulating the sensory inputs of cameras, RADAR systems, and LiDAR sensors. This synthetic data is subsequently fed into the actual drive computer situated within the data center. The drive computer processes this data, unaware of its simulation status, effectively believing it is navigating real-world environments. Subsequently, the drive computer renders driving decisions, including acceleration, braking, and steering responses, which are then fed back into the simulator. This hardware-in-the-loop methodology enables comprehensive testing to evaluate the system's responses to a multitude of scenarios, such as the detection of pedestrians, night-time child safety scenarios, and signage recognition. The outcome of these tests serves to identify potential issues requiring software refinement or validation of system functionality.
How is Nvidia managing the substantial volume of data involved in autonomous vehicle (AV) systems?
Danny Shapiro: Indeed, our endeavor entails an unprecedented undertaking, characterized by the creation of vehicles and fleets on a scale heretofore uncharted. While engineering a few vehicles represents a manageable feat, the transition to managing tens of thousands, hundreds of thousands, or even millions of vehicles on the road poses a distinctly formidable challenge. Consequently, we have meticulously constructed our operational framework from the ground up, encompassing comprehensive vehicle design and manufacturing, as well as the development of a robust data center infrastructure. This comprehensive approach is underpinned by our commitment to gaining an intricate understanding of the scale and intricacies of the challenge at hand. Without this firsthand knowledge, we cannot proficiently deliver solutions to our clientele.
It is important to clarify that we are not in competition with tier one suppliers or original equipment manufacturers. Our pursuit of these endeavors on a smaller scale serves as a pivotal mechanism for acquiring invaluable insights into the methodologies, workflows, and infrastructural components required for the realization of our vision. This insight enables us to offer robust and well-informed solutions to our partners and customers.
Our efforts extend to the development of systematic workflows for data collection, archival, curation, and labeling. These steps are integral to the preparation of datasets for training and validation purposes, and artificial intelligence plays a central role in enhancing the efficiency of these workflows. Moreover, we leverage AI in the very development of AI models. As we traverse vast distances, we employ AI algorithms to classify and categorize the data acquired, given that the majority of miles traveled typically involve familiar and well-understood scenarios. Our objective is to discern and document those rare and exceptional circumstances, thus ensuring their retention in our curated dataset.
Furthermore, we harness the capabilities of synthetic data generation to augment our dataset with infrequent or unusual scenes. A technique known as "neural reconstruction" permits us to transform recorded drives into three-dimensional representations of the environments traversed. These 3D scenes serve as the foundation for creating diverse permutations and scenarios. This approach allows for the identification and manipulation of vehicles within the scene, enabling the generation of an array of novel scenarios, all derived from a single drive. Consequently, we can construct numerous distinct scenes, enriching our dataset with valuable synthetic data for AI training and validation purposes.
How are you developing cybersecurity technologies or partnering with companies advancing in this area?
Danny Shapiro: As previously mentioned, our corporate identity extends beyond the conventional automotive realm, as we originate from the data center industry. Our core mission involves the seamless integration of data center technology and high-performance computing into the automotive domain. Consequently, the extensive knowledge and expertise acquired in managing data centers serving critical functions, such as those in banking and healthcare, are harnessed in our automotive systems.
Within this framework, we have implemented a comprehensive suite of cybersecurity measures that have proven pivotal in safeguarding our automotive solutions. These measures encompass encryption, authentication protocols, and virtualization techniques, each of which plays a vital role in fortifying various system components. The assimilation of these data center technologies into our automotive systems affords us the ability to leverage a wealth of expertise and insights, recognizing the paramount significance of cybersecurity in our endeavors.
Furthermore, our focus extends to the implementation of stringent cybersecurity measures at the device level, encompassing devices interconnected via Bluetooth, Wi-Fi, or cellular modems. These security measures aim to establish effective firewalls, preventing unauthorized access or tampering. At the chip level, we have integrated cutting-edge technology, such as secure boot mechanisms and encrypted over-the-air update protocols, to safeguard against unauthorized modifications.
Moreover, our commitment to cybersecurity extends to the application of artificial intelligence. We are actively engaged in pioneering developments at the data center networking level, wherein AI algorithms are instrumental in the continuous monitoring of chip behavior. AI-equipped systems possess the capacity to detect anomalies by discerning deviations from established norms. For instance, in scenarios where the tire pressure monitoring system initiates a software update, AI algorithms swiftly recognize this as abnormal behavior, enabling rapid intervention and mitigation.
In conclusion, cybersecurity remains a central and paramount aspect of our work. While the full extent of our cybersecurity strategies cannot be disclosed in detail, rest assured that it represents a top-tier priority within our operational framework.
How have you been developing hardware?
Danny Shapiro: Certainly, when reviewing our technological roadmap across various temporal horizons, it becomes evident that our unwavering commitment resides in consistently pushing the boundaries of performance. This pursuit of performance excellence transcends domains, encompassing graphics, computing, and the rapidly advancing field of artificial intelligence. Our enduring objective remains the attainment of industry-leading performance metrics. Concurrently, we maintain a vigilant focus on augmenting energy efficiency, achieved through meticulously crafted strategies that encompass the reduction of die size and the implementation of power management techniques to deactivate dormant chip components.
Our dedication to optimizing energy consumption emanates from our extensive background in designing systems for laptops, smartphones, tablets, and mobile devices. In these contexts, the preservation of battery life stands as a pivotal consideration. Notably, each successive generation of our technology showcases noteworthy advancements in the realm of performance per watt, surpassing the capabilities of its predecessor. Furthermore, our commitment to energy efficiency extends to the software domain, where sophisticated algorithms facilitate the systematic hibernation of underutilized system components, thus contributing to energy conservation.
Central to our roadmap is the primacy of performance, a fundamental principle that has consistently proven to be a linchpin of our success. In the context of autonomous vehicles, this enhanced performance directly translates into an augmentation of safety. Notably, the expanded processing capacity enabled by heightened performance empowers AVs to process data from an augmented array of sensors, including those characterized by higher resolutions. Furthermore, this expanded performance capability permits the execution of more intricate deep neural networks, instrumental in the detection and prediction of behaviors. This includes the ability to distinguish between distracted and attentive pedestrians, a nuanced aspect of AV safety. Ultimately, the correlation between heightened performance and increased safety is indisputable, underscoring the pivotal role of superior performance in the evolution of AV technology.
What are the main challenges in engineering electric vehicles?
Danny Shapiro: Across cities worldwide, there is a notable proliferation of deployment and testing endeavors involving autonomous vehicles. The Bay Area, in particular, stands as a vivid testament to this global trend, where a diverse array of autonomous vehicle prototypes and iterations undergo daily trials. Indeed, certain robo-taxi enterprises in San Francisco have procured licenses permitting operational deployment without a human driver within the vehicle, thus offering their services to paying customers.
While these advancements hold great promise, it is imperative to acknowledge that autonomous vehicles have not yet attained mainstream adoption. Our foremost commitment and emphasis remain dedicated to securing the requisite safety approvals that are indispensable for this transformative technology. Our objective is nothing short of ensuring that these autonomous vehicles exhibit the capacity to adeptly navigate and respond to an exhaustive spectrum of potential scenarios. This ambition is rooted in the understanding that autonomous vehicles, once deployed at scale, will demonstrably surpass human-driven counterparts in terms of safety. Nevertheless, the imperative is to meticulously account for every conceivable circumstance and eventuality.
Our ongoing endeavors revolve around the refinement of the technology underpinning autonomous vehicles, an expansive expansion of testing protocols, and an unwavering commitment to broadening the scope of situations that these vehicles can competently address. This mission extends to the formidable challenge of accommodating the often unpredictable behavior of human drivers, with the overarching aim of enhancing safety for all stakeholders sharing the roadways. Our dedication to this pursuit encapsulates a comprehensive commitment to the safety and well-being of all individuals both within and beyond these autonomous vehicles.
What are some key considerations in the AV industry?
Danny Shapiro: Within the realm of the autonomous vehicle (AV) industry, there exists a plethora of indispensable considerations that merit meticulous examination. It becomes evident that the sphere extends far beyond the confines of isolated chip performance or energy efficiency, as these are but constituents of a much broader equation. The fulcrum of this multifaceted domain is the comprehensive software stack that resides within the autonomous vehicle, underpinning its core functionalities. Yet, it is often the intricate interplay of data collection, artificial intelligence (AI) training, and rigorous simulation testing, accompanied by the ongoing evolution of applications and software, which frequently eludes the purview of the casual observer.
This intricate development workflow, while potentially concealed from the awareness of the average consumer, holds pivotal significance for those entrenched within the industry's inner circles. Beyond the software-centric aspects, the transformative scope of the automotive sector extends to myriad dimensions, notably encompassing automotive design. Envision a future where vehicular collisions become obsolete, precipitating a seismic shift in design paradigms that render traditional steel and airbag usage redundant. Every facet of the design, engineering, and manufacturing process stands poised for profound transformation, where artificial intelligence, particularly generative AI, emerges as a vanguard of this renaissance.
Generative AI, exemplified by technologies like ChatGPT, ushers in a new era of artificial intelligence capable of generating an eclectic array of outputs from diverse inputs. It possesses the capacity to transmute textual input into visually captivating imagery, transmute text into video content, and even craft original visual and video compositions drawn from pre-existing materials. While the technology remains in its incipient stages, its potential to revolutionize multifarious domains, ranging from design and engineering to manufacturing and tailored retail experiences, is undeniably palpable.
Consider the possibility of individualized television advertisements, meticulously tailored to each viewer's preferences, featuring a vehicle traversing their own neighborhood streets and parking in their driveway. AI, in synergy with the Ominiverse, holds the transformative power to reshape every facet of the automotive industry. Consequently, the depth and multidimensionality of this subject matter is readily apparent. Should you embark on an exploratory journey in this transformative arena, we remain at your disposal to furnish supplementary content, video references, or to actively participate in follow-up dialogues that delve deeper into this paradigm-shifting subject. Furthermore, we extend an open invitation for fact verification or quotation authentication as deemed necessary.
Read more about NVIDIA on Wevolver.