GPUs excel in parallel processing for graphics and AI training with scalability, while NPUs focus on low-latency AI inference on edge devices, enhancing privacy by processing data locally. Together, they complement each other in addressing different stages of AI workloads efficiently.

NPU vs GPU: Understanding the Key Differences and Use Cases

The deep neural network models that power today’s most demanding machine-learning applications have grown so large and complex that they are pushing the limits of traditional electronic computing hardware.Photonic hardware, which can perform machine-learning computations with light, offers a faster and more energy-efficient alternative. However, there are some types of neural network computations that a photonic device can’t perform, requiring the use of off-chip electronics or other techniques that hamper speed and efficiency.Building on a decade of research, scientists from MIT and elsewhere have developed a new photonic chip that overcomes these roadblocks. They demonstrated a fully integrated photonic processor that can perform all the key computations of a deep neural network optically on the chip.The optical device was able to complete the key computations for a machine-learning classification task in less than half a nanosecond while achieving more than 92 percent accuracy — performance that is on par with traditional hardware.The chip, composed of interconnected modules that form an optical neural network, is fabricated using commercial foundry processes, which could enable the scaling of the technology and its integration into electronics.In the long run, the photonic processor could lead to faster and more energy-efficient deep learning for computationally demanding applications like lidar, scientific research in astronomy and particle physics, or high-speed telecommunications.“There are a lot of cases where how well the model performs isn’t the only thing that matters, but also how fast you can get an answer. Now that we have an end-to-end system that can run a neural network in optics, at a nanosecond time scale, we can start thinking at a higher level about applications and algorithms,” says Saumil Bandyopadhyay ’17, MEng ’18, PhD ’23, a visiting scientist in the Quantum Photonics and AI Group within the Research Laboratory of Electronics (RLE) and a postdoc at NTT Research, Inc., who is the lead author of a paper on the new chip.Bandyopadhyay is joined on the paper by Alexander Sludds ’18, MEng ’19, PhD ’23; Nicholas Harris PhD ’17; Darius Bunandar PhD ’19; Stefan Krastanov, a former RLE research scientist who is now an assistant professor at the University of Massachusetts at Amherst; Ryan Hamerly, a visiting scientist at RLE and senior scientist at NTT Research; Matthew Streshinsky, a former silicon photonics lead at Nokia who is now co-founder and CEO of Enosemi; Michael Hochberg, president of Periplous, LLC; and Dirk Englund, a professor in the Department of Electrical Engineering and Computer Science, principal investigator of the Quantum Photonics and Artificial Intelligence Group and of RLE, and senior author of the paper. The research&nbsp;<a href="https://www.nature.com/articles/s41566-024-01567-z" target="_blank">appears today in&nbsp;Nature Photonics</a>.<h2>Machine learning with light</h2>Deep neural networks are composed of many interconnected layers of nodes, or neurons, that operate on input data to produce an output. One key operation in a deep neural network involves the use of linear algebra to perform matrix multiplication, which transforms data as it is passed from layer to layer.But in addition to these linear operations, deep neural networks perform nonlinear operations that help the model learn more intricate patterns. Nonlinear operations, like activation functions, give deep neural networks the power to solve complex problems.In 2017, Englund’s group, along with researchers in the lab of Marin Soljačić, the Cecil and Ida Green Professor of Physics,&nbsp;<a href="https://news.mit.edu/2017/new-system-allows-optical-deep-learning-0612" target="_blank">demonstrated an optical neural network on a single photonic chip</a> that could perform matrix multiplication with light.But at the time, the device couldn’t perform nonlinear operations on the chip. Optical data had to be converted into electrical signals and sent to a digital processor to perform nonlinear operations.“Nonlinearity in optics is quite challenging because photons don’t interact with each other very easily. That makes it very power consuming to trigger optical nonlinearities, so it becomes challenging to build a system that can do it in a scalable way,” Bandyopadhyay explains.They overcame that challenge by designing devices called nonlinear optical function units (NOFUs), which combine electronics and optics to implement nonlinear operations on the chip.The researchers built an optical deep neural network on a photonic chip using three layers of devices that perform linear and nonlinear operations.<h2>A fully-integrated network</h2>At the outset, their system encodes the parameters of a deep neural network into light. Then, an array of programmable beamsplitters, which was&nbsp;demonstrated&nbsp;in the 2017 paper, performs matrix multiplication on those inputs.The data then pass to programmable NOFUs, which implement nonlinear functions by siphoning off a small amount of light to photodiodes that convert optical signals to electric current. This process, which eliminates the need for an external amplifier, consumes very little energy.“We stay in the optical domain the whole time, until the end when we want to read out the answer. This enables us to achieve ultra-low latency,” Bandyopadhyay says.Achieving such low latency enabled them to efficiently train a deep neural network on the chip, a process known as in situ&nbsp;training that typically consumes a huge amount of energy in digital hardware.“This is especially useful for systems where you are doing in-domain processing of optical signals, like navigation or telecommunications, but also in systems that you want&nbsp;to learn in real time,” he says.The photonic system achieved more than 96 percent accuracy during training tests and more than 92 percent accuracy during inference, which is comparable to traditional hardware. In addition, the chip performs key computations in less than half a nanosecond. &nbsp; &nbsp;&nbsp;“This work demonstrates that computing — at its essence, the mapping of inputs to outputs — can be compiled onto new architectures of linear and nonlinear physics that enable a fundamentally different scaling law of computation versus effort needed,” says Englund.The entire circuit was fabricated using the same infrastructure and foundry processes that produce CMOS computer chips. This could enable the chip to be manufactured at scale, using tried-and-true techniques that introduce very little error into the fabrication process.Scaling up their device and integrating it with real-world electronics like cameras or telecommunications systems will be a major focus of future work, Bandyopadhyay says. In addition, the researchers want to explore algorithms that can leverage the advantages of optics to train systems faster and with better energy efficiency.

Researchers demonstrated a fully integrated photonic processor that can perform all key computations of a deep neural network optically on the chip, which could enable faster and more energy-efficient deep learning for computationally demanding applications like lidar or high-speed telecommunications. Image: Sampson Wilcox, Research Laboratory of Electronics.

This new device uses light to perform the key operations of a deep neural network on a chip, opening the door to high-speed processors that can learn in real-time.

Photonic processor could enable ultrafast AI computations with extreme energy efficiency

<a href="https://seas.harvard.edu/person/jia-liu">Jia Liu</a>, Assistant Professor of Bioengineering;&nbsp;<a href="https://seas.harvard.edu/person/na-li">Na Li</a>, Winokur Family Professor of Electrical Engineering and Applied Mathematics; and&nbsp;<a href="https://seas.harvard.edu/person/petros-koumoutsakos">Petros Koumoutsakos</a>, Herbert S. Winokur, Jr. Professor of Computing in Science and Engineering at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), have been awarded an Emerging Frontiers in Research and Innovation (EFRI) grant from the National Science Foundation. The $2 million grant will support research into developing a bio-symbiotic system that mimics the scalability, adaptability, and efficiency of biological neural networks capable of performing long-term, complex computational tasks.The research is a collaboration between SEAS, the Wyss Institute for Biologically Inspired Engineering at Harvard University, and the University of Texas at Austin.The grant is part of the Emerging Frontiers in Research and Innovation (EFRI): Biocomputing through EnGINeering Organoid Intelligence program, which supports research and development of organoid intelligence systems and biocomputing research.At SEAS, Liu’s group brings together researchers from a range of disciplines to develop advanced bioelectronics that integrate nanoelectronics, soft electronics, genetic and genomic engineering, and machine learning to interface with biological systems. Previously, Liu’s group demonstrated the integration of brain organoids with soft and flexible electronics as cyborg organoids for recording and controlling brain organoid development. With the EFRI grant, Liu and his team will test the cyborg brain organoid’s ability to perform computational tasks, with the goal of inspiring new AI algorithms, enhancing data processing efficiency, and advancing machine learning beyond current capabilities.“Current artificial intelligence and machine learning technologies do not match the adaptability and efficiency of biological systems,” said Liu. “Our goal is to develop a seamlessly integrated biological and electronics system for long-term 3D neural network computing, truly merging nature and machine intelligence together.""NSF's investment will lead to biological computing with superior power and efficiency by harnessing the mechanisms behind complex biological behavior for smart systems," said Susan Margulies, NSF Assistant Director for Engineering. "Advances in biocomputing will open new opportunities for artificial intelligence, biotechnology, and more sustainable computing."

Liu and team receive NSF EFRI grant for biocomputing research

Harnessing scalability, adaptability, and efficiency of biological neural networks through flexible brain-computer interfaces

Imagine being a passenger in a self-driving car as the vehicle starts veering off the road. It’s not a faulty sensor causing the dangerous situation — it’s a cyberattack. Hackers can access the deep learning (DL) neural networks at the heart of the vehicle’s computer system, compromising the safety of its passengers, as well as other drivers and pedestrians.&nbsp;Stopping such cyberattacks requires understanding them first, but this can be challenging. Finding a computing system’s exact deep neural network has many roadblocks. They are often proprietary and, therefore, inaccessible to investigators without considerable legal intervention. Another common problem is that they are updated frequently, making it difficult for investigating researchers to access the most current network iteration. But a new tool from Georgia Tech could unlock the mysterious malware on myriad neural networks in everything from self-driving cars to the IMDB entertainment database. AI Psychiatry (AiP) is a postmortem cybersecurity forensic tool that uses artificial intelligence to recover the exact models a compromised machine runs on and discover where the fatal error occurred.“We trust self-driving cars with our lives and ChatGPT with our careers, but when those systems fail, how are we going to investigate them?” said <a href="https://saltaformaggio.ece.gatech.edu/">Brendan Saltaformaggio</a>, an associate professor with joint appointments in the School of <a href="https://scp.cc.gatech.edu/">&nbsp;Cybersecurity and Privacy</a> and the School of <a href="https://www.ece.gatech.edu/">Electrical and Computer Engineering</a> (ECE).AiP can recover the original DL model on both the local network’s memory and the graphics processing unit that trains the network. It can accomplish this without any specific knowledge of the model’s framework, platform, or version. Instead, it recreates the model using what&nbsp;<a>Saltaformaggio</a> refers to as “clues,” or common components in all neural networks. These include weights, biases, shapes, and layers from the model’s memory image — a frozen set of the bits and bytes operating when the model is running normally. The memory image is crucial because it enables AiP to compare it with the model post-attack.“These models often refine their information as they go, based on their current environment, so an attack might happen as a result of an attacker poisoning the information a particular model is learning,” said David Oygenblik, an ECE Ph.D. student. “We determined that a memory image would capture all those changes that occur during a runtime.”\Once the model is recovered, AiP can run it on another device, letting investigators test it thoroughly to determine where the flaws lie. AiP has been tested with different versions of both popular machine learning frameworks (TensorFlow and PyTorch) and datasets (CIFAR-10, LISA, and IMDB). It successfully recovered and rehosted 30 models with 100% accuracy.&nbsp;“Before our research, you couldn't go to the cyber ‘crime scene’ and find clues because there was no technique available to do that,” Saltaformaggio said. “That's what we are pioneering in the cyber forensics lab right now — techniques to get that evidence out of a crime scene.”Tools like AiP will allow cyber investigators to see the whole picture immediately. Solving cybercrimes can help prevent future ones, from safeguarding a user’s data to keeping a car on the road.&nbsp;

New tool AI Psychiatry recovers compromised deep-learning models so researchers can understand what went wrong.

The Sherlock Holmes of AI

Even if you’re not very familiar with deep learning, you’ve probably heard about it and how it can, among other things, help automate the driving experience, increase manufacturing efficiency and change the consumer shopping experience. Deep learning is the latest evolution of artificial intelligence (AI) and machine learning. Both machine learning and deep learning are subsets of AI. While traditional machine learning algorithms need to be specifically programmed by people with domain-level expertise, deep learning algorithms utilize neural networks that can be trained by feeding them data. The same network can be used to solve very different problems by training it with different data, removing the requirement of having specific expertise in the problem that needs to be solved. Because of this, deep learning is considered a foundational technology that has the potential to make an impact in many industries.For those new to deep learning, think of it as a technology that can classify things. Many real-world problems can be broken down into a classification problem, for example:<ul><li>Autonomous and semi-autonomous vehicles need to be able to classify roads, other vehicles, people and road signs as shown in Figure 1.</li><li>Smart factories need to be able to classify a product as defective</li><li>Automated retail outlets need to classify if a consumer is removing a product from a store for purchase or just looking at it</li></ul><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1722330537960-Screenshot+2024-07-30+110736.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 1. Classification Example of an Automotive Use Case. Every Object in the Scene Is Determined to Be One of the Five Different Classes.There are two main parts to deep learning: training and inference, as shown in Figure 2. During training, a model of a neural network that has been created to solve a classification problem is fed huge amounts of labeled data, from which it learns how to classify things. In the example scenario shown in Figure 1 of an autonomous vehicle, the model would be fed images of pedestrians, road signs, cars and roads with each of these objects correctly labeled. Real-time performance or power is not an issue during training, so it usually occurs on a desktop or cloud platform.During inference, the trained network that has been deployed into its end application makes a classification decision, such as determining whether or not a part on an assembly line is defective.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1722330524914-Screenshot+2024-07-30+110753.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 2. Typical Development Flow of a Deep Learning Network.For a combination of reasons – including reliability, low latency, privacy, power and cost – it is beneficial for inference to run close to where sensors gather data; that is, at the “edge” of the network. Advanced sensors that can handle some amount of AI are capable of running a simple inference in the sensor itself. For a more complex inference, a separate edge processor is needed.TI Sitara™ processors are great devices for edge processing. With a scalable portfolio integrating Arm® Cortex®-A cores with flexible peripherals, Sitara processors provide industrial-grade solutions with an extensive mix of processing, integration and power efficiency. With TI Deep Learning (TIDL) software, Sitara AM57x processors leverage hardware acceleration to improve the performance of machine learning and deep learning inference at the edge.TIDL software is a set of open-source Linux software packages included on the processor software development kit (SDK) and tools that enable the hardware-accelerated offloading of deep learning inference in an AM57x device. It’s possible to accelerate inference across embedded vision engine (EVE) subsystems, C66x digital signal processor cores or a combination of both, using a set of application programming interfaces built with Open Computing Language (OpenCL).For those who already have an existing neural network framework, TIDL software bridges the gap between the popular Caffe and TensorFlow frameworks with accelerated hardware on the AM57x through an import tool, as shown in Figure 3. To maximize efficiency on the hardware, or for those who don’t have an existing neural network framework, Caffe Jacinto, a TI-developed framework forked from NVIDIA/Caffe and designed for the embedded space, can perform the training part of deep learning.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1722330564360-Screenshot+2024-07-30+110804.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 3. Texas Instruments Deep Learning (TIDL) Software Development Flow.Note that while TI has developed a framework that can be used to train a network model for an embedded application, the training itself still takes place on a desktop or cloud platform. It is the inference that runs on the TI embedded processor.With free software as part of TI’s Processor SDK, training videos and a deep learning TI reference design, whether you’re new to deep learning or a seasoned expert, TI provides the building blocks to get you started quickly and easily.Additional Resources:• <a href="http://www.ti.com/tool/sitara-machine-learning" target="_blank" rel="noopener noreferrer">Visit the TI Sitara Machine Learning page.</a>• <a href="https://e2e.ti.com/blogs_/b/behind_the_wheel/archive/2018/02/08/ai-in-automotive-practical-deep-learning?" target="_blank" rel="noopener noreferrer">Read more about machine learning</a>

Even if you’re not very familiar with deep learning, you’ve probably heard about it and how it can, among
other things, help automate the driving experience, increase manufacturing efficiency and change the consumer
shopping experience.

Bringing the Next Evolution of Machine Learning to the Edge

Neural networks have made a seismic impact on how engineers design controllers for robots, catalyzing more adaptive and efficient machines. Still, these brain-like machine-learning systems are a double-edged sword: Their complexity makes them powerful, but it also makes it difficult to guarantee that a robot powered by a neural network will safely accomplish its task. The traditional way to verify safety and stability is through techniques called Lyapunov functions. If you can find a Lyapunov function whose value consistently decreases, then you can know that unsafe or unstable situations associated with higher values will never happen. For robots controlled by neural networks, though, prior approaches for verifying Lyapunov conditions didn’t scale well to complex machines.Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and elsewhere have now developed new techniques that rigorously certify Lyapunov calculations in more elaborate systems. Their algorithm efficiently searches for and verifies a Lyapunov function, providing a stability guarantee for the system. This approach could potentially enable safer deployment of robots and autonomous vehicles, including aircraft and spacecraft.To outperform previous algorithms, the researchers found a frugal shortcut to the training and verification process. They generated cheaper counterexamples — for example, adversarial data from sensors that could’ve thrown off the controller — and then optimized the robotic system to account for them. Understanding these edge cases helped machines learn how to handle challenging circumstances, which enabled them to operate safely in a wider range of conditions than previously possible. Then, they developed a novel verification formulation that enables the use of a scalable neural network verifier, α,β-CROWN, to provide rigorous worst-case scenario guarantees beyond the counterexamples.“We’ve seen some impressive empirical performances in AI-controlled machines like humanoids and robotic dogs, but these AI controllers lack the formal guarantees that are crucial for safety-critical systems,” says Lujie Yang, MIT electrical engineering and computer science (EECS) PhD student and CSAIL affiliate who is a co-lead author of a new paper on the project alongside Toyota Research Institute researcher Hongkai Dai SM ’12, PhD ’16. “Our work bridges the gap between that level of performance from neural network controllers and the safety guarantees needed to deploy more complex neural network controllers in the real world,” notes Yang.For a digital demonstration, the team simulated how a quadrotor drone with lidar sensors would stabilize in a two-dimensional environment. Their algorithm successfully guided the drone to a stable hover position, using only the limited environmental information provided by the lidar sensors. In two other experiments, their approach enabled the stable operation of two simulated robotic systems over a wider range of conditions: an inverted pendulum and a path-tracking vehicle. These experiments, though modest, are relatively more complex than what the neural network verification community could have done before, especially because they included sensor models. “Unlike common machine learning problems, the rigorous use of neural networks as Lyapunov functions requires solving hard global optimization problems, and thus scalability is the key bottleneck,” says Sicun Gao, associate professor of computer science and engineering at the University of California at San Diego, who wasn’t involved in this work. “The current work makes an important contribution by developing algorithmic approaches that are much better tailored to the particular use of neural networks as Lyapunov functions in control problems. It achieves impressive improvement in scalability and the quality of solutions over existing approaches. The work opens up exciting directions for further development of optimization algorithms for neural Lyapunov methods and the rigorous use of deep learning in control and robotics in general.”Yang and her colleagues’ stability approach has potential wide-ranging applications where guaranteeing safety is crucial. It could help ensure a smoother ride for autonomous vehicles, like aircraft and spacecraft. Likewise, a drone delivering items or mapping out different terrains could benefit from such safety guarantees.The techniques developed here are very general and aren’t just specific to robotics; the same techniques could potentially assist with other applications, such as biomedicine and industrial processing, in the future. While the technique is an upgrade from prior works in terms of scalability, the researchers are exploring how it can perform better in systems with higher dimensions. They’d also like to account for data beyond lidar readings, like images and point clouds.As a future research direction, the team would like to provide the same stability guarantees for systems that are in uncertain environments and subject to disturbances. For instance, if a drone faces a strong gust of wind, Yang and her colleagues want to ensure it’ll still fly steadily and complete the desired task.&nbsp; Also, they intend to apply their method to optimization problems, where the goal would be to minimize the time and distance a robot needs to complete a task while remaining steady. They plan to extend their technique to humanoids and other real-world machines, where a robot needs to stay stable while making contact with its surroundings. Russ Tedrake, the Toyota Professor of EECS, Aeronautics and Astronautics, and Mechanical Engineering at MIT, vice president of robotics research at TRI, and CSAIL member, is a senior author of this research. The paper also credits University of California at Los Angeles PhD student Zhouxing Shi and associate professor Cho-Jui Hsieh, as well as University of Illinois Urbana-Champaign assistant professor Huan Zhang. Their work was supported, in part, by Amazon, the National Science Foundation, the Office of Naval Research, and the AI2050 program at Schmidt Sciences. The researchers’ paper will be presented at the 2024 International Conference on Machine Learning.

MIT CSAIL researchers helped design a new technique that can guarantee the stability of robots controlled by neural networks. This development could eventually lead to safer autonomous vehicles and industrial robots. Image: Alex Shipps/MIT CSAIL

Neural network controllers provide complex robots with stability guarantees, paving the way for the safer deployment of autonomous vehicles and industrial machines.

Creating and verifying stable AI-controlled systems in a rigorous and flexible way

The world of artificial intelligence (AI) has recently seen significant advancements in generative models, a type of machine-learning algorithms that “learn” patterns from set of data in order to generate new, similar sets of data. Generative models are often used for things like drawing images and natural language generation – a famous example are the models used to develop chatGPT.Generative models have had remarkable success in various applications, from image and video generation to composing music and to language modeling. The problem is that we are lacking in theory, when it comes to the capabilities and limitations of generative models; understandably, this gap can seriously affect how we develop and use them down the line.One of the main challenges has been the ability to effectively pick samples from complicated data patterns, especially given the limitations of traditional methods when dealing with the kind of high-dimensional and complex data commonly encountered in modern AI applications.Now, a team of scientists led by Florent Krzakala and Lenka Zdeborová at EPFL has investigated the efficiency of modern neural network-based generative models. The study, now published in&nbsp;PNAS, compares these contemporary methods against traditional sampling techniques, focusing on a specific class of probability distributions related to spin glasses and statistical inference problems.The researchers analyzed generative models that use neural networks in unique ways to learn data distributions and generate new data instances that mimic the original data.The team looked at flow-based generative models, which learn from a relatively simple distribution of data and “flow” to a more complex one; diffusion-based models, which remove noise from data; and generative autoregressive neural networks, which generate sequential data by predicting each new piece based on the previously generated ones.The researchers employed a theoretical framework to analyze the performance of the models in sampling from known probability distributions. This involved mapping the sampling process of these neural network methods to a Bayes optimal denoising problem – essentially, they compared how each model generates data by likening it to a problem of removing noise from information.The scientists drew inspiration from the complex world of spin glasses, materials with intriguing magnetic behavior, to analyze modern data generation techniques. This allowed them to explore how neural network-based generative models navigate the intricate landscapes of data.The approach allowed them to study the nuanced capabilities and limitations of the generative models against more traditional algorithms like Monte Carlo Markov Chains (algorithms used to generate samples from complex probability distributions) and Langevin Dynamics (a technique for sampling from complex distributions by simulating the motion of particles under thermal fluctuations).The study revealed that modern diffusion-based methods may face challenges in sampling due to a first-order phase transition in the algorithm’s denoising path. What this means is that they can run into problems because of sudden change in how they remove noise from the data they're working with. Despite identifying regions where traditional methods outperform, the research also highlighted scenarios where neural network-based models exhibit superior efficiency.This nuanced understanding offers a balanced perspective on the strengths and limitations of both traditional and contemporary sampling methods. The research is a guide to more robust and efficient generative models in AI; by providing a clearer theoretical foundation, it can help develop next-generation neural networks capable of handling complex data generation tasks with unprecedented efficiency and accuracy.<hr>ReferencesDavide Ghio, Yatin Dandi, Florent Krzakala, Lenka Zdeborovà. Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective. PNAS 24 June 2024. DOI:&nbsp;<a href="https://doi.org/10.1073/pnas.2311810121" target="_blank" rel="noopener">10.1073/pnas.2311810121</a>

Researchers at EPFL have made a breakthrough in understanding how neural network-based generative models perform against traditional data sampling techniques in complex systems, unveiling both challenges and opportunities for AI's future in data generation.

Navigating the labyrinth: How AI tackles complex data sampling

Our brains are marvels of efficiency, capable of doing complex tasks while consuming minimal power. But replicating that efficiency with traditional computers has proven difficult. This is where neuromorphic computing comes in. Inspired by the brain's architecture, neuromorphic chips are poised to revolutionize AI, especially at the edge where low-power processing is crucial.This episode of the <a href="https://brainchip.com/" target="_blank" rel="noopener noreferrer">BrainChip</a> podcast features a conversation between BrainChip CEO Sean Hehir and Dr. Jason K. Eshraghian, a leading expert in neuromorphic computing. They explore the advantages of this new technology over traditional AI methods and consider its potential to transform the future of Edge AI.<h2 dir="ltr">Neuromorphic Approach to AI</h2>When asked about the core principles of neuromorphic computing, Dr. Jason K. Eshraghian highlighted some key differences from traditional AI architectures:<ul><li dir="ltr">Brain-Inspired design: Neuromorphic computing takes inspiration from the human brain," explained Dr. Eshraghian. "Unlike CPUs and GPUs that separate processing and memory, neuromorphic chips integrate both functions, mimicking the brain's structure for improved efficiency.”</li><li dir="ltr">Event-driven computation:&nbsp;The human brain doesn't waste energy on constantly processing information. Instead, it relies on a system of "spikes" to transmit signals when relevant activity occurs. Neuromorphic chips emulate this approach, only processing data when needed, further reducing power consumption.</li><li dir="ltr">Sparse representations: The human brain is not uniformly active. Similarly, neuromorphic chips can exploit the inherent sparsity in AI models, focusing on the relevant parts of the data and ignoring the rest. This allows for more efficient use of memory and bandwidth compared to dense representations used in conventional neural networks.</li></ul><h2 dir="ltr">Neuromorphic Chips and Edge AI</h2>The interview shifted gears when the topic of edge AI came up. Dr. Eshraghian expressed his enthusiasm for how neuromorphic computing aligns perfectly with this growing field:"The advantages of neuromorphic chips become particularly compelling when applied to edge AI applications,"&nbsp;Dr. Eshraghian explained. "By ditching the traditional separation of computation and memory, neuromorphic chips achieve significant efficiency gains. This, when combined with their ability only to process data when necessary, makes them ideal for powering intelligent systems directly on devices, closer to the source of the data."He elaborated on the benefits of edge AI:<ul><li dir="ltr">Enabling AI at the edge: The energy efficiency and low latency of neuromorphic architectures make them well-suited for powering AI models directly on edge devices, rather than relying on cloud-based processing. This allows for real-time inference and decision-making without the need to transmit data to remote servers.</li><li dir="ltr">Widespread deployment:&nbsp;While traditional neural networks running on GPUs can struggle to handle massive numbers of concurrent requests for language models, neuromorphic chips like BrainChip's Akida are designed to process many parallel requests efficiently. This makes them a better fit for large-scale edge deployments.</li><li dir="ltr">Sparse event-driven models: The sparse, event-driven nature of neuromorphic computing aligns well with language models like BrainChip's TENN. TENN also uses sparse representations and only processes data when necessary. This synergy allows for highly efficient edge AI systems that can handle real-world workloads.</li></ul><h2 dir="ltr">BrainChip's Akida and TENN: Hardware Solution Aligned with Neuromorphic Principles</h2>The podcast interview discussed BrainChip's Akida, a neuromorphic chip designed with the principles discussed earlier in mind. As Dr. Eshraghian emphasized, the event-driven nature of Akida aligns perfectly with his vision for efficient neuromorphic computing.Akida's architecture departs from traditional approaches by relying on an event-based processing system. This means the chip only activates specific neurons when necessary, mimicking the brain's spiking mechanism. This significantly reduces power consumption compared to constantly processing data streams.But BrainChip doesn't stop there. The introduction of TENN adds another layer of efficiency. <a href="https://brainchip.com/tenns-a-new-approach-to-streaming-and-sequential-data/" target="_blank">TENN</a> is a state-space model, a type of AI architecture that offers several advantages:<ul><li dir="ltr">Lightweight and efficient: TENN models require less memory and computational resources compared to traditional deep learning models. This is particularly beneficial for resource-constrained edge devices.</li><li dir="ltr">Fast processing:&nbsp;TENN models can be processed quickly on neuromorphic chips like Akida, enabling real-time decision-making at the edge.</li><li dir="ltr">Compatibility with Akida:&nbsp;The design of TENN makes it well-suited for implementation on Akida's architecture, further enhancing the efficiency of the entire system.</li></ul>By combining Akida's neuromorphic processing power with the lightweight nature of TENN, BrainChip offers a compelling solution for running powerful AI algorithms on edge devices.&nbsp;The potential of this approach goes beyond just edge AI. As Dr. Jason K. Eshraghian mentions in the podcast, TENN models can also be used in conjunction with large-scale cloud systems. This creates an interesting combination where lightweight models handle tasks at the edge while complex computations can be offloaded to the cloud when necessary.<h2 dir="ltr">Conclusion: A Look Ahead</h2>The conversation with Dr. Eshraghian sheds light on the exciting potential of neuromorphic computing. It's not about creating faster, more powerful AI; it's about achieving efficiency and unlocking new possibilities at the edge of networks. While neuromorphic computing is still evolving, it holds the promise to transform various fields, from industrial automation to personal devices.

In this podcast, Sean Hehir, CEO of BrainChip, chats with Dr. Jason K. Eshraghian about neuromorphic computing benefits over traditional AI and its potential to revolutionize the future of computing. Listen in to learn how this emerging technology is shaping the world of AI.


BrainChip Podcast: Neuromorphic Computing Shaping the Future of AI With Dr. Jason K. Eshraghian

In this episode, we discuss research coming out of the Karlsruhe Institute of Technology to detect the emotions of tennis players better than humans can!

Podcast: Catching Feelings with 69 percent Accuracy: AI Reads Emotions in Sports

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/5bf51127-82c4-4165-8792-9944842f1dbc?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Researchers from the Karlsruhe Institute of Technology (KIT) and the University of Duisburg Essen have managed to use computer-aided neural networks to precisely identify affective states based on the body language of tennis players during the game. For the first time, a model based on artificial intelligence (AI) was trained with data from real competition scenes. The study, published in the journal Knowledge-Based Systems, shows that AI can evaluate body language and emotions with similar accuracy to a human, but it also points to ethical aspects. ( <a href="https://www.sciencedirect.com/science/article/pii/S0950705124004908?via%3Dihub" target="_blank">DOI: 10.1016/j.knosys.2024.111856</a> )For the study “Detection of emotional states from the expressive behavior of tennis players using convolutional neural networks,” researchers from sports science, software development and computer science at KIT and the University of Duisburg Essen have developed a special AI model. Using computer-based pattern recognition programs, they analyzed video recordings of tennis players during real competitions.&nbsp;<h2>68.9 percent hit rate</h2>“Our model detects affective states with an accuracy of up to 68.9 percent. This is comparable to, and in some cases even superior to, the assessment by human observers as well as to previous automated methods,” explains study leader Professor Darko Jekauc from the Institute for Sport and Sports Science (IfSS) at KIT.&nbsp;An important unique feature of the study is that the project team did not train the AI system using simulated or posed situations, but rather using real scenes. To do this, the researchers recorded video sequences of 15 tennis players in a fixed setting. The focus was on body language when a point was lost or gained. The videos showed players with so-called cues, i.e. with cues such as a lowered head, arms thrown in the air in celebration, a hanging racket or differences in walking speed, which could be used to identify the players' affective state.&nbsp;Fed with this data, the AI learned to assign the body language signals to different affective reactions and to recognize whether a point was won (positive body language) or lost (negative body language). "Training in natural contexts represents a significant advance in recognizing real emotional states and enabling predictions in real-world scenarios," explains Jekauc.<h2>Humans and machines recognize negative emotions better than positive ones</h2>The research not only shows that AI algorithms could outperform human observers in emotion recognition in the future. The evaluation revealed another interesting aspect: both AI and humans are better at recognizing negative emotions. “This may be because they are easier to identify due to their clearer expressions,” explains Jekauc. “Psychological theories suggest that humans are evolutionarily more attuned to perceiving negative emotional expressions. For example, because it is crucial for the social context to defuse conflicts in a timely manner.”<h2>Before use: Clarification of ethical aspects necessary</h2>The study sees numerous areas of application for realistic emotion recognition in sport, from optimizing training methods, team dynamics and improving performance to preventing burnout. In addition, other areas could benefit from reliable early detection of emotional states, such as healthcare, educational institutions, customer service and automotive safety.&nbsp;“Although this technology offers significant benefits in the future, the potential risks associated with it must be taken into account, especially with regard to privacy and data misuse,” emphasizes Jekauc. “Our study was strictly based on existing ethical guidelines and data protection regulations. Even with a view to future applications of such technology in practice, it is essential to clarify ethical and legal questions in advance.”&nbsp;<hr><h3 id="isPasted">Original publication</h3>Darko Jekauc, Diana Burkart, Julian Fritsch, Marc Hesenius, Ole Meyer, Saquib Sarfraz, Rainer Schuhehagen: Recognizing affective states from the expressive behavior of tennis players using convolutional neural networks. Knowledge-Based Systems, Vol. 295, 2024. <a href="https://doi.org/10.1016/j.knosys.2024.111856" target="_blank">DOI: 10.1016/j.knosys.2024.111856</a>

KIT researchers have developed an AI model that can use body language to recognize emotions in tennis players. (Photo: PantherMedia / Wavebreakmedia ltd)

Researchers at KIT have developed a model for emotion analysis that can determine affective states with similar accuracy to a human

AI recognizes emotions in real sports situations

<h1 dir="ltr" id="isPasted">Introduction</h1>In an era where rapid technological advancements continuously reshape how industries operate, edge artificial intelligence (AI) has emerged as a critical driver of innovation and efficiency. Edge AI brings computation and data storage closer to the locations where it's needed, accelerating response times while reducing bandwidth use. This technology is particularly transformative in industrial and commercial applications, where real-time data processing can significantly enhance operational efficiency and decision-making.Enter NVIDIA Train Adapt Optimize (TAO) toolkit—an open-source suite designed to simplify and accelerate the creation of high-quality, customized, and enterprise-ready AI models. This toolkit democratizes access to state-of-the-art AI capabilities and streamlines the development process, enabling rapid deployment across a range of devices from cloud infrastructures to edge units.&nbsp;When integrated with Edge Impulse, a leading development platform for edge AI, NVIDIA TAO becomes part of a powerful ecosystem that enhances the capabilities of developers and engineers to bring AI models to production faster. NVIDIA TAO and Edge Impulse represent a formidable combination, poised to accelerate the adoption of edge AI technologies across various sectors by offering tools that reduce complexity and enhance productivity.This article explores how the NVIDIA TAO toolkit, along with Edge Impulse, facilitates the development and deployment of AI models on edge devices, illustrating the tangible benefits and real-world applications of this integration. By harnessing these advanced tools, developers can speed up the AI development lifecycle and achieve greater scalability in their AI projects.<iframe width="540" height="600" src="https://c8056756.sibforms.com/serve/MUIFAMyA6BGksOUPEokHuBnEmZRHhYgJ4RXb4BV0gvzN98ymNIvQdTcBFQnRVfviT6yZSFW_R2g1hENBu7p7Afnyv9Kt0unAAbfFU5mMQhBwI-8hQnKf2r1mmRKhUgDLmtk8bpVSZ7kZGpleCPqtYRv8_kkvODL7F4neGLNGgdpFDm9pPzoYpwbQkc0BU5GVOFoq8CSmD0DDvDjG" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> <h2 dir="ltr">The NVIDIA TAO Toolkit</h2>NVIDIA TAO Toolkit is a great tool for developers aiming to harness the power of AI without the complexity typically associated with such endeavors. As an open-source solution, it democratizes advanced technology, allowing a broad range of developers—from startups to large enterprises—to create customized, high-performance AI models that are both scalable and adaptable.<h3 dir="ltr">Overview of NVIDIA TAO</h3>At its core, the NVIDIA TAO Toolkit facilitates AI model development by leveraging NVIDIA's extensive expertise in AI and deep learning. Designed to streamline the process of model training and optimization, NVIDIA TAO empowers developers to build sophisticated AI models without deep specialization in AI technologies. Its open-source nature ensures that it is continually refined by contributions from the global development community, enhancing its capabilities and accessibility.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE1MjMyMTA3NTEyLWltYWdlMi5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib" alt="nvidia-tao-workflow">Fig. 1: Working with Nvidia TAO toolkit. Source:&nbsp;<a href="https://developer.nvidia.com/tao-toolkit-usecases-whitepaper/1-introduction" target="_blank" rel="noopener noreferrer">Nvidia</a><h3 dir="ltr">The Power of Transfer Learning</h3>One of the most significant features of the NVIDIA TAO Toolkit is its utilization of transfer learning. This technique revolutionizes how models are built by allowing developers to start with an existing model—pre-trained by NVIDIA or third-party sources on vast and diverse data sets—and customize it to their specific needs with relatively little data. This method drastically reduces the time and resources required for model training and enhances the model’s accuracy and performance from the outset.Transfer learning mimics human learning dynamics, where knowledge gained in one context is repurposed for another. For AI development, this means a developer can take a model that recognizes general objects and retrain it to recognize specialized equipment or scenarios specific to their industry. This flexibility opens up a plethora of applications, from enhancing manufacturing processes with visual inspection tools to improving safety protocols through real-time anomaly detection.<h2 dir="ltr"><iframe width="640" height="360" src="https://www.youtube.com/embed/kErfKDZ50To?&amp;ab_channel=NVIDIADeveloper&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;</iframe>Key Features and Benefits of NVIDIA TAO</h2>The NVIDIA TAO Toolkit offers a suite of capabilities that streamline the development process for AI models. Here, we explore how its features cater to efficiency, accuracy, customization, and deployment flexibility.<h3 dir="ltr">Efficient Model Training</h3>The NVIDIA TAO Toolkit simplifies the model training process significantly. Developers can leverage TAO’s AutoML capability, which automates much of the model tuning. This not only speeds up the model development cycle but also makes it more accessible to non-specialists. The AutoML feature efficiently navigates through the complexities of model tuning, allowing developers to focus more on application development rather than the nuances of model configuration.<h3 dir="ltr">High Accuracy and Customization</h3>NVIDIA is renowned for its high-performance computing and deep learning technologies, and TAO is no exception. It enables the creation of highly accurate AI models tailored to specific needs using state-of-the-art vision transformers and NVIDIA's pre-trained models. These capabilities allow for the development of bespoke AI solutions that are finely tuned to the unique requirements of different use cases, from healthcare diagnostics to automated surveillance systems.<h3 dir="ltr">Optimization for Inference and Deployment Flexibility</h3>Once a model is trained, ensuring it performs efficiently on the intended hardware is crucial. NVIDIA TAO facilitates this by enabling significant optimizations for inference, enhancing model accuracy and performance across GPUs, CPUs, and cloud servers. Additionally, Edge Impulse extends this adaptability to various deployment platforms, including microcontrollers and various other edge devices, ensuring that models integrate smoothly with existing infrastructures.<h3 dir="ltr">AI-Assisted Data Annotations</h3>Preparing data for training AI models is often labor-intensive. NVIDIA TAO eases this burden with AI-assisted data annotation capabilities. This feature accelerates the preparation of training data, making it faster and less costly.<h2 dir="ltr">Integrating NVIDIA TAO with Edge Impulse</h2>The collaboration between NVIDIA TAO and Edge Impulse presents a synergistic approach to edge AI development, significantly reducing time-to-market while increasing the scalability and efficiency of AI solutions. This integration facilitates a streamlined workflow from model creation to deployment, addressing key challenges in edge AI development.<h3 dir="ltr">Enhancing Edge AI Development</h3>By incorporating NVIDIA TAO into the Edge Impulse development platform, developers gain access to a robust set of tools designed to expedite the creation of efficient models. This integration leverages the strengths of both platforms—TAO’s powerful model training and Edge Impulse’s sophisticated deployment capabilities. Together, they simplify the development cycle, enabling developers to rapidly prototype, iterate, and deploy AI models that are both robust and scalable.<h3 dir="ltr">Comprehensive Edge AI Solution</h3>The combined power of NVIDIA TAO and Edge Impulse provides a comprehensive solution for edge AI. From collecting data on edge devices to training and validating models and finally optimizing libraries to run efficiently on target hardware, this integration covers all bases. It supports a wide range of devices, from low-power microcontrollers to more capable GPUs, ensuring that developers can deploy their AI solutions across diverse modern IoT and edge computing environments.Developers can tailor their solutions to specific hardware and performance needs, optimizing resource usage and operational efficiency. Additionally, the combined ecosystem supports a variety of silicon and sensors, making it an all-in-one solution for edge AI applications.<h2 dir="ltr">Practical Applications and Success Stories</h2>The integration of NVIDIA TAO with Edge Impulse offers a powerful platform for practical applications in various sectors. Here, we explore some specific use cases that demonstrate the effectiveness of this technology.PCB Classification:&nbsp;In electronics manufacturing, ensuring that all components are correctly placed and functional on a PCB is crucial. A minor error can lead to a malfunctioning device, leading to increased costs and reduced consumer trust. The PCBClassification model, developed using NVIDIA TAO, addresses this challenge effectively.<a href="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pcb_classification" target="_blank">[1]</a>This model utilizes an advanced AI architecture known as global context vision transformer (GCViT), initially trained on a comprehensive image dataset (NVImageNet). The model's training involves transfer learning, where a model developed for one task is partially retrained to perform another. In this case, the base model, proficient in general image recognition, is fine-tuned to specifically identify missing or defective components in PCB images.The process is straightforward yet powerful: images of PCBs are fed into the model, which has been trained to recognize various components like resistors, capacitors, and inductors. The model assesses each component's presence or absence and checks for defects. This automated inspection helps quickly spot errors during the manufacturing process, reducing the likelihood of defective products reaching the market.The accuracy of this model is notable, achieving near-perfect precision in tests. This high level of performance is crucial for industries where the cost of errors can be exceptionally high, both financially and in terms of brand reputation.Reducing Baggage Delays at Airports:&nbsp;The case of Trifork using NVIDIA TAO for baggage tracking at airports highlights the transformative potential of these technologies in enhancing operational efficiency.<a href="https://resources.nvidia.com/en-us-metropolis-software-success-stories/trifork" target="_blank">[2]</a> By deploying vision AI solutions, Trifork has been able to track millions of pieces of luggage, significantly reducing baggage delays and lost baggage incidents. This application not only improves the passenger experience but also optimizes the workflow and resource allocation at airports.Industrial Defect Detection:&nbsp;Another compelling application is in the field of industrial defect detection.<a href="https://developer.nvidia.com/blog/transforming-industrial-defect-detection-with-nvidia-tao-and-vision-ai-models/" target="_blank">[3]</a> Using NVIDIA TAO, developers have created models that can identify defects in various industrial products with high precision. This application is particularly beneficial for quality control in manufacturing, where detecting and addressing defects early in the production process can lead to substantial cost savings and maintain high-quality standards.Retail Recognition:&nbsp;In the retail sector, the application of NVIDIA TAO for retail recognition tasks illustrates how edge AI can transform inventory management. The models could help with identifying a vast array of retail Stock Keeping Units (SKUs), facilitating better stock management, customer service, and logistical efficiency. By automating the recognition process, retailers can ensure more accurate inventory tracking and faster response times, leading to enhanced operational efficiency and customer satisfaction.These practical applications and success stories demonstrate the broad utility and effectiveness of NVIDIA TAO when integrated with tools like Edge Impulse. The technology not only drives innovation and efficiency in existing processes but also opens up new possibilities for smart manufacturing, retail management, and beyond.Further reading:&nbsp;<a href="https://www.wevolver.com/article/a-practical-guide-to-understanding-sensor-fusion-with-edge-impulse-building-a-smart-running-jacket" target="_blank" id="isPasted">A Practical Guide to Understanding Sensor Fusion with Edge Impulse: Building a Smart Running Jacket</a><h2 dir="ltr">Getting Started and Resources</h2>To begin utilizing NVIDIA TAO and Edge Impulse to develop and deploy AI models on edge devices, it's essential to engage with a variety of resources and steps designed to streamline this process.Firstly, a thorough exploration of the available documentation is crucial. Both <a href="https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/nvidia-tao" target="_blank">Edge Impulse</a> and <a href="https://developer.nvidia.com/tao-toolkit" target="_blank">NVIDIA</a> provide comprehensive guides and tutorials on their respective websites. These documents are invaluable for understanding how to effectively integrate NVIDIA TAO models into Edge Impulse projects.Additionally, Edge Impulse and Nvidia provide YouTube videos that detail the application and integration of NVIDIA TAO. These resources complement the written documentation and provide a medium through which users can understand how to work with these technologies.<iframe width="640" height="360" src="https://www.youtube.com/embed/bwquofcF94g?&amp;ab_channel=EdgeImpulse&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Support doesn’t end with documentation and tutorials. NVIDIA and Edge Impulse have active forums like - <a href="https://forum.edgeimpulse.com/" target="_blank">Edge Impulse Forums</a> and <a href="https://forums.developer.nvidia.com/c/accelerated-computing/intelligent-video-analytics/tao-toolkit/17" target="_blank">NVIDIA Developer Forums</a>, where users can engage with a community of like-minded individuals. These forums are a great place to seek advice, ask questions, and share experiences related to edge AI development.By leveraging these resources, developers can effectively navigate the initial learning curve and start applying NVIDIA TAO and Edge Impulse to real-world projects, gradually building up to more complex applications as familiarity with the tools increases.Further reading:&nbsp;<a href="https://www.wevolver.com/article/building-high-performance-edge-ai-solutions-with-nvidia-jetson-and-edge-impulse" target="_blank" rel="noopener noreferrer">Building High-Performance Edge AI Solutions with Nvidia Jetson and Edge Impulse</a><h2 dir="ltr">Conclusion</h2>The integration of NVIDIA TAO and Edge Impulse brings in a new era of edge AI development, offering developers powerful tools to enhance the efficiency and effectiveness of their AI models. This combination brings together NVIDIA's advanced AI modeling capabilities with Edge Impulse's robust customization and deployment framework, enabling rapid development, optimization, and deployment of AI applications across a variety of platforms and devices.For developers and companies venturing into edge AI, the synergy between NVIDIA TAO and Edge Impulse provides a competitive edge. By simplifying complex processes and reducing the time-to-market, this integration democratizes advanced AI technologies and empowers a broader range of users to innovate and solve real-world problems.<iframe width="540" height="600" src="https://c8056756.sibforms.com/serve/MUIFAMyA6BGksOUPEokHuBnEmZRHhYgJ4RXb4BV0gvzN98ymNIvQdTcBFQnRVfviT6yZSFW_R2g1hENBu7p7Afnyv9Kt0unAAbfFU5mMQhBwI-8hQnKf2r1mmRKhUgDLmtk8bpVSZ7kZGpleCPqtYRv8_kkvODL7F4neGLNGgdpFDm9pPzoYpwbQkc0BU5GVOFoq8CSmD0DDvDjG" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> <h2 dir="ltr">References</h2>[1] PCBClassification Model Card, Nvidia, 17 Oct 2023, [Online], Available from: <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pcb_classification" target="_blank">https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pcb_classification</a>[2] Trifork - Reducing baggage delay at airports with vision AI, Nvidia, [Online], Available from: <a href="https://resources.nvidia.com/en-us-metropolis-software-success-stories/trifork" target="_blank">https://resources.nvidia.com/en-us-metropolis-software-success-stories/trifork</a>[3] Nirmal Kumar Juluru, Zaid Pervaiz Bhat and Samuel Ochoa, ‘Transforming Industrial Defect Detection with NVIDIA TAO and Vision AI Models’, Nvidia, [Online], Available from: <a href="https://developer.nvidia.com/blog/transforming-industrial-defect-detection-with-nvidia-tao-and-vision-ai-models/" target="_blank">https://developer.nvidia.com/blog/transforming-industrial-defect-detection-with-nvidia-tao-and-vision-ai-models/</a>

NVIDIA TAO and Edge Impulse integrate to streamline the development and deployment of edge AI models, empowering developers to quickly build and optimize AI applications for diverse hardware environments.

Accelerating Edge AI Model Development Using Nvidia TAO and Edge Impulse

<h1 dir="ltr" id="isPasted">Introduction</h1>The modern workplace is rapidly changing and AI is emerging as a transformative force. With the growing use of AI-powered tools, companies are rethinking how to approach hiring, employee onboarding, and enhancing the efficiency, development, and well-being of employees.In this article, we delve into the role of AI in the workplace, highlighting its impact on hiring processes, improving employee retention, and workplace conditions. We present case studies of companies who have found success with AI tools. We also explore future trends and ethical considerations, providing an overview of how AI is positioned to help companies hire more effectively and create better places to work.<h1 dir="ltr">AI in the Hiring Process</h1>Hiring is time-consuming and costly for companies, and often frustratingly involves processes that overlook the most qualified candidates. Some problems with candidate screening, matching, and interviewing are:<ul><li dir="ltr">Over-reliance on keywords and algorithms, which may screen out very qualified candidates whose resumes do not precisely match job descriptions.</li><li dir="ltr">Bias in recruitment, which can hinder diversity and inclusion efforts and lead to a homogenized workforce.</li><li dir="ltr">High-volume applications and initial screenings, a task that relies heavily on the judgment of recruiters and is subject to many human factors.[1]</li></ul>Companies can address these challenges through the implementation of AI technologies in the recruitment process. The following are a few examples of the potential for AI to help companies make hiring more efficient and inclusive.<h2 dir="ltr">Streamlining candidate screening</h2>AI-powered applicant tracking systems can streamline sourcing and screening automating resume analysis and identifying qualified candidates efficiently and accurately. These are some case studies where AI is already improving efficiency in candidate screening.<ul><li dir="ltr">IBM Watson recruitment at Hilton&nbsp;– By integrating IBM Watson for candidate screening and matching, the Hilton hospitality chain saw a 50% reduction in time-to-fill, improved candidate quality, and enhanced hiring manager satisfaction.</li><li dir="ltr">Pymetrics gamified assessments at Deloitte&nbsp;– Deloitte, the US auditing and tax services company incorporated Pymetrics’ AI platform for gamified assessments into their candidate screening process. As a result, the company saw a 60% reduction in time spent on initial screenings, a 30% increase in diverse hires, and enhanced candidate experience.</li><li dir="ltr">HireVue video interviews at McDonald’s&nbsp;– The fast food chain McDonald’s implemented HireVue’s AI-enhanced video screening to streamline their high-volume hiring process. Use of the platform resulted in a 40% reduction in time-time hire, improved candidate engagement, and enhanced scalability.[2]</li></ul><h2 dir="ltr">Enhancing candidate matching</h2>AI algorithms can be used to match job skills and requirements without the over-reliance on specific job description keywords that is often the case with traditional algorithms. AI algorithms can use natural language processing to analyze candidate data and match actual qualifications with job requirements, improving the accuracy of candidate matching.[3]Additionally, by focusing on skills, qualifications, and experience rather than personal characteristics, AI can help reduce unconscious bias in recruitment, leading to a more diverse and inclusive workforce.[4]<h2 dir="ltr">Interview and assessment automation</h2>Companies can harness the power of AI using deep learning and natural language processing to improve the interview process for both candidates and recruiters. Image credit: mikemacmarketing | Source: wikimedia.orgAI-driven recruitment strategies can help make the interview process a much better experience for candidates and recruiters alike. AI tools can automate interview scheduling, conduct initial screenings, ask follow-up questions, and use natural language processing to analyze responses, making the interview process more effective and efficient.[5] In the case studies discussed above, the introduction of AI tools improved the recruitment experience for hiring managers and candidates, and improved candidate engagement.By incorporating AI tools into hiring, organizations can streamline candidate screening, improve matching accuracy, reduce bias, enhance efficiency, and ultimately create a more inclusive and effective hiring environment.<h1 dir="ltr">AI's Role in Employee Onboarding and Retention</h1>While it is important to find and hire the best candidates for the job, it is equally important to make sure that new employees are quickly brought up to speed and productive. Perhaps even more critical is to keep turnover rates low and retain employees by making sure that they are satisfied with the job. Companies can use AI-powered tools to build a more effective and engaged workforce.<h2 dir="ltr">Automated onboarding processes</h2>AI-powered recruitment software can analyze the experience, knowledge, and learning style of a new hire and generate tailored onboarding programs. This type of onboarding takes the employee’s personality and background into account, to help a new employee quickly become engaged and part of company culture.This type of streamlined onboarding means that a new employee becomes productive faster and may be more satisfied with the job overall. Research shows that a successful onboarding experience leads to greater employee retention — employees are twice as likely to be satisfied at work and contribute to a positive company culture.[6]<h2 dir="ltr">Predictive analytics for employee turnover</h2>AI is a powerful tool because it can collect and process large volumes of data. This ability to sift through data and identify patterns and correlations can be leveraged to pinpoint the things that contribute to employee turnover. Companies can use historical turnover data to develop models that predict potential turnover risks.With the power of AI, companies can better understand the specific reasons for turnover, make improvements, and test the outcome. For example, if historical turnover data points to dissatisfaction with limited opportunities for growth, the company can offer career development programs, use AI to monitor the result, and make improvements to the programs.[7]AI tools can have a significant impact on employee onboarding and retention, leading to more engaged, productive, and loyal employees.<h1 dir="ltr">Improving Workplace Conditions with AI</h1>Working conditions are another factor in employee satisfaction and retention. Companies can leverage AI-powered tools to optimize the workspace, support career development, and maintain employee health.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1710149861252-1NAGBQZbUUYzSHV_ckw0Z_sG2he6dCo8.jpg" style="width: 100%px;" class="fr-fic fr-dib">Employers can leverage AI to optimize the working environment while improving employee well-being and retention. Image credit: Alice Mai Ah | Source: wikimedia.org<h2 dir="ltr" id="isPasted">AI for workspace optimization</h2>Using AI, companies can create smart work environments that adapt to employee patterns, preferences, and usage. Automatic environmental adjustments like lighting, temperature, and desk configurations can lead to increased comfort, productivity, and energy cost savings for the company.AI systems can also be used to optimize resources like meeting rooms, workstations, and common areas, making sure facilities are used effectively and that employee needs are supported.<h2 dir="ltr">AI-enhanced training and development</h2>As discussed above, AI can be used during onboarding to get new employees working quickly. In addition to new hire training, AI tools can be used for ongoing growth, tailoring development programs to each employee’s learning styles, career goals, and performance. Using AI analytics, organizations can track the effectiveness of development programs and continuously improve them, leading to better career growth opportunities and satisfied employees.<h2 dir="ltr">Ensuring health and well-being</h2>Predictive AI analytics can play a crucial role in improving employee mental health by analyzing patterns in employee data and feedback to catch signs of stress or burnout. Companies can use this data to take proactive measures that support the well-being of employees.AI-powered ergonomic assessments and activity reminders can also help improve the workplace. Wearable devices and AI applications can provide data on workplace ergonomics and suggest improvements. Wearables can also remind employees to take breaks and stretch to maintain both mental and physical health.<h1 dir="ltr">Case Studies: AI-Driven Success in Hiring and Workplace Enhancement</h1>Companies today are already leveraging powerful AI tools to enhance hiring and the workplace. These are just a few case studies that show how companies are seeing success with these tools.<ul><li dir="ltr">ClearCompany&nbsp;– ClearCompany, a recruiting software platform, has automated employment verification, document submission, and analysis of learning pase using AI-driven onboarding software. This has resulted in a more efficient onboarding process and increased focus on strategic work that drives company growth.[6]</li><li dir="ltr">Genpact&nbsp;– This company has used AI to automate their entire onboarding experience, providing personalized experiences for new hires, which has helped speed up the onboarding process and improve retention.[8]</li><li dir="ltr">Tier 4 Group&nbsp;– This talent acquisition firm worked with TechScreen to streamline their tech hiring processes, resulting in a more effective, ethical, and considerate candidate screening process.[9]</li><li dir="ltr">Iceland Foods – During the pandemic, Iceland Foods partnered with Sapia.ai to adapt their recruitment processes to a changing world. AI automation helped the company find the right candidates in the midst of global upheaval, demonstrating the resilience of AI-driven recruitment.[10]</li></ul>These examples illustrate the diverse applications of AI in hiring and workplace enhancement, from improving candidate experience and streamlining the recruitment process, to fostering internal development and supporting ethical hiring practices.<h1 dir="ltr">Looking ahead: the next generation of AI in the workplace</h1>The integration of AI into the workplace is poised to have a huge impact on how we work. Here are some ways that we predict AI will transform the workplace in specific industries.<ul><li dir="ltr">Healthcare&nbsp;– By integrating data like medical imaging, lab results, and checkups, AI advancements will likely lead to more efficient healthcare systems, as well as more effective care, with better outcomes for patients.</li><li dir="ltr">Finance&nbsp;– The role of AI in fraud detection, personalized financial advising, and record keeping is likely to grow, but a strong emphasis on human oversight and ethical decision-making will remain a key part of the finance industry.</li><li dir="ltr">Education&nbsp;– In education, AI will be able to analyze and provide insights into student learning, and will help institutions create more engaging and personalized experiences for both students and teachers.</li><li dir="ltr">Retail&nbsp;– Predictive analytics will enable retailers to understand customer behavior, manage inventory, and improve customer service. AI has the potential to boost customer satisfaction and operational efficiency.</li><li dir="ltr">Manufacturing&nbsp;– AI will significantly impact how work is done in the manufacturing industry as well. Collaborative robots (cobots), predictive maintenance, and smart factories are AI-powered tools that will likely improve the safety and efficiency of manufacturing.</li></ul><h1 dir="ltr">Ethical considerations: balancing technology and humanity</h1>AI is a promising tool for improving workplaces and the efficiency, health, and well-being of workers. And because AI is for the benefit of humanity, it is important to examine ethical considerations and maintain a balance between technology and human values.When using AI tools in hiring and HR functions, It is critical for companies to remain vigilant against bias and discrimination. Laws, standards, and guidelines that require AI tools to be audited for bias are already in place. It is also important for companies to monitor AI tools for bias during recruiting, hiring, and promotion.[11]In an increasingly automated workplace, it is also essential for companies to balance AI and human interaction. While AI magnifies our human capabilities and streamlines the work that we do, it is important for companies to foster human relationships and empathy, and to take an ethical stance to put humans first when it comes to AI strategies.<h2 dir="ltr">Conclusion</h2>The integration of AI into the hiring process, employee onboarding, and workplace improvement is a significant shift towards a more data-driven, efficient, and employee-centric approach to company management. As underscored by the case studies above, there are already tangible benefits to AI-driven strategies, from streamlined hiring processes to personalized onboarding experiences, to optimized workplace conditions that prioritize employee health and well-being.As AI continues to evolve and is increasingly integrated into our workplaces, it is important to address ethical considerations and maintain a balance between technological advancements and human values. The future of AI in the workplace holds great promise for improving how we hire and work, and has the power to transform industries, improve the lives of workers, and shape how we think about work.<h2>References</h2>[1] “Navigating the Maze: The Fracture’s in Today’s IT Recruiting Process &amp; the AI Solution”. TechScreen. November 27, 2023. Accessed from <a href="https://techscreen.com/navigating-the-maze-the-fractures-in-todays-it-recruiting-process-the-ai-solution/" target="_blank">https://techscreen.com/navigating-the-maze-the-fractures-in-todays-it-recruiting-process-the-ai-solution/</a>&nbsp;[2] Jessica Dennis. “AI Recruiting: Uses, Advantages, &amp; Disadvantages”. TechnologyAdvice. 22 May 2023. Accessed from <a href="https://technologyadvice.com/blog/human-resources/ai-recruiting/" target="_blank">https://technologyadvice.com/blog/human-resources/ai-recruiting/</a>&nbsp;[3] Ranganath Ponnaboyina, et al. “Smart Recruitment System Using Deep Learning with Natural Language Processing”. Intelligent Systems and Sustainable Computing, Vol. 289. 29 May 2022. Accessed from <a href="https://link.springer.com/chapter/10.1007/978-981-19-0011-2_58" target="_blank">https://link.springer.com/chapter/10.1007/978-981-19-0011-2_58</a>&nbsp;[4] “Pros and Cons of Using AI in Recruiting”. Recruiter.com. Accessed from <a href="https://www.recruiter.com/recruiting/pros-and-cons-of-using-ai-in-recruiting/" target="_blank">https://www.recruiter.com/recruiting/pros-and-cons-of-using-ai-in-recruiting/</a>&nbsp;[5] Paul Arolye. “Navigating Recruitment Challenges: 5 Ways AI Can Revolutionize HR”. eLearning Industry. 27 January 2024. Accessed from <a href="https://elearningindustry.com/navigating-recruitment-challenges-ways-ai-can-revolutionize-hr" target="_blank">https://elearningindustry.com/navigating-recruitment-challenges-ways-ai-can-revolutionize-hr</a>&nbsp;[6] ClearCompany. “3 Benefits of Automating Employee Onboarding with AI-Driven Software”. 5 September 2023. Accessed from <a href="https://blog.clearcompany.com/benefits-of-automating-employee-onboarding-with-ai" target="_blank">https://blog.clearcompany.com/benefits-of-automating-employee-onboarding-with-ai</a>&nbsp;[7] Cabot Jaffee. “Leveraging AI to predict and reduce employee turnover.” Talent Management. 14 August 2023. Accessed from <a href="https://www.talentmgt.com/articles/2023/08/14/leveraging-ai-to-predict-and-reduce-employee-turnover/" target="_blank">https://www.talentmgt.com/articles/2023/08/14/leveraging-ai-to-predict-and-reduce-employee-turnover/</a>&nbsp;[8] Maria Korolov. “AI takes aim at employee turnover”. CIO. 7 April 2022. Accessed from <a href="https://www.cio.com/article/307980/ai-takes-aim-at-employee-turnover.html" target="_blank">https://www.cio.com/article/307980/ai-takes-aim-at-employee-turnover.html</a>&nbsp;[9] TechScreen. “Case Studies in Tech Hiring: AI Success Stories and Lessons Learned from Top Companies”. 28 December, 2023. Accessed from <a href="https://techscreen.com/case-studies-in-tech-hiring-ai-success-stories-and-lessons-learned-from-top-companies/" target="_blank">https://techscreen.com/case-studies-in-tech-hiring-ai-success-stories-and-lessons-learned-from-top-companies/</a>&nbsp;[10] Sapia.ai. “How Iceland Foods smashed its volume hiring goals”. Accessed from <a href="https://sapia.ai/resources/case-study/iceland/" target="_blank">https://sapia.ai/resources/case-study/iceland/</a>&nbsp;[11] Roy Mauer. “HR Must Be Vigilant About the Ethical Use of AI Technology”. SHRM. 26 September 2022. Accessed from <a href="https://www.shrm.org/topics-tools/news/technology/hr-must-vigilant-ethical-use-ai-technology" target="_blank">https://www.shrm.org/topics-tools/news/technology/hr-must-vigilant-ethical-use-ai-technology</a>&nbsp;

 AI is having an impact on hiring processes, employee retention, and workplace conditions. 

How AI will help companies hire more people and make the workplace better

We’ve all heard of Large Language Models, or LLMs – massive scale deep learning models trained on huge amounts of text that form the basis for chatbots like OpenAI’s ChatGPT. Next-generation Multimodal Models (MMs) can learn from inputs beyond text, including video, images, and sound.Creating MM models at a smaller scale poses significant challenges, including the problem of being robust to non-random missing information. This is information that a model doesn’t have, often due to some biased availability in resources. It is thus critical to ensure the model does not learn the patterns of biased missingness in making its predictions.<h2>MultiModN turns this around</h2>In response to this problem, researchers from the&nbsp;<a href="https://www.epfl.ch/labs/ml4ed/" target="_blank">Machine Learning for Education</a> (ML4ED) and&nbsp;<a href="https://www.epfl.ch/labs/mlo/" target="_blank">Machine Learning and Optimization</a> (MLO) Laboratories in<a href="https://www.epfl.ch/schools/ic/" target="_blank">&nbsp;EPFL’s School of Computer and Communication Sciences</a> have developed and tested the exact opposite to a Large Language Model.Spearheaded by Professor Mary-Anne Hartley, head of the Laboratory for<a href="https://www.epfl.ch/labs/mlo/igh-intelligent-global-health/" target="_blank">&nbsp;intelligent Global Health Technologies</a> hosted jointly in the MLO and the Yale School of Medicine and Professor Tanja Käser, head of ML4ED,&nbsp;MultiModN&nbsp;is a unique&nbsp;<a href="https://arxiv.org/abs/2309.14118" target="_blank" rel="noopener">Modular Multimodal Model</a>, presented recently at the&nbsp;<a href="https://neurips.cc/Conferences/2023" target="_blank" rel="noopener">NeurIPS2023</a> conference.Like existing Multimodal Models&nbsp;MultiModN can learn from text, images, video, and sound. Unlike existing MMs, it is made up of any number of smaller, self-contained, and input-specific modules that can be selected depending on the information available, and then strung together in a sequence of any number, combination, or type of input. Itcan then output any number, or combination, of predictions.“We evaluated&nbsp;MultiModN across ten real-world tasks including medical diagnosis support, academic performance prediction, and weather forecasting. Through these experiments, we believe that&nbsp;MultiModN is the first inherently interpretable, MNAR-resistant approach to multimodal modelling,” explained Vinitra Swamy, a PhD student with ML4ED and MLO and joint first author on the project.<h3>A first use case: medical decision-making</h3>The first use case for&nbsp;MultiModN will be as a clinical decision support system for medical personnel in low-resource settings. In healthcare, clinical data is often missing, perhaps due to resource constraints (a patient can’t afford the test) or resource abundance (the test is redundant due to a superior one that was performed).&nbsp;MultiModN is able to learn from this real-world data without adopting its biases, as well as adapting predictions to any combination or number of inputs.“Missingness is a hallmark of data in low-resource settings and when models learn these patterns of missingness, they may encode bias into their predictions. The need for flexibility in the face of unpredictably available resources is what inspired&nbsp;MultiModN,” explained Hartley, who is also a medical doctor.<h3>From the lab to real life</h3>Publication, however, is just the first step towards implementation. Hartley has been working with colleagues at Lausanne University Hospital<a href="https://www.chuv.ch/fr/chuv-home/" target="_blank" rel="noopener">&nbsp;(CHUV</a>) and Inselspital, University Hospital Bern&nbsp;<a href="https://www.insel.ch/fr/" target="_blank" rel="noopener">uBern</a> to conduct clinical studies focused on pneumonia and tuberculosis diagnosis in low resource settings and they are recruiting thousands of patients in South Africa, Tanzania, Namibia and Benin.The research teams undertook a large training initiative, teaching more than 100 doctors to systematically collect multimodal data including images and ultrasound video, so that&nbsp;MultiModN can be trained to be sensitive to real data coming from low resource regions.“We are collecting exactly the kind of complex multimodal data that&nbsp;MultiModN is designed to handle,” said Dr Noémie Boillat-Blanco, an infectious diseases doctor at CHUV. “We are excited to see a model that appreciates the complexity of missing resources in our settings and of systematic missingness of routine clinical assessments,” added Dr Kristina Keitel at Inselspital, University Hospital Bern.<h3>Machine learning for the public good</h3>The development and training of&nbsp;MultiModN is a continuation of EPFL efforts to adapt machine learning tools to reality and for the public good. It comes not long after the launch of<a href="https://actu.epfl.ch/news/epfl-s-new-large-language-model-for-medical-knowle/" target="_blank">&nbsp;Meditron</a>, the world’s best performing open source LLM also designed to help guide clinical decision-making.Both tools align with the mission of the new&nbsp;<a href="https://actu.epfl.ch/news/opening-of-the-new-epfl-ai-center-a-hub-for-ai-i-2/" target="_blank">EPFL AI Center</a> that focuses on how responsible and effective AI can advance technological innovation for the benefit of all sectors of society.Dr. Mary-Anne Hartley is a keynote speaker at the upcoming Applied Machine Learning Days (AMLD) at the SwissTech Convention Center March 23-26, 2024. Vinitra Swamy will be presenting MultiModN at AMLD’s Applied eXplainable AI track. Discover other keynotes<a href="https://2024.appliedmldays.org/media-9-speakers" target="_blank" rel="noopener">here</a>&nbsp;and register<a href="https://2024.appliedmldays.org/media-12-registration" target="_blank" rel="noopener">here</a>.

Machine Learning Neural Network © iStock

Researchers at EPFL have developed a new, uniquely modular machine learning model for flexible decision-making. It is able to input any mode of text, video, image, sound, and time-series and then output any number, or combination, of predictions. 

Anything-in-anything-out: a new modular AI model

Behrooz Tahmasebi — an MIT PhD student in the Department of Electrical Engineering and Computer Science (EECS) and an affiliate of the Computer Science and Artificial Intelligence Laboratory (CSAIL) — was taking a mathematics course on differential equations in late 2021 when a glimmer of inspiration struck. In that class, he learned for the first time about Weyl’s law, which had been formulated 110 years earlier by the German mathematician Hermann Weyl. Tahmasebi realized it might have some relevance to the computer science problem he was then wrestling with, even though the connection appeared — on the surface — to be thin, at best. Weyl’s law, he says, provides a formula that measures the complexity of the spectral information, or data, contained within the fundamental frequencies of a drum head or guitar string.Tahmasebi was, at the same time, thinking about measuring the complexity of the input data to a neural network, wondering whether that complexity could be reduced by taking into account some of the symmetries inherent to the dataset. Such a reduction, in turn, could facilitate — as well as speed up — machine learning processes.Weyl’s law, conceived about a century before the boom in machine learning, had traditionally been applied to very different physical situations — such as those concerning the vibrations of a string or the spectrum of electromagnetic (black-body) radiation given off by a heated object. Nevertheless, Tahmasebi believed that a customized version of that law might help with the machine learning problem he was pursuing. And if the approach panned out, the payoff could be considerable.He spoke with his advisor, Stefanie Jegelka — an associate professor in EECS and affiliate of CSAIL and the MIT Institute for Data, Systems, and Society — who believed the idea was definitely worth looking into. As Tahmasebi saw it, Weyl’s law had to do with gauging the complexity of data, and so did this project. But Weyl’s law, in its original form, said nothing about symmetry.He and Jegelka have now succeeded in modifying Weyl’s law so that symmetry can be factored into the assessment of a dataset’s complexity. “To the best of my knowledge,” Tahmasebi says, “this is the first time Weyl’s law has been used to determine how machine learning can be enhanced by symmetry.”The&nbsp;<a href="https://arxiv.org/abs/2303.14269" target="_blank">paper</a> he and Jegelka wrote earned a “Spotlight” designation when it was presented at the December 2023 conference on Neural Information Processing Systems — widely regarded as the world’s top conference on machine learning.This work, comments Soledad Villar, an applied mathematician at Johns Hopkins University, “shows that models that satisfy the symmetries of the problem are not only correct but also can produce predictions with smaller errors, using a small amount of training points. [This] is especially important in scientific domains, like computational chemistry, where training data can be scarce.”In their paper, Tahmasebi and Jegelka explored the ways in which symmetries, or so-called “invariances,” could benefit machine learning. Suppose, for example, the goal of a particular computer run is to pick out every image that contains the numeral 3. That task can be a lot easier, and go a lot quicker, if the algorithm can identify the 3 regardless of where it is placed in the box — whether it’s exactly in the center or off to the side — and whether it is pointed right-side up, upside down, or oriented at a random angle. An algorithm equipped with the latter capability can take advantage of the symmetries of translation and rotations, meaning that a 3, or any other object, is not changed in itself by altering its position or by rotating it around an arbitrary axis. It is said to be invariant to those shifts. The same logic can be applied to algorithms charged with identifying dogs or cats. A dog is a dog is a dog, one might say, irrespective of how it is embedded within an image.&nbsp;The point of the entire exercise, the authors explain, is to exploit a dataset’s intrinsic symmetries in order to reduce the complexity of machine learning tasks. That, in turn, can lead to a reduction in the amount of data needed for learning. Concretely, the new work answers the question: How many fewer data are needed to train a machine learning model if the data contain symmetries?There are two ways of achieving a gain, or benefit, by capitalizing on the symmetries present. The first has to do with the size of the sample to be looked at. Let’s imagine that you are charged, for instance, with analyzing an image that has mirror symmetry — the right side being an exact replica, or mirror image, of the left. In that case, you don’t have to look at every pixel; you can get all the information you need from half of the image — a factor of two improvement. If, on the other hand, the image can be partitioned into 10 identical parts, you can get a factor of 10 improvement. This kind of boosting effect is linear.To take another example, imagine you are sifting through a dataset, trying to find sequences of blocks that have seven different colors — black, blue, green, purple, red, white, and yellow. Your job becomes much easier if you don’t care about the order in which the blocks are arranged. If the order mattered, there would be 5,040 different combinations to look for. But if all you care about are sequences of blocks in which all seven colors appear, then you have reduced the number of things — or sequences — you are searching for from 5,040 to just one.Tahmasebi and Jegelka discovered that it is possible to achieve a different kind of gain — one that is exponential — that can be reaped for symmetries that operate over many dimensions. This advantage is related to the notion that the complexity of a learning task grows exponentially with the dimensionality of the data space. Making use of a multidimensional symmetry can therefore yield a disproportionately large return. “This is a new contribution that is basically telling us that symmetries of higher dimension are more important because they can give us an exponential gain,” Tahmasebi says.&nbsp;The NeurIPS 2023 paper that he wrote with Jegelka contains two theorems that were proved mathematically. “The first theorem shows that an improvement in sample complexity is achievable with the general algorithm we provide,” Tahmasebi says. The second theorem complements the first, he added, “showing that this is the best possible gain you can get; nothing else is achievable.”He and Jegelka have provided a formula that predicts the gain one can obtain from a particular symmetry in a given application. A virtue of this formula is its generality, Tahmasebi notes. “It works for any symmetry and any input space.” It works not only for symmetries that are known today, but it could also be applied in the future to symmetries that are yet to be discovered. The latter prospect is not too farfetched to consider, given that the search for new symmetries has long been a major thrust in physics. That suggests that, as more symmetries are found, the methodology introduced by Tahmasebi and Jegelka should only get better over time.According to Haggai Maron, a computer scientist at Technion (the Israel Institute of Technology) and NVIDIA who was not involved in the work, the approach presented in the paper “diverges substantially from related previous works, adopting a geometric perspective and employing tools from differential geometry. This theoretical contribution lends mathematical support to the emerging subfield of ‘Geometric Deep Learning,’ which has applications in graph learning, 3D data, and more. The paper helps establish a theoretical basis to guide further developments in this rapidly expanding research area.”

New MIT research provides a theoretical proof for a phenomenon observed in practice: that encoding symmetries in the machine learning model helps the model learn with fewer data. Image: Alex Shipps/MIT CSAIL

Exploiting the symmetry within datasets, MIT researchers show, can decrease the amount of data needed for training neural networks.

How symmetry can come to the aid of machine learning

In this episode, we talk about MIT researchers making a smart tool (like a robot scientist) that uses AI to understand and explain how other AI brains (neural networks) work.

neural network

Profiles