Researchers in the emerging field of spatial computing have developed a prototype augmented reality headset that uses holographic imaging to overlay full-color, 3D moving images on the lenses of what would appear to be an ordinary pair of glasses. Unlike the bulky headsets of present-day augmented reality systems, the new approach delivers a visually satisfying 3D viewing experience in a compact, comfortable, and attractive form factor suitable for all-day wear.“Our headset appears to the outside world just like an everyday pair of glasses, but what the wearer sees through the lenses is an enriched world overlaid with vibrant, full-color 3D computed imagery,” said <a href="https://web.stanford.edu/~gordonwz/" data-extlink="" target="_blank">Gordon Wetzstein</a>, an associate professor of electrical engineering and an expert in the fast-emerging field of spatial computing. Wetzstein and a team of engineers introduce their device in a new paper in the journal <a href="https://www.nature.com/articles/s41586-024-07386-0" data-extlink="" target="_blank">Nature</a>. Additional information about this advance is available at <a href="https://www.computationalimaging.org/publications/holographicAR/" data-extlink="" target="_blank">this website</a>, created by the research team.Though only a prototype now, such a technology, they say, could transform fields stretching from gaming and entertainment to training and education – anywhere computed imagery might enhance or inform the wearer’s understanding of the world around them.“One could imagine a surgeon wearing such glasses to plan a delicate or complex surgery or airplane mechanic using them to learn to work on the latest jet engine,” <a href="https://manugopa.github.io/" data-extlink="" target="_blank">Manu Gopakumar</a>, a doctoral student in the Wetzstein-led <a href="https://www.computationalimaging.org/" data-extlink="" target="_blank">Stanford Computational Imaging lab</a> and co-first author of the paper said.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1715819863917-2_20240430_holographic_ar_displays_n6a0965_0.png.webp" style="width: 100%px;" class="fr-fic fr-dib">New holographic augmented reality system that enables more compact 3D displays | Image courtesy of Andrew Brodhead<h2 id="isPasted">Barriers overcome</h2>The new approach is the first to thread a complex maze of engineering requirements that have so far produced either ungainly headsets or less-than-satisfying 3D visual experiences that can leave the wearer visually fatigued, or even a bit nauseous at times.“There is no other augmented reality system out there now with comparable compact form factor or that matches our 3D image quality,” said <a href="https://gunyeal.github.io/" data-extlink="" target="_blank">Gun-Yeal Lee</a>, a postdoctoral researcher in the Stanford Computational Imaging lab and co-first author of the paper.To succeed, the researchers have overcome technical barriers through a combination of AI-enhanced holographic imaging and new nanophotonic device approaches. The first hurdle was that the techniques for displaying augmented reality imagery often require the use of complex optical systems. In these systems, the user does not actually see the real world through the lenses of the headset. Instead, cameras mounted on the exterior of the headset capture the world in real time and combine that imagery with computed imagery. The resulting blended image is then projected to the user’s eye stereoscopically.“The user sees a digitized approximation of the real world with computed imagery overlaid. It’s sort of augmented virtual reality, not true augmented reality,” explained Lee.These systems, Wetzstein explains, are necessarily bulky because they use magnifying lenses between the wearer’s eye and the projection screens that require a minimum distance between the eye, the lenses, and the screens, leading to additional size.“Beyond bulkiness, these limitations can also lead to unsatisfactory perceptual realism and, often, visual discomfort,” said <a href="https://choisuyeon.github.io/" data-extlink="" target="_blank">Suyeon Choi</a>, a doctoral student in the Stanford Computational Imaging lab and co-author of the paper.<h2>Killer app</h2>To produce more visually satisfying 3D images, Wetzstein leapfrogged traditional stereoscopic approaches in favor of holography, a Nobel-winning visual technique developed in the late-1940s. Despite great promise in 3D imaging, more widespread adoption of holography has been limited by an inability to portray accurate 3D depth cues, leading to an underwhelming, sometimes nausea-inducing, visual experience.The Wetzstein team used AI to improve the depth cues in the holographic images. Then, using advances in nanophotonics and waveguide display technologies, the researchers were able to project computed holograms onto the lenses of the glasses without relying on bulky additional optics.A waveguide is constructed by etching nanometer-scale patterns onto the lens surface. Small holographic displays mounted at each temple project the computed imagery through the etched patterns which bounce the light within the lens before it is delivered directly to the viewer’s eye. Looking through the glasses’ lenses, the user sees both the real world and the full-color, 3D computed images displayed on top.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1715819889720-3_20240430_holographic_ar_displays_n6a1169_0.png.webp" style="width: 100%px;" class="fr-fic fr-dib">Using nanophotonic technologies called metasurface optics, the researchers designed and fabricated a novel waveguide design that can relay 3D hologram information of RGB visible light into a single compact device with high transparency. These nanophotonic waveguide samples were fabricated in-house at Stanford Nanofabrication Facility and Stanford Nano Shared Facilities. | Image courtesy of Andrew Brodhead<h2 id="isPasted">Life-like quality</h2>The 3D effect is enhanced because it is created both stereoscopically, in the sense that each eye gets to see a slightly different image as they would in traditional 3D imaging, and holographically.“With holography, you also get the full 3D volume in front of each eye increasing the life-like 3D image quality,” said <a href="https://bchao1.github.io/" data-extlink="" target="_blank">Brian Chao</a>, a doctoral student in the Stanford Computational Imaging lab and also co-author of the paper.The ultimate outcome of the new waveguide display techniques and the improvement in holographic imaging is a true-to-life 3D visual experience that is both visually satisfying to the user without the fatigue that has challenged earlier approaches.“Holographic displays have long been considered the ultimate 3D technique, but it’s never quite achieved that big commercial breakthrough,” Wetzstein said. “Maybe now they have the killer app they’ve been waiting for all these years.”Additional authors are from The University of Hong Kong and NVIDIA. Wetzstein is also member of&nbsp;<a href="http://biox.stanford.edu/" data-extlink="" target="_blank">Stanford Bio-X</a>, the&nbsp;<a href="https://humanperformance.stanford.edu/" data-extlink="" target="_blank">Wu Tsai Human Performance Alliance</a>, and the&nbsp;<a href="https://neuroinstitute.stanford.edu/" data-extlink="" target="_blank">Wu Tsai Neurosciences Institute</a>.

Prototype of the compact augmented reality glasses. Through holography and AI, these glasses can display full-color, 3D moving images over an otherwise direct view of the real world. | Image courtesy of Andrew Brodhead

Combining advances in display technologies, holographic imaging, and artificial intelligence, engineers at Stanford say they have produced a leap forward for augmented reality.

AI and holography bring 3D augmented reality to regular glasses

The <a href="https://wevlv.co/mik-2024" target="_blank" rel="noopener noreferrer">Vision, Robotics &amp; Motion Trade</a> event is an annual event held in Brabanthallen in 's-Hertogenbosch, the Netherlands, that offers a complete overview of solutions for smart automation and digitalisation opportunities. The next two-day event will be held between the 5th and 6th of June, 2024The event aims to provide insights into robotics, vision, and motion implementation in the manufacturing sector, enabling engineers and technical specialists within the MKB/OEM high-tech and manufacturing industry to learn from industry leaders about how to address challenges, such as consumer demands and increasing automation. &nbsp;With over 130 exhibitors, including manufacturers, system integrators, suppliers, and advisors in attendance, this event promises to be more exciting than the previous year.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE1NzYwNzU1Mzk4LTUyOTYxNjE2OTE2XzhjNmE4NTA4ZDZfay5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">This year's event will continue to explore automation in the manufacturing industry. This year's trade show will be expanded to include a new pillar: Digital Innovation. This pillar focuses on digitization opportunities and solutions for companies in the manufacturing industry. “We see that digitalization and automation always go hand in hand, without digitalization you can hardly automate. Therefore, it can hardly exist separately. That makes the combination at the fair very interesting,” says Timo van Leent, manager of the Events Department at Mikrocentrum. “Entrepreneurs can get informed about the latest trends. Moreover, the stories of others are often inspiring.”Exhibitors will demonstrate the latest trends and technology in smart production automation with use cases, and visitors will be able to engage directly with manufacturing companies and hear about cutting-edge technological developments and applications.<h2 dir="ltr">Robotics Demonstrations and AI Applications</h2>One of the most engaging demo areas of the floor plan hosts a variety of robotic systems in full operation. &nbsp;Industrial robots and collaborative robots will be exhibited, performing specific tasks and working with human operators. Experts will showcase the versatility and efficiency of these robotic systems in performing complex operations and executing tasks with speed and precision.Furthermore, the demo areas will have several AI applications that utilize machine learning and computer vision technologies. Attendees can witness intelligent algorithms analyzing data and AI-powered vision systems identifying defects in real time. These demos pinpoint the power of AI in manufacturing and other industries.Industry experts in robotics and AI will assist with suitable solutions and implementations on the exhibition floor. The various speakers will guide with smart and practical digitalization solutions and discuss the potential impact of advancements in AI and robotics on manufacturing industries. The demo areas will have interactive workshops to allow attendees to engage in discussions with authority figures in the field.&nbsp;<h2 dir="ltr">The Significance of this Event to the Manufacturing Industry</h2>The importance of this event lies in its approach to addressing manufacturers' needs through new technological innovations. Manufacturers' problems are multifaceted, from rising consumer demands to product delivery. In today's fast-paced world, where efficiency, quality control, and sustainability are the core of production, manufacturers face an uphill task in adapting and meeting demands.Charlotte de Jong, program manager of Vision, Robotics, and Motion at Mikrocentrum, said more and more manufacturing companies want to automate for various reasons. The shortage in the labor market exemplifies one of these reasons. The shortage means companies have to find ways to work more efficiently. Robotics and AI offer innovative solutions to address this global challenge. Robotics and AI can help in automating repetitive tasks with high precision and speed, while they help to streamline manufacturing processes and minimize waste, thereby maximizing output and reducing operational costs.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE1NzYwNzcyMDQ3LTUyOTYxMDE2MDc3X2YwZjMxMTkyN2Rfay5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">AI also ensures that manufacturers maintain a high level of quality and meet regulatory standards. Leveraging advanced solutions, such as automated inspection systems, can help to identify defects accurately, ensuring only products of the highest quality are manufactured and delivered to consumers. When the risk of defects is reduced, manufacturers ensure quality control and uphold their reputation.Furthermore, robotics and AI can enable greater flexibility. In today's consumer market, manufacturers must adapt quickly to consumer dynamism and market demands. Traditional manufacturing often slacks in responding to rapid changes, leading to inefficiencies and missed opportunities. Robotics and AI, in conjunction with human efforts, will facilitate quick reconfiguration of production lines and offer analytical insights that will help manufacturers adjust to demand fluctuations and optimize production schedules for greater results.Finally, this power-packed event will make clear how these transformative solutions can bring sustainability throughout the production. With more eyes now on environmental impact, manufacturers are under immense pressure to abide by sustainable practices during production. Robotics and AI can help manufacturers achieve sustainability goals by reducing energy consumption and minimizing waste. These advanced solutions can regulate energy usage and repurpose materials to reduce waste and costs.&nbsp;AI will unlock new levels of innovation and productivity, and the manufacturing industry can greatly benefit from this transformative solution. The upcoming Vision, Robotics, and Motion event hopes to create more awareness of the advantages of robotics and AI and how their implementation in the manufacturing sector will increase competitiveness, enhance efficiency, and enable flexibility.&nbsp;<h2 dir="ltr">Digital Innovation: A New Pillar of this Year’s Event</h2>A new addition to this year’s event is digital innovation, which will be one of the key highlight of the trade fair. This new pillar will reflect the increasing importance of digitalization in manufacturing. It focuses on digitalization opportunities and solutions for manufacturing companies.&nbsp;Manager of the Events Department, Timo Van Leent, said digitalization and automation are intertwined. "Without digitalization, you can't automate," he said. That's what makes the upcoming event in the Netherlands interesting. Attendees can learn about the latest trends and discover innovative technologies that will revolutionize manufacturing. From maintenance to logistics, digital innovation will unlock new opportunities for enhanced productivity, enabling more agile and responsive manufacturing lines.Leent reveals that while there's a clamor to adopt digitalization and automation in manufacturing, small manufacturing companies are slightly disadvantaged because they need higher budgets to experiment. However, he states that these companies don't need to follow the latest innovation trends. They can gain insights at the Vision, Robotics, and Motion trade fair into technologies that can help them achieve digitalization and automation at their level.Attending engineers will gain insight into digitalization and automation from exhibitors and have the opportunity to build a network. De Jong states that networking is fundamental at the event. "Visitors get to know many people while having the chance to check out some technologies." In addition to the exhibition and the demos, Manufacturers and engineers will be given an extensive lecture on the use of the software. Learning and discovery are essential aspects of the "Digital Innovation" pillar.<h2 dir="ltr">Embracing Digital Innovation in Other Industries</h2>Beyond manufacturing, the technologies showcased at this year’s event will also influence other industries, such as healthcare and agriculture. In healthcare, AI-powered robotic-assisted surgeries are becoming prevalent in patient care. In agriculture, robotics and AI are enhancing sustainable crop yields.&nbsp;The latest robotics innovations are poised to transform these domains globally, focusing on efficiency and productivity. Attendees can learn of their practical applications and hope for tangible benefits beyond manufacturing.&nbsp;<h2 dir="ltr">Vital Learning Opportunities</h2>Trade fairs like this will serve as a crucial learning platform for students, professionals, entrepreneurs, and even robotics enthusiasts. Attendees will have hands-on learning, equipped with the skills and knowledge required to thrive in this fast-paced, technologically driven world.&nbsp;<h2 dir="ltr" id="isPasted">Preparing for the Future</h2>The innovations showcased at this year’s Vision, Robotics, and Motion event offer a glimpse into the exciting possibilities of automation in manufacturing and beyond. This event will prepare us for the tomorrow we anticipate—a prosperous and sustainable future that will harness the full potential of robotics and AI.&nbsp;What:&nbsp;Vision, Robotics &amp; Motion TradeWhen:&nbsp;5th and 6th of June, 2024Where:&nbsp;'s-Hertogenbosch, the Netherlands,&nbsp;<a href="https://wevlv.co/mik-2024" target="_blank" rel="noopener noreferrer" class="button-main-action">Register to attend for free here!</a>

June 5th-6th, 2024, 's-Hertogenbosch, Netherlands

Vision, Robotics, & Motion Event 2024: The Future of Smart Production, Automation and Digitalization

In this episode, we discuss how a team from Carnegie Mellon University spearheading non-invasive brain computer interface solutions has had a significant breakthrough in improving their accuracy.

Podcast: All The Brain Chip Implant Benefits & None of The Risks

Parents spend roughly 300 hours each year cleaning up after their kids. That’s nearly two weeks that could be otherwise spent doing, well, anything else. Researchers in Carnegie Mellon University’s <a href="https://www.meche.engineering.cmu.edu/index.html" rel="noopener" target="_blank" id="isPasted">Department of Mechanical Engineering</a>, in collaboration with Google DeepMind and the University of Washington, are hopeful that parents will regain tidying time and then some with <a href="https://linchangyi1.github.io/LocoMan/" rel="noopener" target="_blank">LocoMan</a>, a four-legged version of the Jetsons’ beloved maid “Rosey the Robot,” that could lend a hand in the not-so-distant future.<iframe width="640" height="360" src="https://www.youtube.com/embed/d-GLiRPiHIE?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;&nbsp;</iframe>Quadrupedal robots, designed to move on four legs much like a dog, are already found leading search and rescue missions, surveying construction sites, and moving parts around factories. Their ability to navigate complex environments with a low center of gravity and light weight make quadrupedal robots a top contender for household use, too. Until now, the missing piece was the dexterity and versatility skills necessary for them to be efficient in daily life.By installing two custom-designed, lightweight manipulators (think robot arms) onto the front legs of a quadrupedal robot, <a href="https://engineering.cmu.edu/directory/bios/zhao-ding.html" target="_blank">Ding Zhao</a> has created a robot versatile enough to open doors, pour drinks, plug a phone in to charge, and—perhaps best of all—clean up after your on-the-go toddlers.<blockquote>While many researchers work on humanoid robots that look like humans, with potential to directly learn from humans or even replace humans in some tasks, we focus on robots that are complementary to humans to team up with them.<cite>Ding Zhao, Associate Professor, Mechanical Engineering</cite></blockquote>“While many researcher work on humanoid robots that look like humans, with potential to directly learn from humans or even replace humans in some tasks, we focus on robots that are complementary to humans to team up with them,” said Zhao, associate professor of mechanical engineering.Unlike humanoid robots that come at a high cost and hefty weight, Zhao’s manipulators can be added to existing lower-cost quadrupedal robots and are compact, lightweight, and easy to fabricate, making them an accessible technology.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE1NzM1MTQ3ODg3LTA1MDktZW0taG9tZS1oZWxwZXItcm9ib3QucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">The core idea is to leverage the leg as an integral component of the “robotic arm” for executing 6D pose manipulation.“We used four commercially available servos and 3D printed a few other parts to bring them to life,” Changyi Lin, a Ph.D. candidate in the CMU Safe AI Lab and the first author of the paper explained.Zhao’s team introduced the robot to a class of five preschool students at The Cyert Center for Early Education. After explaining to children how robots work and what we use them for, the children had the opportunity to interact with Zhao’s robot. The students were impressed with the robot’s capabilities and didn’t want it to leave their classroom.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE1NzM1MTc3MDc1LTA1MTMtZW0yLWhvbWUtaGVscGVyLXJvYm90LnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Ding Zhao (center), with children from the Cyert Center, interacting with the home-helper robot.The group continues to develop LocoMan to perform tasks autonomously by integrating advanced perception and intelligence planning capabilities into its existing whole-body controller.“We anticipate that robots like LocoMan will be ready to help out at home, or in daycares, within the next three years,” said Zhao. “It is a pleasant job just to imagine that I may no longer need to handle my kids’ messy fun.”This work will be presented at the loco-manipulation workshop and the manipulation skills workshop of the 2024 IEEE International Conference on Robotics and Automation.

New locomotion and manipulation technology from Ding Zhao’s lab will enable four-legged robots to lend a hand in the not-so-distant future.

Manipulation technology makes home-helper robot possible

Imagine a slime-like robot that can seamlessly change its shape to squeeze through narrow spaces, which could be deployed inside the human body to remove an unwanted item.While such a robot does not yet exist outside a laboratory, researchers are working to develop reconfigurable soft robots for applications in health care, wearable devices, and industrial systems.But how can one control a squishy robot that doesn’t have joints, limbs, or fingers that can be manipulated, and instead can drastically alter its entire shape at will? MIT researchers are working to answer that question.They developed a control algorithm that can autonomously learn how to move, stretch, and shape a reconfigurable robot to complete a specific task, even when that task requires the robot to change its morphology multiple times. The team also built a simulator to test control algorithms for deformable soft robots on a series of challenging, shape-changing tasks.<iframe width="640" height="360" src="https://www.youtube.com/embed/IieJM8wYUKo?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Their method completed each of the eight tasks they evaluated while outperforming other algorithms. The technique worked especially well on multifaceted tasks. For instance, in one test, the robot had to reduce its height while growing two tiny legs to squeeze through a narrow pipe, and then un-grow those legs and extend its torso to open the pipe’s lid.While reconfigurable soft robots are still in their infancy, such a technique could someday enable general-purpose robots that can adapt their shapes to accomplish diverse tasks.“When people think about soft robots, they tend to think about robots that are elastic, but return to their original shape. Our robot is like slime and can actually change its morphology. It is very striking that our method worked so well because we are dealing with something very new,” says Boyuan Chen, an electrical engineering and computer science (EECS) graduate student and co-author of a <a href="https://arxiv.org/pdf/2401.13231" target="_blank">paper on this approach</a>.Chen’s co-authors include lead author Suning Huang, an undergraduate student at Tsinghua University in China who completed this work while a visiting student at MIT; Huazhe Xu, an assistant professor at Tsinghua University; and senior author Vincent Sitzmann, an assistant professor of EECS at MIT who leads the Scene Representation Group in the Computer Science and Artificial Intelligence Laboratory. The research will be presented at the International Conference on Learning Representations.<h2>Controlling dynamic motion</h2>Scientists often teach robots to complete tasks using a machine-learning approach known as reinforcement learning, which is a trial-and-error process in which the robot is rewarded for actions that move it closer to a goal.This can be effective when the robot’s moving parts are consistent and well-defined, like a gripper with three fingers. With a robotic gripper, a reinforcement learning algorithm might move one finger slightly, learning by trial and error whether that motion earns it a reward. Then it would move on to the next finger, and so on.But shape-shifting robots, which are controlled by magnetic fields, can dynamically squish, bend, or elongate their entire bodies.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1715580226145-Ditto-gym-1.gif" style="width: 100%px;" class="fr-fic fr-dib">The researchers built a simulator to test control algorithms for deformable soft robots on a series of challenging, shape-changing tasks. Here, a reconfigurable robot learns to elongate and curve its soft body to weave around obstacles and reach a target. Image: Courtesy of the researchers“Such a robot could have thousands of small pieces of muscle to control, so it is very hard to learn in a traditional way,” says Chen.To solve this problem, he and his collaborators had to think about it differently. Rather than moving each tiny muscle individually, their reinforcement learning algorithm begins by learning to control groups of adjacent muscles that work together.Then, after the algorithm has explored the space of possible actions by focusing on groups of muscles, it drills down into finer detail to optimize the policy, or action plan, it has learned. In this way, the control algorithm follows a coarse-to-fine methodology.“Coarse-to-fine means that when you take a random action, that random action is likely to make a difference. The change in the outcome is likely very significant because you coarsely control several muscles at the same time,” Sitzmann says.To enable this, the researchers treat a robot’s action space, or how it can move in a certain area, like an image.Their machine-learning model uses images of the robot’s environment to generate a 2D action space, which includes the robot and the area around it. They simulate robot motion using what is known as the material-point-method, where the action space is covered by points, like image pixels, and overlayed with a grid.The same way nearby pixels in an image are related (like the pixels that form a tree in a photo), they built their algorithm to understand that nearby action points have stronger correlations. Points around the robot’s “shoulder” will move similarly when it changes shape, while points on the robot’s “leg” will also move similarly, but in a different way than those on the “shoulder.”In addition, the researchers use the same machine-learning model to look at the environment and predict the actions the robot should take, which makes it more efficient.<h2>Building a simulator</h2>After developing this approach, the researchers needed a way to test it, so they created a simulation environment called DittoGym.DittoGym features eight tasks that evaluate a reconfigurable robot’s ability to dynamically change shape. In one, the robot must elongate and curve its body so it can weave around obstacles to reach a target point. In another, it must change its shape to mimic letters of the alphabet.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1715580352121-ezgif-1-783e5d8fbe.gif" style="width: 100%px;" class="fr-fic fr-dib">In this simulation, the reconfigurable soft robot, trained using the researchers' control algorithm, must change its shape to mimic objects, like stars, and the letters M-I-T. Image: Courtesy of the researchers“Our task selection in DittoGym follows both generic reinforcement learning benchmark design principles and the specific needs of reconfigurable robots. Each task is designed to represent certain properties that we deem important, such as the capability to navigate through long-horizon explorations, the ability to analyze the environment, and interact with external objects,” Huang says. “We believe they together can give users a comprehensive understanding of the flexibility of reconfigurable robots and the effectiveness of our reinforcement learning scheme.”Their algorithm outperformed baseline methods and was the only technique suitable for completing multistage tasks that required several shape changes.“We have a stronger correlation between action points that are closer to each other, and I think that is key to making this work so well,” says Chen.While it may be many years before shape-shifting robots are deployed in the real world, Chen and his collaborators hope their work inspires other scientists not only to study reconfigurable soft robots but also to think about leveraging 2D action spaces for other complex control problems.

A new machine-learning technique can train and control a reconfigurable soft robot that can dynamically change its shape to complete a task. The researchers, from MIT and elsewhere, also built a simulator that can evaluate control algorithms for shape-shifting soft robots. Image: Courtesy of the researchers; MIT News

A new algorithm learns to squish, bend, or stretch a robot’s entire body to accomplish diverse tasks like avoiding obstacles or retrieving items.

A better way to control shape-shifting soft robots

Researchers from the Karlsruhe Institute of Technology (KIT) and the University of Duisburg Essen have managed to use computer-aided neural networks to precisely identify affective states based on the body language of tennis players during the game. For the first time, a model based on artificial intelligence (AI) was trained with data from real competition scenes. The study, published in the journal Knowledge-Based Systems, shows that AI can evaluate body language and emotions with similar accuracy to a human, but it also points to ethical aspects. ( <a href="https://www.sciencedirect.com/science/article/pii/S0950705124004908?via%3Dihub" target="_blank">DOI: 10.1016/j.knosys.2024.111856</a> )For the study “Detection of emotional states from the expressive behavior of tennis players using convolutional neural networks,” researchers from sports science, software development and computer science at KIT and the University of Duisburg Essen have developed a special AI model. Using computer-based pattern recognition programs, they analyzed video recordings of tennis players during real competitions.&nbsp;<h2>68.9 percent hit rate</h2>“Our model detects affective states with an accuracy of up to 68.9 percent. This is comparable to, and in some cases even superior to, the assessment by human observers as well as to previous automated methods,” explains study leader Professor Darko Jekauc from the Institute for Sport and Sports Science (IfSS) at KIT.&nbsp;An important unique feature of the study is that the project team did not train the AI system using simulated or posed situations, but rather using real scenes. To do this, the researchers recorded video sequences of 15 tennis players in a fixed setting. The focus was on body language when a point was lost or gained. The videos showed players with so-called cues, i.e. with cues such as a lowered head, arms thrown in the air in celebration, a hanging racket or differences in walking speed, which could be used to identify the players' affective state.&nbsp;Fed with this data, the AI learned to assign the body language signals to different affective reactions and to recognize whether a point was won (positive body language) or lost (negative body language). "Training in natural contexts represents a significant advance in recognizing real emotional states and enabling predictions in real-world scenarios," explains Jekauc.<h2>Humans and machines recognize negative emotions better than positive ones</h2>The research not only shows that AI algorithms could outperform human observers in emotion recognition in the future. The evaluation revealed another interesting aspect: both AI and humans are better at recognizing negative emotions. “This may be because they are easier to identify due to their clearer expressions,” explains Jekauc. “Psychological theories suggest that humans are evolutionarily more attuned to perceiving negative emotional expressions. For example, because it is crucial for the social context to defuse conflicts in a timely manner.”<h2>Before use: Clarification of ethical aspects necessary</h2>The study sees numerous areas of application for realistic emotion recognition in sport, from optimizing training methods, team dynamics and improving performance to preventing burnout. In addition, other areas could benefit from reliable early detection of emotional states, such as healthcare, educational institutions, customer service and automotive safety.&nbsp;“Although this technology offers significant benefits in the future, the potential risks associated with it must be taken into account, especially with regard to privacy and data misuse,” emphasizes Jekauc. “Our study was strictly based on existing ethical guidelines and data protection regulations. Even with a view to future applications of such technology in practice, it is essential to clarify ethical and legal questions in advance.”&nbsp;<hr><h3 id="isPasted">Original publication</h3>Darko Jekauc, Diana Burkart, Julian Fritsch, Marc Hesenius, Ole Meyer, Saquib Sarfraz, Rainer Schuhehagen: Recognizing affective states from the expressive behavior of tennis players using convolutional neural networks. Knowledge-Based Systems, Vol. 295, 2024.&nbsp;<a href="https://doi.org/10.1016/j.knosys.2024.111856" target="_blank">DOI: 10.1016/j.knosys.2024.111856</a>

KIT researchers have developed an AI model that can use body language to recognize emotions in tennis players. (Photo: PantherMedia / Wavebreakmedia ltd)

Researchers at KIT have developed a model for emotion analysis that can determine affective states with similar accuracy to a human

AI recognizes emotions in real sports situations

Continual advancements in space technology are reshaping our capacity to manage Earth's affairs with precision and efficiency. Amidst the complexities of urbanization, climate change, and global connectivity, the need for sophisticated infrastructure has become increasingly apparent.&nbsp;Satellites, with advanced functionalities, have emerged as indispensable tools for data acquisition, analysis, and communication, which are crucial for informed decision-making on Earth.With their high-resolution cameras, multispectral sensors, and robust communication capabilities, we can capture and transmit data from a wide range of Earth-related activities, from urban planning and disaster management to agricultural monitoring and environmental protection.By continuously orbiting the Earth, satellites capture detailed images and other sensor data covering the whole planet’s surface. We can use this data to track changes in landscapes, monitor weather patterns, and observe natural disasters like forest fires, floods, and hurricanes in real time.This data is collected and transmitted back to Earth, where it can undergo rigorous analysis. Machine learning algorithms, data analytics, and other advanced computing techniques allow us to interpret the data with ease. For example, predictive modeling is utilized to forecast weather changes or track the migration patterns of endangered species. The ability to process real-time data enables quick decision-making, which is especially crucial during responses to natural disasters and emergencies.&nbsp;Communication is another critical functionality of satellites. They ensure communication lines are able to remain open, even in the most remote parts of the world. This facilitates global connectivity, allowing data, voice, and video communication to be transmitted across diverse geographical landscapes.<h1 dir="ltr">Maintenance in space</h1>On Earth, if something breaks down in a factory or a power plant, we don’t build a replacement. Instead, we send a repair team with spare parts, and even better yet, we monitor the components' wear and replace them before they break down. This preventive maintenance approach minimizes downtime and reduces the costs associated with sudden breakdowns.This methodology is critical in industries where reliability and uptime are essential. By continuously monitoring equipment health through sensors and predictive analytics, we can identify potential problems before they lead to operational failures. However, this approach faces unique challenges in space, where sending a repair team is not feasible due to the vast distances and extreme conditions.The transition from Earth-based maintenance to space requires innovative solutions to overcome these challenges. The harsh space environment, including vacuum conditions, microgravity, and cosmic radiation, makes traditional maintenance methods impractical. Additionally, the delay in communication between Earth and spacecraft further complicates real-time interventions, necessitating a more autonomous approach to maintenance and repair operations in space.<a href="https://www.ant61.com/" target="_blank" rel="noopener noreferrer">ANT61&nbsp;</a>develops autonomous robots that enable the maintenance of satellites without the need to put human lives at risk. ANT61 is providing solutions for both: their Beacon product allows satellite operators to understand what went wrong with their satellites and restore them back to operation.For larger and more expensive satellites, they are building robots that will dock, refuel, and, in the future, refurbish satellites, prolonging their useful life. At the core of these robots lies ANT61 BrainTM, the innovative devices that combine machine vision and decision-making technology, enabling the autonomy of these maintenance robots. Autonomy is very important as, due to the speed of light limitations, it won’t be possible to control every movement of these robots remotely from Earth.<h3 dir="ltr">BrainChip AkidaTM&nbsp;: A revolutionary neuromorphic processor for edge AI applications</h3>The first generation of the ANT61 Brain uses the <a href="https://brainchip.com/akida-neural-processor-soc/" target="_blank" rel="noopener noreferrer">BrainChip AkidaTM</a> chip for power-efficient AI and is currently on board the Optimus-1 satellite, which was deployed recently by the SpaceX Transporter-10 mission. ANT61 will test the ANT61 Brain later this year and perform training and evaluation of various neural networks that we will use for future in-orbit servicing technology demonstrations.The BrainChip Akida™ chip is a groundbreaking advancement in neuromorphic computing, designed to mimic the neural architecture of the human brain. This technology stands out for its ability to process data directly at the point of acquisition, greatly enhancing efficiency by reducing the reliance on distant cloud-based systems for data processing.Akida's unique capability lies in its event-based neural processor, which processes data only when needed, using data sparsity to significantly reduce the number of operations required. This leads to lower power consumption and heat dissipation, making it highly suitable for the demanding conditions of space. Each Akida node consists of four Neural Network Layer Engines (NPEs), which can be configured for various functions, such as convolutional layers or fully connected layers, and support incremental learning and high-speed inference. This configuration flexibility allows the chip to efficiently handle a wide range of AI tasks without needing constant connectivity or high power draw.Akida's scalable architecture enables the chip to support up to 256 nodes in a mesh network, boosting computational power while maintaining a small footprint. This scalability is crucial for space applications where efficiency and performance are vital. The entire neural network can be embedded directly within the Akida fabric, eliminating the need for frequent data transfers that consume significant power and reduce system responsiveness.Another key feature of the Akida chip is its on-chip learning capability, which allows it to adapt and improve its performance over time without external intervention. This feature is particularly beneficial in space, where conditions can change rapidly and autonomy is crucial. By processing and learning from data locally, Akida enhances the autonomy of satellite maintenance robots like those developed by ANT61, enabling them to perform complex tasks with minimal latency and enhanced reliability.The ANT61 team chose to partner with BrainChip because they believe that neuromorphic technology will bring the same exponential improvement in AI as 20 years ago. The shift from CPU to GPU opened doors to deep neural network applications, which are at the core of all AI technologies today. Neuromorphic technology is also perfect for space: the lower power consumption means less heat dissipation, and we can get up to five times more computational power for the same electrical power budget.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1715339791069-Akida_1000_chip-1.jpg" style="width: 100%px;" class="fr-fic fr-dib">Akida AKD1000 is a reference chip implemented with TSMC at 28nm<h1 dir="ltr">The ANT61 vision for the future</h1>Humanity’s expansion to the cosmos requires infrastructure that can only be built by robots. With its in-orbit servicing experience, ANT61 aims to become the main supplier of the robotic workforce for Moon and Mars installations. This will enable companies from Earth to project their ambition to space, providing valuable services and resources for future off-world factories and cities.Neuromorphic computing, exemplified by technologies like the BrainChip Akida™ chip, is set to play a pivotal role in this spacefaring era. This form of computing brings several advantages crucial for space applications, such as low power consumption, high efficiency, and the ability to perform complex computations needed for autonomy. By mimicking the human brain, neuromorphic processors can process large volumes of sensor data efficiently, making real-time decisions without the need to send data back to Earth for processing.The integration of neuromorphic computing into space robotics promises to significantly improve these machines' capabilities. Robots equipped with advanced AI processors can better handle the unpredictable nature of space environments, adapting to new challenges autonomously and performing tasks ranging from construction to maintenance and repair. This capability ensures that space habitats are built efficiently and can be sustained with minimal human oversight.ANT61's vision of utilizing these technologies to support human expansion into space reflects a forward-thinking approach that will not only facilitate the construction of off-world establishments but also ensure their long-term sustainability and success.&nbsp;

Remote satellite maintenance is crucial to critical data acquisition and global communication


Advancing Earth Management Through Neuromorphic Computing Powered Satellites

<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1715299294035-5712x4284.jpg" style="width: 100%px;" class="fr-fic fr-dib">The robot spontaneously changed its gait to cross a terrain with gaps. 2024 EPFL/BioRob-CC-BY-SA 4.0With the help of a form of machine learning called deep reinforcement learning (DRL), the EPFL robot notably learned to transition from trotting to pronking – a leaping, arch-backed gait used by animals like springbok and gazelles – to navigate a challenging terrain with gaps ranging from 14-30cm. The study, led by the&nbsp;<a href="https://www.epfl.ch/labs/biorob/" target="_blank">BioRobotics Laboratory</a> in EPFL’s School of Engineering, offers new insights into why and how such gait transitions occur in animals. “Previous research has introduced energy efficiency and musculoskeletal injury avoidance as the two main explanations for gait transitions. More recently, biologists have argued that stability on flat terrain could be more important. But animal and robotic experiments have shown that these hypotheses are not always valid, especially on uneven ground,” says PhD student Milad Shafiee, first author on a paper published in <a href="https://www.nature.com/articles/s41467-024-47443-w" target="_blank" rel="noopener">Nature Communications</a>.Shafiee and co-authors Guillaume Bellegarda and BioRobotics Lab head Auke Ijspeert were therefore interested in a new hypothesis for why gait transitions occur: viability, or fall avoidance. To test this hypothesis, they used DRL to train a quadruped robot to cross various terrains. On flat terrain, they found that different gaits showed different levels of robustness against random pushes, and that the robot switched from a walk to a trot to maintain viability, just as quadruped animals do when they accelerate. And when confronted with successive gaps in the experimental surface, the robot spontaneously switched from trotting to pronking to avoid falls. Moreover, viability was the only factor that was improved by such gait transitions.“We showed that on flat terrain and challenging discrete terrain, viability leads to the emergence of gait transitions, but that energy efficiency is not necessarily improved,” Shafiee explains. “It seems that energy efficiency, which was previously thought to be a driver of such transitions, may be more of a consequence. When an animal is navigating challenging terrain, it’s likely that its first priority is not falling, followed by energy efficiency.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1715299423668-ezgif-3-e8b7f9ed78.jpg" style="width: 100%px;" class="fr-fic fr-dib">Guillaume Bellegarda, Milad Shafiee and Auke Ijspeert. 2024 EPFL/Jamani Caillet - CC-BY-SA 4.0<h2>A bio-inspired learning architecture</h2>To model locomotion control in their robot, the researchers considered the three interacting elements that drive animal movement: the brain, the spinal cord, and sensory feedback from the body. They used DRL to train a neural network to imitate the spinal cord’s transmission of brain signals to the body as the robot crossed an experimental terrain. Then, the team assigned different weights to three possible learning goals: energy efficiency, force reduction, and viability. A series of computer simulations revealed that of these three goals, viability was the only one that prompted the robot to automatically – without instruction from the scientists – change its gait.The team emphasizes that these observations represent the first learning-based locomotion framework in which gait transitions emerge spontaneously during the learning process, as well as the most dynamic crossing of such large consecutive gaps for a quadrupedal robot.“Our bio-inspired learning architecture demonstrated state-of-the-art quadruped robot agility on the challenging terrain,” Shafiee says.The researchers aim to expand on their work with additional experiments that place different types of robots in a wider variety of challenging environments. In addition to further elucidating animal locomotion, they hope that ultimately, their work will enable the more widespread use of robots for biological research, reducing reliance on animal models and the associated ethics concerns.<hr>ReferencesShafiee, M., Bellegarda, G. &amp; Ijspeert, A. Viability leads to the emergence of gait transitions in learning agile quadrupedal locomotion on challenging terrains. Nat Commun 15, 3073 (2024). <a href="https://doi.org/10.1038/s41467-024-47443-w" target="_blank" rel="noopener">https://doi.org/10.1038/s41467-024-47443-w</a>

Guillaume Bellegarda, Milad Shafiee and Auke Ijspeert. 2024 EPFL/Jamani Caillet - CC-BY-SA 4.0

A four-legged robot trained with machine learning by EPFL researchers has learned to avoid falls by spontaneously switching between walking, trotting, and pronking – a milestone for roboticists as well as biologists interested in animal locomotion.

Trotting robots reveal emergence of animal gait transitions

<h1 dir="ltr" id="isPasted">Introduction</h1>In an era where rapid technological advancements continuously reshape how industries operate, edge artificial intelligence (AI) has emerged as a critical driver of innovation and efficiency. Edge AI brings computation and data storage closer to the locations where it's needed, accelerating response times while reducing bandwidth use. This technology is particularly transformative in industrial and commercial applications, where real-time data processing can significantly enhance operational efficiency and decision-making.Enter NVIDIA Train Adapt Optimize (TAO) toolkit—an open-source suite designed to simplify and accelerate the creation of high-quality, customized, and enterprise-ready AI models. This toolkit democratizes access to state-of-the-art AI capabilities and streamlines the development process, enabling rapid deployment across a range of devices from cloud infrastructures to edge units.&nbsp;When integrated with Edge Impulse, a leading development platform for edge AI, NVIDIA TAO becomes part of a powerful ecosystem that enhances the capabilities of developers and engineers to bring AI models to production faster. NVIDIA TAO and Edge Impulse represent a formidable combination, poised to accelerate the adoption of edge AI technologies across various sectors by offering tools that reduce complexity and enhance productivity.This article explores how the NVIDIA TAO toolkit, along with Edge Impulse, facilitates the development and deployment of AI models on edge devices, illustrating the tangible benefits and real-world applications of this integration. By harnessing these advanced tools, developers can speed up the AI development lifecycle and achieve greater scalability in their AI projects.<iframe width="540" height="600" src="https://c8056756.sibforms.com/serve/MUIFAMyA6BGksOUPEokHuBnEmZRHhYgJ4RXb4BV0gvzN98ymNIvQdTcBFQnRVfviT6yZSFW_R2g1hENBu7p7Afnyv9Kt0unAAbfFU5mMQhBwI-8hQnKf2r1mmRKhUgDLmtk8bpVSZ7kZGpleCPqtYRv8_kkvODL7F4neGLNGgdpFDm9pPzoYpwbQkc0BU5GVOFoq8CSmD0DDvDjG" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> <h2 dir="ltr">The NVIDIA TAO Toolkit</h2>NVIDIA TAO Toolkit is a great tool for developers aiming to harness the power of AI without the complexity typically associated with such endeavors. As an open-source solution, it democratizes advanced technology, allowing a broad range of developers—from startups to large enterprises—to create customized, high-performance AI models that are both scalable and adaptable.<h3 dir="ltr">Overview of NVIDIA TAO</h3>At its core, the NVIDIA TAO Toolkit facilitates AI model development by leveraging NVIDIA's extensive expertise in AI and deep learning. Designed to streamline the process of model training and optimization, NVIDIA TAO empowers developers to build sophisticated AI models without deep specialization in AI technologies. Its open-source nature ensures that it is continually refined by contributions from the global development community, enhancing its capabilities and accessibility.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE1MjMyMTA3NTEyLWltYWdlMi5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib" alt="nvidia-tao-workflow">Fig. 1: Working with Nvidia TAO toolkit. Source:&nbsp;<a href="https://developer.nvidia.com/tao-toolkit-usecases-whitepaper/1-introduction" target="_blank" rel="noopener noreferrer">Nvidia</a><h3 dir="ltr">The Power of Transfer Learning</h3>One of the most significant features of the NVIDIA TAO Toolkit is its utilization of transfer learning. This technique revolutionizes how models are built by allowing developers to start with an existing model—pre-trained by NVIDIA or third-party sources on vast and diverse data sets—and customize it to their specific needs with relatively little data. This method drastically reduces the time and resources required for model training and enhances the model’s accuracy and performance from the outset.Transfer learning mimics human learning dynamics, where knowledge gained in one context is repurposed for another. For AI development, this means a developer can take a model that recognizes general objects and retrain it to recognize specialized equipment or scenarios specific to their industry. This flexibility opens up a plethora of applications, from enhancing manufacturing processes with visual inspection tools to improving safety protocols through real-time anomaly detection.<h2 dir="ltr"><iframe width="640" height="360" src="https://www.youtube.com/embed/kErfKDZ50To?&amp;ab_channel=NVIDIADeveloper&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;</iframe>Key Features and Benefits of NVIDIA TAO</h2>The NVIDIA TAO Toolkit offers a suite of capabilities that streamline the development process for AI models. Here, we explore how its features cater to efficiency, accuracy, customization, and deployment flexibility.<h3 dir="ltr">Efficient Model Training</h3>The NVIDIA TAO Toolkit simplifies the model training process significantly. Developers can leverage TAO’s AutoML capability, which automates much of the model tuning. This not only speeds up the model development cycle but also makes it more accessible to non-specialists. The AutoML feature efficiently navigates through the complexities of model tuning, allowing developers to focus more on application development rather than the nuances of model configuration.<h3 dir="ltr">High Accuracy and Customization</h3>NVIDIA is renowned for its high-performance computing and deep learning technologies, and TAO is no exception. It enables the creation of highly accurate AI models tailored to specific needs using state-of-the-art vision transformers and NVIDIA's pre-trained models. These capabilities allow for the development of bespoke AI solutions that are finely tuned to the unique requirements of different use cases, from healthcare diagnostics to automated surveillance systems.<h3 dir="ltr">Optimization for Inference and Deployment Flexibility</h3>Once a model is trained, ensuring it performs efficiently on the intended hardware is crucial. NVIDIA TAO facilitates this by enabling significant optimizations for inference, enhancing model accuracy and performance across GPUs, CPUs, and cloud servers. Additionally, Edge Impulse extends this adaptability to various deployment platforms, including microcontrollers and various other edge devices, ensuring that models integrate smoothly with existing infrastructures.<h3 dir="ltr">AI-Assisted Data Annotations</h3>Preparing data for training AI models is often labor-intensive. NVIDIA TAO eases this burden with AI-assisted data annotation capabilities. This feature accelerates the preparation of training data, making it faster and less costly.<h2 dir="ltr">Integrating NVIDIA TAO with Edge Impulse</h2>The collaboration between NVIDIA TAO and Edge Impulse presents a synergistic approach to edge AI development, significantly reducing time-to-market while increasing the scalability and efficiency of AI solutions. This integration facilitates a streamlined workflow from model creation to deployment, addressing key challenges in edge AI development.<h3 dir="ltr">Enhancing Edge AI Development</h3>By incorporating NVIDIA TAO into the Edge Impulse development platform, developers gain access to a robust set of tools designed to expedite the creation of efficient models. This integration leverages the strengths of both platforms—TAO’s powerful model training and Edge Impulse’s sophisticated deployment capabilities. Together, they simplify the development cycle, enabling developers to rapidly prototype, iterate, and deploy AI models that are both robust and scalable.<h3 dir="ltr">Comprehensive Edge AI Solution</h3>The combined power of NVIDIA TAO and Edge Impulse provides a comprehensive solution for edge AI. From collecting data on edge devices to training and validating models and finally optimizing libraries to run efficiently on target hardware, this integration covers all bases. It supports a wide range of devices, from low-power microcontrollers to more capable GPUs, ensuring that developers can deploy their AI solutions across diverse modern IoT and edge computing environments.Developers can tailor their solutions to specific hardware and performance needs, optimizing resource usage and operational efficiency. Additionally, the combined ecosystem supports a variety of silicon and sensors, making it an all-in-one solution for edge AI applications.<h2 dir="ltr">Practical Applications and Success Stories</h2>The integration of NVIDIA TAO with Edge Impulse offers a powerful platform for practical applications in various sectors. Here, we explore some specific use cases that demonstrate the effectiveness of this technology.PCB Classification:&nbsp;In electronics manufacturing, ensuring that all components are correctly placed and functional on a PCB is crucial. A minor error can lead to a malfunctioning device, leading to increased costs and reduced consumer trust. The PCBClassification model, developed using NVIDIA TAO, addresses this challenge effectively.<a href="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pcb_classification" target="_blank">[1]</a>This model utilizes an advanced AI architecture known as global context vision transformer (GCViT), initially trained on a comprehensive image dataset (NVImageNet). The model's training involves transfer learning, where a model developed for one task is partially retrained to perform another. In this case, the base model, proficient in general image recognition, is fine-tuned to specifically identify missing or defective components in PCB images.The process is straightforward yet powerful: images of PCBs are fed into the model, which has been trained to recognize various components like resistors, capacitors, and inductors. The model assesses each component's presence or absence and checks for defects. This automated inspection helps quickly spot errors during the manufacturing process, reducing the likelihood of defective products reaching the market.The accuracy of this model is notable, achieving near-perfect precision in tests. This high level of performance is crucial for industries where the cost of errors can be exceptionally high, both financially and in terms of brand reputation.Reducing Baggage Delays at Airports:&nbsp;The case of Trifork using NVIDIA TAO for baggage tracking at airports highlights the transformative potential of these technologies in enhancing operational efficiency.<a href="https://resources.nvidia.com/en-us-metropolis-software-success-stories/trifork" target="_blank">[2]</a> By deploying vision AI solutions, Trifork has been able to track millions of pieces of luggage, significantly reducing baggage delays and lost baggage incidents. This application not only improves the passenger experience but also optimizes the workflow and resource allocation at airports.Industrial Defect Detection:&nbsp;Another compelling application is in the field of industrial defect detection.<a href="https://developer.nvidia.com/blog/transforming-industrial-defect-detection-with-nvidia-tao-and-vision-ai-models/" target="_blank">[3]</a> Using NVIDIA TAO, developers have created models that can identify defects in various industrial products with high precision. This application is particularly beneficial for quality control in manufacturing, where detecting and addressing defects early in the production process can lead to substantial cost savings and maintain high-quality standards.Retail Recognition:&nbsp;In the retail sector, the application of NVIDIA TAO for retail recognition tasks illustrates how edge AI can transform inventory management. The models could help with identifying a vast array of retail Stock Keeping Units (SKUs), facilitating better stock management, customer service, and logistical efficiency. By automating the recognition process, retailers can ensure more accurate inventory tracking and faster response times, leading to enhanced operational efficiency and customer satisfaction.These practical applications and success stories demonstrate the broad utility and effectiveness of NVIDIA TAO when integrated with tools like Edge Impulse. The technology not only drives innovation and efficiency in existing processes but also opens up new possibilities for smart manufacturing, retail management, and beyond.Further reading:&nbsp;<a href="https://www.wevolver.com/article/a-practical-guide-to-understanding-sensor-fusion-with-edge-impulse-building-a-smart-running-jacket" target="_blank" id="isPasted">A Practical Guide to Understanding Sensor Fusion with Edge Impulse: Building a Smart Running Jacket</a><h2 dir="ltr">Getting Started and Resources</h2>To begin utilizing NVIDIA TAO and Edge Impulse to develop and deploy AI models on edge devices, it's essential to engage with a variety of resources and steps designed to streamline this process.Firstly, a thorough exploration of the available documentation is crucial. Both <a href="https://docs.edgeimpulse.com/docs/edge-impulse-studio/learning-blocks/nvidia-tao" target="_blank">Edge Impulse</a> and <a href="https://developer.nvidia.com/tao-toolkit" target="_blank">NVIDIA</a> provide comprehensive guides and tutorials on their respective websites. These documents are invaluable for understanding how to effectively integrate NVIDIA TAO models into Edge Impulse projects.Additionally, Edge Impulse and Nvidia provide YouTube videos that detail the application and integration of NVIDIA TAO. These resources complement the written documentation and provide a medium through which users can understand how to work with these technologies.<iframe width="640" height="360" src="https://www.youtube.com/embed/bwquofcF94g?&amp;ab_channel=EdgeImpulse&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Support doesn’t end with documentation and tutorials. NVIDIA and Edge Impulse have active forums like - <a href="https://forum.edgeimpulse.com/" target="_blank">Edge Impulse Forums</a> and <a href="https://forums.developer.nvidia.com/c/accelerated-computing/intelligent-video-analytics/tao-toolkit/17" target="_blank">NVIDIA Developer Forums</a>, where users can engage with a community of like-minded individuals. These forums are a great place to seek advice, ask questions, and share experiences related to edge AI development.By leveraging these resources, developers can effectively navigate the initial learning curve and start applying NVIDIA TAO and Edge Impulse to real-world projects, gradually building up to more complex applications as familiarity with the tools increases.Further reading:&nbsp;<a href="https://www.wevolver.com/article/building-high-performance-edge-ai-solutions-with-nvidia-jetson-and-edge-impulse" target="_blank" rel="noopener noreferrer">Building High-Performance Edge AI Solutions with Nvidia Jetson and Edge Impulse</a><h2 dir="ltr">Conclusion</h2>The integration of NVIDIA TAO and Edge Impulse brings in a new era of edge AI development, offering developers powerful tools to enhance the efficiency and effectiveness of their AI models. This combination brings together NVIDIA's advanced AI modeling capabilities with Edge Impulse's robust customization and deployment framework, enabling rapid development, optimization, and deployment of AI applications across a variety of platforms and devices.For developers and companies venturing into edge AI, the synergy between NVIDIA TAO and Edge Impulse provides a competitive edge. By simplifying complex processes and reducing the time-to-market, this integration democratizes advanced AI technologies and empowers a broader range of users to innovate and solve real-world problems.<iframe width="540" height="600" src="https://c8056756.sibforms.com/serve/MUIFAMyA6BGksOUPEokHuBnEmZRHhYgJ4RXb4BV0gvzN98ymNIvQdTcBFQnRVfviT6yZSFW_R2g1hENBu7p7Afnyv9Kt0unAAbfFU5mMQhBwI-8hQnKf2r1mmRKhUgDLmtk8bpVSZ7kZGpleCPqtYRv8_kkvODL7F4neGLNGgdpFDm9pPzoYpwbQkc0BU5GVOFoq8CSmD0DDvDjG" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> <h2 dir="ltr">References</h2>[1] PCBClassification Model Card, Nvidia, 17 Oct 2023, [Online], Available from: <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pcb_classification" target="_blank">https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pcb_classification</a>[2] Trifork - Reducing baggage delay at airports with vision AI, Nvidia, [Online], Available from: <a href="https://resources.nvidia.com/en-us-metropolis-software-success-stories/trifork" target="_blank">https://resources.nvidia.com/en-us-metropolis-software-success-stories/trifork</a>[3] Nirmal Kumar Juluru, Zaid Pervaiz Bhat and Samuel Ochoa, ‘Transforming Industrial Defect Detection with NVIDIA TAO and Vision AI Models’, Nvidia, [Online], Available from: <a href="https://developer.nvidia.com/blog/transforming-industrial-defect-detection-with-nvidia-tao-and-vision-ai-models/" target="_blank">https://developer.nvidia.com/blog/transforming-industrial-defect-detection-with-nvidia-tao-and-vision-ai-models/</a>

NVIDIA TAO and Edge Impulse integrate to streamline the development and deployment of edge AI models, empowering developers to quickly build and optimize AI applications for diverse hardware environments.

Accelerating Edge AI Model Development Using Nvidia TAO and Edge Impulse

<h2 dir="ltr" id="isPasted">Join Us for Our Upcoming Webinar: Powering Edge and Endpoint Applications: A deep dive into Reality AI Tools® from Renesas</h2>Mohammed Dogar, Vice President, Head of Global Business Development &amp; Ecosystem, and Jeff Sieracki, Director of AI COE, will sit with tech writer Samir Jaber to unpack the <a href="https://www.renesas.com/us/en/products/microcontrollers-microprocessors/reality-ai#featured_products" target="_blank" rel="noopener noreferrer">Reality AI Tools</a> suite and discuss their impact on Edge AI applications.Why Attend?<ul><li dir="ltr">Learn:&nbsp;Understand the primary functions and benefits of using Reality AI Tools®</li><li dir="ltr">Connect: Hear from Mohammed Dogar, Vice President, Head of Global Business Development &amp; Ecosystem, and Jeff Sieracki, Director of AI COE</li><li dir="ltr">Engage: Opportunities to ask questions and interact with our experts</li></ul>Event Details:<ul><li dir="ltr">Topic: Powering Edge and Endpoint Applications: A deep dive into Reality AI Tools® from Renesas</li><li dir="ltr">Date:&nbsp;May 28th 2024</li><li dir="ltr">Time:&nbsp;5pm CET (add others)</li><li dir="ltr">Location:&nbsp;Online (via Zoom). Registrants will be sent the webinar video after the event.&nbsp;</li></ul><a href="https://us02web.zoom.us/webinar/register/4017151721849/WN_ahhKxtJ0TuWs-Klj5BfGgg" target="_blank" rel="noopener noreferrer" class="button-main-action">Register for free here</a><h2>Webinar speakers</h2><h3>Mo Dogar</h3>Mo Dogar is head of Global Business Development and Technology Ecosystem at Renesas Electronics, responsible for promotion and business expansion of the complete Microcontroller portfolio and other key products. He is instrumental in driving global business growth and alignment of marketing, sales and development strategies particularly in the areas of IoT, e-AI, Security, smart embedded electronics and communications.&nbsp;In addition, Mr. Dogar helps provide the vision and thought leadership behind product and solution development smart society and the evolving IoT economy. With extensive experience as a keynote speaker and thought leader in the electronics industry, recent speaking engagements include keynote addresses at CES, Arm Techcon, Embedded World, Industrial Open Day and a number of other AIoT and electronics events globally. He studied Electronics &amp; Information Engineering at the University of Huddersfield, UK and received Project Management qualifications with an MBA from the Open University in the UK and has worked for a number of industrial and technology companies including Rockwell Automation. In 2001 he joined NEC and subsequently Renesas and has worked in various roles including Engineering, Applications, Marketing and Business Development. &nbsp;<h2>Dr. Jeff Sieracki</h2>Dr. Jeff Sieracki is an extremely applied mathematician who has focused his career on the intersection of modern signal processing and machine learning. Former co-founder and CTO of Reality AI, he joined Renesas as part of an acquisition in 2022 and is now head of AI Engineering at the Renesas AIoT Center of Excellence. Jeff’s work spans signal and sensing problems in commercial industry, medical devices, and DoD applications. He has started, built, and exited three companies, one in each of those application areas. He has been awarded more than 50 technology patents. Jeff holds a PhD in Applied Mathematics and Scientific Computing and was a founding member of the Norbert Weiner Center for Harmonic Analysis and Applications at the University of Maryland, College Park. He holds a Masters in Applied Mathematics and a BS in Physics from the John Hopkins University.<h2>Samir Jaber</h2>Samir Jaber is an editor, writer, and industry expert on topics of technology, science, and engineering. He is the editor-in-chief of the 2024 State of Edge AI and 2023 Edge AI Technology reports with Wevolver. Samir is the Chief Editor and Founder of Wryters, a content marketing and consulting agency. He has comprehensive experience working with Fortune 500 companies and industry leaders as a writer, editor, content manager, and consultant. He is an online content specialist with an academic background in mechanical engineering, nanotechnology, and scientific research. Samir is also a featured author in 30+ industrial magazines with a focus on Artificial Intelligence (AI), the Internet of Things (IoT), 3D printing, Autonomous Vehicles (AV), nanotechnology, materials science, and sustainability. His experience includes award-winning engineering research and patented engineering design in the fields of nanofabrication and microfluidics.This event is a must-attend for all engineers interested in the tools enabling AI at the edge. If you can't make it on the day, all registrants will be sent the webinar recording.&nbsp;&nbsp;<a href="http://Samir Jaber is an editor, writer, and industry expert on topics of technology, science, and engineering. He is the editor-in-chief of the 2024 State of Edge AI and 2023 Edge AI Technology reports with Wevolver. Samir is the Chief Editor and Founder of Wryters, a content marketing and consulting agency. He has comprehensive experience working with Fortune 500 companies and industry leaders as a writer, editor, content manager, and consultant. He is an online content specialist with an academic background in mechanical engineering, nanotechnology, and scientific research. Samir is also a featured author in 30+ industrial magazines with a focus on Artificial Intelligence (AI), the Internet of Things (IoT), 3D printing, Autonomous Vehicles (AV), nanotechnology, materials science, and sustainability. His experience includes award-winning engineering research and patented engineering design in the fields of nanofabrication and microfluidics." target="_blank" rel="noopener noreferrer" class="button-main-action">Register for free now</a>

Credit: Evan Stoddard, Advanced Applications Group Embedded Software Engineer at Avnet

Mohammed Dogar, Vice President, Head of Global Business Development & Ecosystem, and Jeff Sieracki, Director of AI COE, will sit with tech writer Samir Jaber to unpack the Reality AI Tools suite and discuss their impact on Edge AI applications.

Webinar: Powering Edge and Endpoint Applications - A deep dive into Reality AI Tools® from Renesas

Space exploration has captivated humanity for decades with its promise of adventure and scientific discovery. But venturing beyond our atmosphere comes with immense challenges. This is where a new wave of technology is poised to revolutionize the space industry: artificial intelligence (AI) and robotics.&nbsp;This episode of the BrainChip podcast is a conversation between BrainChip CTO Dr Tony Lewis and Mikhail Asavkin, founder of Australian space tech company ANT61. They talk about how AI and intelligent robots are no longer science fiction but the key to unlocking the future of space exploration.&nbsp;<h2 dir="ltr">Reusable Rockets and the Democratization of Space</h2>Over the past decade, we have witnessed a remarkable transformation propelled by innovation and the rise of private industry in the space sector. The ability to recover and relaunch rockets has dramatically reduced the cost of space access, opening the door for a new era of private space exploration companies.&nbsp;This democratization of space has spurred innovation and ignited a global race to push the boundaries of what's possible. Mikhail Asavkin, drawing from his extensive experience in space engineering, sheds light on this evolutionary journey and explains how companies like ANT61 are now able to focus their resources on cutting-edge technologies like AI and robotics, thanks to the more affordable launch options provided by reusable rockets.&nbsp;This shift has fundamentally changed the landscape of space exploration, making it a more accessible and dynamic field.<h2 dir="ltr">The Impact of AI on Space Exploration</h2>Mikhail Asavkin and Dr Tony Lewis also discuss the transformative role of AI in space exploration. Here's why AI is the silent hero propelling us further into the cosmos:<ul><li dir="ltr">Due to the speed of light, communicating with robots on the Moon or Mars in real time is impossible. This is where AI steps in. By enabling robots to make autonomous decisions based on sensor data and their surroundings, AI eliminates the need for constant human intervention from Earth.</li><li dir="ltr">Space exploration demands more than just following pre-programmed instructions. Robots must be able to perceive their environment, identify objects, and navigate complex situations. Advancements in AI, particularly image recognition and deep learning, allow robots to "see" and understand their surroundings, enabling them to perform intricate tasks and make critical decisions independently.</li></ul><h2 dir="ltr">Neuromorphic Computing in Space</h2>The limitations of traditional AI systems, particularly their high power consumption, pose a significant challenge for space exploration. Here's where neuromorphic computing enables new ways of exploring space.&nbsp;Inspired by the structure and function of the human brain, neuromorphic chips are designed to process information similarly. This translates to a dramatic reduction in power consumption compared to traditional AI hardware like GPUs (Graphics Processing Units). For spacefaring robots, this is a game-changer.Neuromorphic technology offers a multitude of benefits for space AI:<ul><li dir="ltr">Neuromorphic chips enable this dream by requiring significantly less power to perform complex tasks like object detection and real-time decision-making.</li><li dir="ltr">Neuromorphic chips excel at real-time processing, allowing robots to make critical decisions without delay. This is essential for tasks like navigating hazardous terrain or performing delicate scientific operations.</li></ul><h2 dir="ltr">Training AI on the Go</h2>Traditionally, training AI models involves massive datasets downloaded from Earth and meticulously labeled by humans – a time-consuming and impractical process in the vast expanse of space.Neuromorphic computers, with their ability to learn and adapt in real time, make this possible. The AI model can continuously improve its capabilities and decision-making abilities by processing sensor data and surroundings. By training on-site, the AI model can adapt to specific situations and environments, making its tasks more robust and efficient.However, the podcast also discusses a significant challenge that remains: ensuring the safety and well-being of human explorers venturing into the unknown. This necessitates further advancements in AI, particularly in areas like risk assessment, anomaly detection, and real-time decision-making.<h2 dir="ltr">Conclusion: New Era in Space Exploration</h2>From enabling autonomous robots to construct lunar habitats to empowering AI models to learn and adapt on the fly, these technologies are revolutionizing how we venture beyond our planet. As we look towards a future where humanity establishes a presence on other celestial bodies, AI and robotics will be our indispensable companions.Listen to the full podcast <a href="https://www.youtube.com/watch?v=yyVWVjYM-nQ&t=3s" target="_blank" rel="noopener noreferrer">here.</a>

In this episode, ANT61 Founder and CEO Mikhail Asavkin joins BrainChip CTO Dr. Tony Lewis to discuss the evolution of Space Technology and its impact on daily life. 

BrainChip Podcast: Robotics in Space discussion with ANT61 Founder Mikhail Asavkin

The architectures of embedded systems continue to evolve as newer technologies become available to design engineers. What used to be a single central processing unit surrounded by interface and logic circuitry is now a multiprocessor multi-core design with advanced integrated peripheral functionality—many with their own dedicated microcontrollers and resources to contend with.The once clearly delineated computer edge is now a much vaster and more distributed architecture. Computers that were islands to themselves are now a brick in the wall as they integrate into larger global high-speed connectivity. The hierarchal nature of globally and locally distributed microcontrollers means dedicated processing can occur closer to where data is being generated. The network is the computer.As higher-resolution data is sensed and processed, chunks of information extraction at each level often result in huge data sets that require a massive pool of immediate and virtual low-latency storage. Smart houses integrate into a higher level of the distributed processing hierarchy to behave in a city management process domain. Eventually, the need for our power grid to become a dynamically intelligent entity will be more apparent as electric vehicles and charges become the norm.In all cases, the Internet of Things (IoT) is central. Everything is connected: from handheld devices to cars and homes, traffic lights to biohazard sensors, factories to cities. This results in each system having multiple edges—some secure, some less so. In this evolving landscape of connectivity, machine learning is crucial as security concerns continually increase with hostile forces trying to wreak havoc.<h2>Machine Learning Is Key</h2>Designers of modern factories, appliances, and more seek machine learning (ML) solutions to tackle new challenges and allow companies to advance and compete. However, low latencies are required to allow useful data integration from extensive and varied sources. As a result, data must be accessed and transported globally very quickly to make these higher-level "thinking machines" respond in an effective and reasonable amount of time.&nbsp;This poses a challenge. Process-intensive security schemes and algorithms can add latency, meaning hardware security acceleration is needed and must be robust enough to protect against threats. Often, vital infrastructure is online and requires high-security protection.To meet these processing and security requirements, this new generation of designs calls for highly integrated MCUs. Communications data rates have grown from 50 bits per second (bps) to 10 gigabits per second (Gbps). Software-controlled bit banging will not work at these higher rates, so very high-speed communications hardware must be incorporated into modern microcontrollers.Additionally, wireless options like Wi-Fi® and Bluetooth®, as well as wired technologies like Ethernet and USB 3.x communications, must peacefully coexist in one system. Intelligent edge devices like facility routers can implement each of these simultaneously. Modern microcontrollers house several encapsulated protocol standards using dedicated accelerated hardware to further unburden CPUs.Analog and digital signal processing (DSP) requirements must also be addressed, especially with the many IoT devices that include sensor systems as part of their functional requirements. DSP functionality is critical to meet response times for industrial IoT control loops, whether it be simple thermostats or complex high-speed analog data streams from real-time factories. All this increased performance and functionality must be low power and feature power management techniques, as server farms already use tremendous amounts of energy.<h2>Meet the MCX N Series</h2>NXP Semiconductors' new <a href="https://www.mouser.com/new/nxp-semiconductors/nxp-mcx-n-mcus/" target="_blank">MCX N</a> Series advanced processor engines are general-purpose and application-specific microcontrollers based on Arm® Cortex®-M33 CPUs. The MCX N Series devices feature a rich mix of hardware-accelerated peripherals, communications, and signal processing, focusing on scalability and ease of development (Figure 1).<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE0OTg2MjY2ODM1LU1DWF9CbG9ja19EaWFncmFtLndlYnAiLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">Figure 1: The high-speed Arm processing cores are surrounded by advanced memory interfaces and controllers, analog converters and DSP accelerators, communications interfaces, and security. (Source: NXP Semiconductors)At the heart of the microcontroller is a pair of 150MHz Cortex-M33 processors with up to 2MB of flash memory (code and data) and 512kB of SRAM. Additional dedicated hardware for DSP and CORDIC (pseudo multiplication and division) acceleration functions allows fast implementation of math-intensive algorithm control loops. One of the key components of this microcontroller is a hardware accelerator, eIQ® Neutron Neural Processing Unit (NPU).The MCX N Series features a plethora of communications interfaces, including Ethernet, USB FS (12Mbits/sec), USB HS (480Mbits/sec), I³C, CAN FD, and UART, as well as advanced mixed-signal interfaces like four single-ended or two differential 16-bit analog-to-digital converter (ADC) channels, 12- and 14-bit digital-to-analog converters (DACs), op amps, comparators, and stable temperature voltage references. A digital Synchronous Audio Interface (SAI) can generate waveforms and levels, and touch-sensing interfaces reduce form factors for handheld and space-constrained designs.The microcontrollers also feature NXP’s EdgeLock® Secure Enclave, Core Profile to ensure device-wide security intelligence, run time attestation, silicon root trust, and key management. Other security features include extensive cryptographic services, trust provisioning, and simplified path certifications. A high-performance crypto engine is also built in to perform on-the-fly encryption and decryption that provides a secure boot.<h3>Machine Learning with the MCX N Series</h3>An important feature of the NXP MCX N microcontrollers is the elQ® Neutron NPU (Figure 2). The NPU's scalability to billions or trillions of operations per cycle has the power to implement different learning techniques. One technique, the convolutional neural network (CNN) architecture, is a feed-forward neural network that learns feature engineering through filters. Also supported are recurrent neural networks (RNN), which are well suited for sequential time series data sets as part of the deep learning algorithms. Temporal convolution networks (TCN), which employ casual convolutions and dilations, provide adaptive learning and are also supported by the NXP elQ Neutron NPU.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE0OTg2Mjg3NTg0LU5YUCBlbFEgTmV1dHJvbiBOUFUud2VicCIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Figure 2: The MCX N microcontrollers’ scalable eIQ NPU is a dedicated controller core with in-line dequantization, activation, and pooling. It boasts learning times that are 30 times faster than conventional computational cores. (Source: NXP Semiconductors)<h2 id="isPasted">The Road to Development</h2>Unified software tools are essential for design engineers and software developers. The NXP MCUXpresso suite of development software and tools can help designers with a variety of tasks, including RTOS development.A choice of integrated development environments (IDEs), including MCUXpresso for VS Code, MCUXpresso IDE, IAR Embedded Workbench and Arm Keil MDK, work with a configuration tool to configure peripherals and multifunction pins. Bootables and utilities work with debug tools like Segger, NXP MCU-Link, FreeMASTER, and PEmicro to provide breakpoint, trace, and control/monitor functions that aid in software and firmware development. &nbsp;GUI Guider helps design HMI graphics libraries and displays.The elQ machine learning software development environment is essential, leveraging inference engines, neural network compilers, optimized libraries, and deep learning toolkits. The hardware abstraction layers support importing Pytorch, TensorFlow and ONNX models and running inference with TensorFlow Lite for Microcontrollers. These neural nets help learn and output useful information for sensor data in anomaly detection and predictive maintenance, voice and keyword recognition response, as well as image and video processing for object detection and recognition.The eIQ Toolkit provides a machine learning workflow and tools that enable graphical or command line interface to build, profile, and deploy your machine learning models with runtime insights. The eIQ portal is a graphic tool that lets designers create, optimize, debug, and export machine learning models. It can also import models from TensorFlow.NXP offers a wide range of application software, examples, reference designs, application notes, articles, and blogs on their website for applications such as audio, cloud connectivity, machine learning, motor control, RTOS, security, voice processing, touch sensing, and both wired and wireless connectivity.<h2>Conclusion</h2>Finding the right processor solution can take time and research. Not every design needs high-end supercomputing horsepower, but many applications in the Industrial and IoT edge require a significant amount of processing power and high-end peripherals to perform.The NXP MCX N Series is designed to meet these needs and more as it addresses the challenges of modern intelligent edge applications, such as IoT, security, power constraints, machine learning, and cost. With a mature and comprehensive development environment, the MCX N Series is ready to help engineers develop their designs, including newer technologies like neural networking and machine learning.

NXP Semiconductors' new MCX N Series advanced processor engines are general-purpose and application-specific microcontrollers based on Arm® Cortex®-M33 CPUs. 

The Evolving Edge: Transformation Is Now

<section id="isPasted">One concern about AI is that it can imitate voices, a practice called voice spoofing. Deepfakes are a prominent type of voice spoofing facilitated by deep learning, a process in which AI modeled after neural networks in the human brain analyzes data and draws connections. Criminals can use voice spoofing to impersonate someone’s family member and ask for money. Voice spoofing is also a concern in manufacturing settings where verbal commands are widely used to control machinery. Criminals can engineer voices to manipulate and disrupt the machinery or steal data.&nbsp;Machines and manufacturing personnel need a way to verify that verbal commands are legitimate. Such protections do exist but cannot stop skilled attacks like deepfakes, requiring additional safeguards.&nbsp;Dr. Nitesh Saxena, a professor in the Department of Computer Science and Engineering at Texas A&amp;M University, and Yingying (Jennifer) Chen, professor and department chair of Electrical and Computer Engineering at Rutgers University, are working to protect voice control technology against manipulation. Smartwatches may be part of their solution.</section><blockquote>Improving the security of voice authentication is critical in many applications, especially in manufacturing domains.<cite>Dr. Nitesh Saxena</cite></blockquote><section>Wearable devices like smartwatches are already equipped with microphones and accelerometers, which measure vibrations. Saxena and Chen plan to develop software that can be installed on smartwatches or other wearable devices to help authenticate commands in manufacturing settings.&nbsp;Voice control technologies are susceptible to attack because existing defenses only assess a voice sample’s acoustic properties. Saxena and Chen hope to enhance security by requiring verification of both the acoustic properties and the vibration signals of voice samples. They believe this will improve security because it will be harder for criminals to fake both measures simultaneously.&nbsp;“Improving the security of voice authentication is critical in many applications, especially in manufacturing domains. Thanks to the support from our sponsor, this project will allow us to explore this important line of research,” Saxena said. His&nbsp;<a href="https://spies.engr.tamu.edu/" rel="noopener" target="_blank">lab</a> has pioneered work in voice-based security.&nbsp;</section>

Dr. Nitesh Saxena is teaming up with the Texas A&M Engineering Experiment Station and Rutgers University to create software that could stop voice-control mischief in manufacturing settings.

Fighting Deepfakes with Smartwatches

<section id="isPasted">In order to deploy mobile robots, self-driving cars, or drones into the real world, they need to be able to observe and understand the world around them, just like humans do. PhD candidate at TU/e Daan de Geus developed algorithms for automatic image recognition that are faster and more accurate than existing models. Last Wednesday, he obtained his PhD (with honors) at the Department of Electrical Engineering.</section><section tabindex="-1">De Geus thumbs through his dissertation and shows us a photo of a street, featuring different objects such as people, vehicles, traffic posts, and traffic lights (see image below). “Mobile robots and self-driving cars need to know what’s happening around them,” he says.Among other things, they need to be able to recognize and localize objects, so they can take these into account. “This allows them to move around or toward a certain object, and maybe pick it up to carry out a task.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1714637661762-csm_frankfurt_000001_030669_images_0a92c1e725.jpg" style="width: 100%px;" class="fr-fic fr-dib">Traffic in Frankfurt. Image: Daan de GeusTo make robots aware of their environment, different computer vision techniques are used. It concerns an area of research that revolves around automatic extraction of relevant information from camera footage. “Simply put, we try to make models that can distill as much information from a photo as possible,” the PhD candidate explains.“The goal of computer vision is to make a system that mimics our own visual system, allowing computers to see in the same way humans do and, as a result, interact properly with the world around them.”His research focuses on improving image recognition techniques in the area of scene understanding. This is a small, but crucial part of computer vision. “Its goal is to recognize different objects and regions in an image and to give them a semantic label, such as ‘streetlight’, ‘road’, ‘car’, or ‘person’,” he explains.In other words, the objects are given a significance that is clear to humans.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1714637694452-csm_scene_understanding_example_189e7cc585.jpg" style="width: 100%px;" class="fr-fic fr-dib">An example of scene understanding. Objects (person, traffic light, car) and image regions (the road) are automatically identified, color-coded, and semantically labeled. Image: Daan de Geus<h2 id="isPasted">Accuracy and efficiency</h2>For automatic image analyses, neural networks are used – systems that learn to carry out a certain task by receiving training involving a large amount of data. By training these neural networks in a focused manner, you end up with different models that specialize in specific tasks, such as identifying all cars in an image.In the first part of his dissertation, De Geus looked at how you can improve the accuracy and efficiency of those models. “Improving one aspect is often at the expense of another. To get more accurate results, existing methods tend to lose some of their efficiency, and vice versa,” he explains.“Which makes sense, because greater accuracy often demands more computing power, which directly impacts efficiency.” The big question, therefore, is: how can you improve these aspects without having to compromise?He came up with several solutions to this conundrum. “Efficiency is important for two reasons. You want the algorithm to use as little energy as possible and to make a prediction as quickly as possible,” he says. Speed is of the essence in self-driving vehicles, as cars must be able to respond to situations that arise in a timely manner.“If the calculation takes a second, it may already be too late to take action.”Using a process known as model unification, he combined two models to create a more efficient one. “Some tasks focus on the objects in the foreground, such as cars and people; others on ‘background regions’ such as vegetation and the sky. These tasks are performed by two different neural network modules, as each module specializes in a different task,” he explains.“Using two network modules isn’t very efficient, because you have to run them parallel to each other.” He found out that you can supply extra information to the network module processing the background information, allowing it to also identify foreground objects.“As a result, the module for foreground objects is no longer necessary, which greatly enhances efficiency. This makes this model twice as fast as its precursors, while its accuracy is comparable.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1714637727049-csm_image_with_patches_0ce2b38ff7.jpg" style="width: 100%px;" class="fr-fic fr-dib">Image with image regions. Image: Daan de GeusAnother way of improving efficiency is based on the observation that many regions in an image are very similar. For instance, the sky generally takes up a large part of any picture you take outside. “In spite of the fact that a lot of information is similar, neural networks process each image region separately, which is very inefficient,” says De Geus.The entire image is divided up into patches consisting of pixels (see image below). Normally, these would be labeled and assessed individually, but the PhD candidate developed a method for clustering patches with comparable information, reducing the total number of patches and, by extension, requiring less computing power.“This allows us to improve speed by as much as 110%, without taking away from the accuracy.”Incidentally, this method for grouping similar image regions is widely applicable to many different models and a great many applications other than self-driving cars and mobile robots. For example, you could use these algorithms to segment medical images. “Pretty much in any situation where you need to automatically analyze images.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1714637752641-csm_token_sharing_diagram_c977af3012.jpg" style="width: 100%px;" class="fr-fic fr-dib">To improve efficiency, semantically comparable patches can be clustered. Image: Daan de Geus<h3 id="isPasted">Multiple abstraction levels</h3>In the second part of his dissertation, he focused on what he calls multiple abstraction levels. “Existing algorithms either focus on entire objects, such as cars, or on their components, such as car tires or the license plate,” he explains.His goals was to develop an algorithm that can simultaneously understand an image at multiple abstraction levels. “This will allow a mobile robot to observe both the car and its components, as well as other objects and the background, all at the same time, which gives it a comprehensive picture of the surroundings,” says De Geus.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1714637778377-csm_frankfurt_000001_030669_panoptic_5abb4ba438.jpg" style="width: 100%px;" class="fr-fic fr-dib">Automatic recognition of objects and image regions. Image: Daan de GeusYou can have both algorithms carry out their work separately and merge the results afterwards, but this is cumbersome. And it can also lead to conflicts between the two calculations. Instead, he developed a new algorithm that can identify objects and components all at once.“That’s not only more accurate, but also more efficient.” What’s more, this improved model is widely applicable and has many advantages. “It means, for instance, that a robot isn’t just able to recognize a door, but also the door handle, and to understand that the latter is part of the door. Which, in turn, allows it to open the door,” he says to illustrate.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1714637801563-csm_frankfurt_000001_030669_pps_3b32a5d47d.jpg" style="width: 100%px;" class="fr-fic fr-dib">A more detailed segmentation, where components of objects are also identified. Image: Daan de GeusTo be able to use self-driving cares or mobile robots in practice, there’s a lot left to do still. “Image recognition is just a tiny part of the puzzle,” he admits.&nbsp;And there are plenty of remaining challenges in the area. For one thing, models could be developed that can recognize more objects at once and that operate even faster and more accurately.The generalization of the model could also be improved, so it also produces good results when the data are different from the data sets with which the network was trained, allowing it to perform better in real-world scenarios.But at least his dissertation constitutes a step forward in making the system more capable, accurate, and efficient.<hr>Source: <a href="https://www.cursor.tue.nl/en/news/2024/april/week-3/home-stretch-teaching-robots-to-see-better/" target="_blank" rel="noreferrer">Cursor/Martina Silbrníková</a>Daan de Geusdefended his thesis“<a href="https://research.tue.nl/en/publications/characterization-of-abdominal-aortic-aneurysms-using-time-resolve" target="_blank" rel="noreferrer">Advances in Scene Understanding - Towards Efficient Image Segmentation at Multiple Abstraction Levels</a>” at the Department of Electrical Engineering on April 17th, 2024.Supervisors: Gijs Dubbelman en Peter de With</section>

In his doctoral research, Daan de Geus worked on advanced image processing methods that allow robots and cars to better recognize what they see around them.

Teaching robots to see better

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/96f146e3-d84e-4bc9-8fa4-af61e7ce87ed?dark=false" class="fr-draggable" data-gtm-yt-inspected-9="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>The full article will continue below.<hr>Pursuing a viable alternative to invasive brain-computer interfaces (BCIs) has been a continued research focus of Carnegie Mellon University’s <a href="https://www.cmu.edu/bme/helab/index.html" rel="noopener" target="_blank">He Lab</a>. In 2019, the group used a noninvasive BCI to successfully demonstrate, for the first time, that a mind-controlled robotic arm had the ability to continuously track and follow a computer cursor. As technology has improved, their AI-powered deep learning approach has become more robust and effective. In new work published in PNAS Nexus, the group demonstrates that humans can control continuous tracking of a moving object all by thinking about it, with unmatched performance.Noninvasive BCIs bring a host of advantages, in contrast to their invasive counterparts (think Neuralink or Synchron). These include increased safety, cost-effectiveness, and an ability to be used by numerous patients, as well as the general population. Amid these benefits, however, noninvasive BCIs face challenges because their recordings are less accurate and difficult to interpret.<iframe width="640" height="360" src="https://www.youtube.com/embed/EoKHw6bdoZs?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-9="true" id="917213689" title="Bin He: Neuroengineering and Dynamic Brain Mapping">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</iframe>In He’s <a href="https://academic.oup.com/pnasnexus/article-lookup/doi/10.1093/pnasnexus/pgae145" rel="noopener" target="_blank" id="isPasted">recent study</a>, a group of 28 human participants were given a complex BCI task to track an object in a two-dimensional space all by thinking about it. During the task, an electroencephalography (EEG) method recorded their activity, from outside the brain. Using AI to train a deep neural network, the He group then directly decoded and interpreted human intentions for continuous object movement using the BCI sensor data. Overall, the work demonstrates the excellent performance of non-invasive BCI for a brain-controlled computerized device.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE0NTE3ODg4OTAzLTA0MzAtZW0tbm9uaW52YXNpdmUtYmNpLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Source:&nbsp;He Lab“The innovation in AI technology has enabled us to greatly improve the performance versus conventional techniques, and shed light for wide human application in the future,” expressed <a href="https://engineering.cmu.edu/directory/bios/he-bin.html" rel="noopener" target="_blank" id="isPasted">Bin He</a>, professor of <a href="https://www.cmu.edu/bme/" rel="noopener" target="_blank">biomedical engineering</a> at Carnegie Mellon University.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1714517922793-0430-em-2-noninvasive-bci.gif" style="width: 100%px;" class="fr-fic fr-dib">This video shows the cursor and target trajectories throughout a single 60-second trial with lag adjusted. The randomly moving target is represented by a yellow circle, while the cursor that the subject controlled using the BCI system is shown as a blue circle. A human subject controlled the blue cursor using the deep learning-based BCI decoder just by thinking about it. Source:&nbsp;He LabMoreover, the capability of the group’s AI-powered BCI suggests a direct application to continuously controlling a robotic device.“We are currently testing this AI-powered noninvasive BCI technology to control sophisticated tasks of a robotic arm,” said He. “Also, we are further testing its applicability to not only able-body subjects, but also stroke patients suffering motor impairments.” In a few years, this may lead to AI-powered assistive robots becoming available to a broad range of potential users.To this end, motor-impaired patients who are suffering from spinal cord injury, stroke, or other movement impairment, but do not want to receive an implant, stand to benefit immensely from research in this vein. “We keep pushing noninvasive neuroengineering solutions that can help everybody,” added He.This work was supported in part by the National Institute of Neurological Disorders and Stroke, National Center for Complementary and Integrative Health, and National Institute of Biomedical Imaging and Bioengineering.Other collaborators on the PNAS Nexus paper include the first authors Dylan Forenzo, BME Ph.D. student, and Hao Zhu, former BME Masters student; Jenn Shanahan, former lab technician; and Jaehyun Lim, BME undergraduate student.

Achieving a noteworthy milestone to advance noninvasive brain-controlled interfaces, researchers used AI technology to improve the decoding of human intention and control a continuously moving virtual object all by thinking about it, with unmatched performance.

Refined AI approach improves noninvasive BCI performance

Artificial intelligence (AI) is a wide-ranging tool that enables people to rethink how we integrate information, analyze data, and use the resulting insights to improve decision making


Artificial Intelligence: A Comprehensive Guide to its Engineering Principles and Applications

Plainsight-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

THINC Lab-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

THINC Lab-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

THINC Lab-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

THINC Lab-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

THINC Lab-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NAVER LABS Europe-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

NAVER LABS Europe-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NAVER LABS Europe-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NAVER LABS Europe-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

NAVER LABS Europe-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Syntiant-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Syntiant-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Syntiant-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Syntiant-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Syntiant-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

The Linley Group-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

The Linley Group-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

The Linley Group-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

The Linley Group-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

The Linley Group-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Monogoto-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Monogoto-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Monogoto-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Monogoto-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Monogoto-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Introduction

Edge AI Market Analysis and Trends

Healthcare and Medical Applications

Industrial IoT and Manufacturing

Smart Cities and Urban Infrastructure

Retail and Customer Experience

Energy Efficiency and Sustainability

Agriculture and Food Production 

Automotive and Transportation

Generative AI at the Edge

Edge AI Challenges and Real-World Mitigations

The Future of Edge AI

Exploring the Dynamic World of Edge AI Applications Across Industries


2024 State of Edge AI Report

A.I.

An AI-generated depiction of a modern day chipset

Gaining a comprehensive understanding of the distinct capabilities and applications of TPUs and GPUs is crucial for developers and researchers aiming to navigate the complex and rapidly changing terrain of artificial intelligence.

TPU vs GPU in AI: A Comprehensive Guide to Their Roles and Impact on Artificial Intelligence

This article provides an in-depth exploration of the evolution, fundamental concepts, diverse types, operating principles, and practical applications of ASICs.

What is an ASIC: A Comprehensive Guide to Understanding Application-Specific Integrated Circuits

Nvidia’s CUDA cores are specialized processing units within Nvidia graphics cards designed for handling complex parallel computations efficiently, making them pivotal in high-performance computing, gaming, and various graphics rendering applications.

Understanding Nvidia CUDA Cores: A Comprehensive Guide

Clustering - A Common Unsupervised Learning Technique

Understanding What is Unsupervised Learning, the Mechanisms, Types, and Applications of Various Algorithms and Challenges it presents in Machine Learning

What is Unsupervised Learning: A Comprehensive Guide

Exploring the key concepts related to Unsupervised vs Supervised Learning, understanding the fundamental principles, major algorithms and their real-world applications, and practical distinctions between supervised and unsupervised learning.

Unsupervised vs Supervised Learning: A Comprehensive Comparison

A.I.

2024 State of Edge AI Report

Introduction

1. Edge AI Market Analysis and Trends

2. Healthcare and Medical Applications

3. Industrial IoT and Manufacturing

4. Smart Cities and Urban Infrastructure

5. Retail and Customer Experience

6. Energy Efficiency and Sustainability

7. Agriculture and Food Production

8. Automotive and Transportation

9. Generative AI at the Edge

10. Edge AI Challenges and Real-World Mitigations

11. The Future of Edge AI

Artificial Intelligence: A Comprehensive Guide to its Engineering Principles and Applications

Profiles