A less wasteful way to train large language models, such as the GPT series, finishes in the same amount of time for up to 30% less energy, according to a new study from the University of Michigan.The approach could save enough energy to power <a href="https://www.eia.gov/energyexplained/use-of-energy/electricity-use-in-homes.php" target="_blank" rel="noreferrer noopener">1.1 million U.S. homes</a> in 2026, based on Wells Fargo’s projections of <a href="https://www.marketwatch.com/story/ai-could-demand-a-shocking-amount-of-electricity-check-out-this-chart-e91e306d" target="_blank" rel="noreferrer noopener">AI power demand</a>. It could also take a bite out of the International Monetary Fund’s prediction that data centers could account for <a href="https://www.imf.org/en/Blogs/Articles/2024/08/15/carbon-emissions-from-ai-and-crypto-are-surging-and-tax-policy-can-help" target="_blank" rel="noreferrer noopener">1.2% of the world’s carbon emissions by 2027</a>—and the <a href="https://theconversation.com/ais-excessive-water-consumption-threatens-to-drown-out-its-environmental-contributions-225854">water demands</a> that come with that energy use.Some experts say that these costs could be outweighed by environmental benefits. They argue that AI could be a <a href="https://news.un.org/en/story/2023/11/1143187" target="_blank" rel="noreferrer noopener">“game changer” for fighting climate change</a> by identifying ways to optimize supply chains and the grid, manage our energy needs, and improve research on climate change. Still, that doesn’t excuse squandering energy, and some of the power used to train AI has zero impact on training time and model accuracy.“Why spend something when there’s no point?” said <a href="https://eecs.engin.umich.edu/people/chowdhury-mosharaf/" target="_blank" rel="noreferrer noopener">Mosharaf Chowdhury</a>, an associate professor of computer science and engineering and the corresponding author of the study presented at the <a href="https://sigops.org/s/conferences/sosp/2024/index.html" target="_blank" rel="noreferrer noopener">30th Symposium on Operating Systems Principles</a> last Monday.“We can’t keep building bigger and bigger data centers because we won’t have the power to run them,” said Chowdhury. “If we can reduce the energy consumed by AI, we can reduce AI’s carbon footprint and cooling requirements and allow for more computation to fit within our current energy constraints.”The energy waste is created when AI training is unequally divided between GPUs, which are computer processors specialized for large data and graphics applications. Although it opens the door for waste, splitting the work is necessary for processing huge datasets.“AI models today are so large, they cannot fit inside a single computer processor,” said <a href="https://jaewonchung.me/about" target="_blank" rel="noreferrer noopener">Jae-Won Chung</a>, a doctoral student in computer science and engineering and the first author of the study. “They need to be divided into tens of thousands of processors to be trained, but dividing the models in perfectly equal sizes across all processors is practically impossible.”The training jobs are so difficult to evenly split up because some tasks need to be grouped together on the same processor—like how each installment of a book series will be grouped together in an organized shelf. Depending on how the tasks are grouped, some processors might get stuck with the AI-training equivalent of the Encyclopedia Britannica while others get assigned a fantasy trilogy.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1731630851359-AI_waste_Infographic-768x1470.jpg" style="width: 100%px;" class="fr-fic fr-dib">Infographic credit: Rachel Voigt, Michigan Engineering.Because current training methods run each processor at top speed, processors with a lighter load will finish their calculations before other processors. This doesn’t speed up training, which isn’t complete until every processor finishes its job—but it is wasteful because faster calculations require more energy. In addition, problems such as faulty hardware or network delays create energy waste by slowing down a single processor’s computing speed.To save energy, the researchers developed a software tool, called&nbsp;<a href="https://ml.energy/zeus/research_overview/perseus/" target="_blank" rel="noreferrer noopener">Perseus</a>, that identifies a critical path, or a series of subtasks that will take the longest time to complete. Then, Perseus slows down processors that aren’t on the critical path so that they all finish their jobs around the same time—eliminating unnecessary power use.“Reducing the power cost of AI can have important implications for equitable AI access,” said Chowdhury. “If a country doesn’t have enough power to run a big model, they might need to use services from far away, or be stuck running smaller, less accurate models. This gap could further perpetuate disparity between different communities.”The team tested Perseus by training GPT-3, three other large language models and one computer vision model.Perseus is an open-sourced tool available as part of&nbsp;<a href="https://ml.energy/zeus/" target="_blank" rel="noreferrer noopener">Zeus</a>, a tool for measuring and optimizing AI energy consumption.&nbsp;

Smarter use of processor speeds saves energy without compromising training speed and performance.

Up to 30 percent of the power used to train AI is wasted. Here's how to fix it.

In this episode, we explore how AI co-pilots are equipping doctors with powerful tools to enhance decision-making and patient care and discuss how it could impact you - a potential patient - in the not distant future.

Podcast: Will AI Assistants Make Your Doctor Better?

Large Language Model (LLM) training involves teaching AI models to understand and generate human-like text by processing vast amounts of data, significantly enhancing their language comprehension and production capabilities.

LLM Training: Mastering the Art of Language Model Development

In the classic cartoon “The Jetsons,” Rosie the robotic maid seamlessly switches from vacuuming the house to cooking dinner to taking out the trash. But in real life, training a general-purpose robot remains a major challenge.Typically, engineers collect data that are specific to a certain robot and task, which they use to train the robot in a controlled environment. However, gathering these data is costly and time-consuming, and the robot will likely struggle to adapt to environments or tasks it hasn’t seen before.To train better general-purpose robots, MIT researchers developed a versatile technique that combines a huge amount of heterogeneous&nbsp;data from many of sources into one system that can teach any robot a wide range of tasks.Their method involves aligning data from varied domains, like simulations and real robots, and multiple modalities, including vision sensors and robotic arm position encoders, into a shared “language” that a generative AI model can process.By combining such an enormous amount of data, this approach can be used to train a robot to perform a variety of tasks without the need to start training it from scratch each time.This method could be faster and less expensive than traditional techniques because it requires far fewer task-specific data. In addition, it outperformed training from scratch by more than 20 percent in simulation and real-world experiments.“In robotics, people often claim that we don’t have enough training data. But in my view, another big problem is that the data come from so many different domains, modalities, and robot hardware. Our work shows how you’d be able to train a robot with all of them put together,” says Lirui Wang, an electrical engineering and computer science (EECS) graduate student and lead author of a <a href="https://arxiv.org/pdf/2409.20537" target="_blank">paper on this technique</a>.Wang’s co-authors include fellow EECS graduate student Jialiang Zhao; Xinlei Chen, a research scientist at Meta; and senior author Kaiming He, an associate professor in EECS and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL). The research will be presented&nbsp;at the Conference on Neural Information Processing Systems.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1730247515325-MIT-HeteroGenRobot-02-press.jpg" style="width: 100%px;" class="fr-fic fr-dib">A figure shows how the new technique aligns data from varied domains, like simulation and real robots, and multiple modalities, including vision sensors and robotic arm position encoders, into a shared “language” that a generative AI model can process. Image: Courtesy of the researchers<h2>Inspired by LLMs</h2>A robotic “policy” takes in sensor observations, like camera images or proprioceptive measurements that track the speed and position a robotic arm, and then tells a robot how and where to move.Policies are typically trained using imitation learning, meaning a human demonstrates actions or teleoperates a robot to generate data, which are fed into an AI model that learns the policy. Because this method uses a small amount of task-specific data, robots often fail when their environment or task changes.To develop a better approach, Wang and his collaborators drew inspiration from large language models like GPT-4.These models are pretrained using an enormous amount of diverse language data and then fine-tuned by feeding them a small amount of task-specific data. Pretraining on so much data helps the models adapt to perform well on a variety of tasks.“In the language domain, the data are all just sentences. In robotics, given all the heterogeneity in the data, if you want to pretrain in a similar manner, we need a different architecture,” he says.Robotic data take many forms, from camera images to language instructions to depth maps. At the same time, each robot is mechanically unique, with a different number and orientation of arms, grippers, and sensors. Plus, the environments where data are collected vary widely.The MIT researchers developed a new architecture called Heterogeneous Pretrained Transformers (HPT) that unifies data from these varied modalities and domains.They put a machine-learning model known as a transformer into the middle of their architecture, which processes vision and proprioception inputs. A transformer is the same type of model that forms the backbone of large language models.The researchers align data from vision and proprioception into the same type of input, called a token, which the transformer can process. Each input is represented with the same fixed number of tokens.Then the transformer maps all inputs into one shared space, growing into a huge, pretrained model as it processes and learns from more data. The larger the transformer becomes, the better it will perform.A user only needs to feed HPT a small amount of data on their robot’s design, setup, and the task they want it to perform. Then HPT transfers the knowledge the transformer grained during pretraining to learn the new task.<h2>Enabling dexterous motions</h2>One of the biggest challenges of developing HPT was building the massive dataset to pretrain the transformer, which included 52 datasets with more than 200,000 robot trajectories in four categories, including human demo videos and simulation.The researchers also needed to develop an efficient way to turn raw proprioception signals from an array of sensors into data the transformer could handle.“Proprioception is key to enable a lot of dexterous motions. Because the number of tokens is in our architecture always the same, we place the same importance on proprioception and vision,” Wang explains.When they tested HPT, it improved robot performance by more than 20 percent on simulation and real-world tasks, compared with training from scratch each time. Even when the task was very different from the pretraining data, HPT still improved performance.“This paper provides a novel approach to training a single policy across multiple robot embodiments. This enables training across diverse datasets, enabling robot learning methods to significantly scale up the size of datasets that they can train on.&nbsp;It also allows the model to quickly adapt to new&nbsp;robot embodiments, which is important as new robot designs are continuously being produced,” says David Held, associate professor at the Carnegie Mellon University Robotics Institute, who was not involved with this work.In the future, the researchers want to study how data diversity could boost the performance of HPT. They also want to enhance HPT so it can process unlabeled data like GPT-4 and other large language models.“Our dream is to have a universal robot brain that you could download and use for your robot without any training at all. While we are just in the early stages, we are going to keep pushing hard and hope scaling leads to a breakthrough in robotic policies, like it did with large language models,” he says.

Researchers filmed multiple instances of a robotic arm feeding co-author Jialiang Zhao's adorable dog, Momo. The videos were included in datasets to train the robot. Image: Courtesy of the researchers

Inspired by large language models, researchers develop a training technique that pools diverse data to teach robots new skills.

A faster, better way to train general-purpose robots

In recent years, the field of robotics has seen remarkable advancements in gripper technology that pushes the boundaries of dexterity and manipulation. Shadow Robot is at the forefront of this evolution as a pioneer in innovative robotics development. Shadow’s approach to the innovative solutions has caught the attention of industry leaders, including Google DeepMind, a leading AI research lab focused on developing artificial intelligence capable of performing a wide range of tasks. This collaboration has led to the creation of DEX-EE, a groundbreaking gripper that may revolutionize robotic manipulation and machine learning research.<h2 dir="ltr">The Challenge from Google DeepMind</h2>In 2017, Google DeepMind approached Shadow Robot with a big challenge. The DeepMind project was conducting research on generalized intelligence and the role of embodiment in learning—how physical interaction with the world plays a part in learning, especially for tasks that involve understanding and manipulating objects. The study required a robot with high sensitivity and a robust design.For their research, the DeepMind team needed a design that could reliably run long-term and perform physically demanding experiments without constraints within its workspace. The study required a robot that could strike its environment repeatedly to learn how to manipulate objects. This meant that the robot needed to be resilient, sensor-rich and provide large volumes of high-quality data, while operating for prolonged periods without interruption for repairs and without the need to restart experiments.The challenge pushed the boundaries of existing robotic capabilities because it required a solution that was both robust and highly precise. Google DeepMind approached Shadow Robot for this challenge. Shadow Robot has partnered with numerous research projects and the company’s flagship Shadow Dexterous Hand has contributed to major advancements in the field of robotics.&nbsp;Read more about the Shadow Dexterous Hand and its contributions in our <a href="https://www.wevolver.com/article/dexterous-robotic-hands-part-1-unraveling-the-history-technology-and-applications-of-dexterous-robotic-hands">three-part Shadow Dexterous Hand series</a>.<h3 dir="ltr">DEX-EE: Shadow Robot’s Innovative Solution</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1730204504132-Untitled+design+%285%29.png" style="width: 100%px;" class="fr-fic fr-dib">In response to the challenge from Google DeepMind, Shadow Robot developed DEX-EE, a groundbreaking dexterous robot hand, as well as a major advancement in the field of machine learning robots. DEX-EE is designed to perform with dynamic and controlled motion in a robust and reliable package. The hand enables long-running experiments without interruptions due to hardware failure.One of DEX-EE’s most innovative features is its high-speed network sensors, which provide rich data including position, force, and inertial measurement. Each finger has hundreds of channels of tactile sensing, and Shadow’s advanced fingertip sensors have hundreds of taxels (short for tactile pixel—the individual sensing elements that make up a tactile sensor), each with a massive dynamic range. With this wealth of sensory data, researchers can gather huge amounts of data from dexterous robotic tasks.DEX-EE was developed with an intensive focus on reliability and robustness. Through iterative design and testing over five years, Shadow created a new dexterous hand that could withstand the physical demands of Google DeepMind’s research. As a result of this focused development, the DEX-EE is a functional, reliable, and data-rich research tool. DEX-EE’s capabilities position it as an ideal hardware platform for dexterous robotic research into cognition and manipulation.<h2 dir="ltr">DEX-EE Technical Specifications and Advancements</h2>DEX-EE is a significant leap in dexterous robotic hand technology. The robot has high-bandwidth torque and position control loops, which enable delicate and precise fingertip dexterity, while torque and inertial measurement throughout the hand make it sensitive to interactions with its environment.The hand also features tactile fingertip sensors based on stereo-camera technology, which gives the hand abundant 3D data about the environment in a robust package. Additional, multi-taxel, 3-DOF sensors on the middle and proximal fingers provide crucial information during grasping and manipulation tasks.Compared to earlier models of the Shadow Hand, DEX-EE offers greater reliability and repairability. It was designed for easy maintenance with minimal training, incorporates fail-safes and a shutdown routing that reduces instances of failure and downtime. The hand’s advanced sensor networks provide rich data, including position, force, and inertial measurement, with hundreds of channels of tactile sensing per finger. Shadow’s innovative fingertip sensors have hundreds of taxels each, with a dynamic range that significantly surpasses the capabilities of other dexterous hands.DEX-EE is also integrated with ROS (Robot Operating System), making it versatile as a research tool. This combination of robustness, advanced sensing capabilities, and ease of integration positions DEX-EE as an ideal platform for dexterous manipulation research that is capable of withstanding the demands of long-term machine learning experiments.<h2 dir="ltr" id="isPasted">Applications in Machine Learning and AI Research</h2><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1730197463018-shadowrobot-grip.png" style="width: 100%px;" class="fr-fic fr-dib">Image: DEX-EE performs machine learning experiments with high sensitivity and robustness | Source:&nbsp;<a href="https://www.shadowrobot.com/dex-ee/" target="_blank" rel="noopener noreferrer">shadowrobot.com</a>DEX-EE represents the next major advancement in dexterous manipulation research because it provides a robust platform for complex machine learning experiments like Google DeepMind. The advanced sensor networks and tactile capabilities of DEX-EE enable researchers to gather enormous amounts of data during experiments and push the boundaries of what’s possible in robotic learning.Google DeepMind has leveraged DEX-EE’s capabilities in their latest AI system DemoStart, which helps robots learn to perform complex tasks that require dexterous movement. With these systems, Google DeepMind has achieved remarkable results, like inserting a plug into a socket, tightening a bolt, and other tasks that were previously challenging for robots to master.DEX-EE’s durability allows for long-running experiments without interruptions, making it ideal for reinforcement learning protocols that involve repeated physical interactions with the environment. This has already helped researchers explore new frontiers in embodied AI, investigating how physical interactions contribute to learning and the development of generalized intelligence.The hand’s integration with ROS further enhances its versatility as a research tool because it allows for easier implementation of new and existing learning algorithms and control strategies. With this combination of robustness, advanced sensing, and flexibility, DEX-EE is positioned to be a transformative platform for exploring the future of dexterous manipulation in AI and robotics.<h2 dir="ltr">Shadow Robot's Unique Approach to Robotics Development</h2>Shadow Robot continues to distinguish itself as a robotics company through problem-solving approaches and focus on custom solutions. Rather than offering off-the-shelf products that don’t meet a particular need, Shadow engages deeply with clients to understand their specific issues and develop tailored solutions.This collaborative approach has been exemplified in the development of DEX-EE, where Shadow worked closely with Google DeepMind over five years. Shadow’s willingness to rethink fundamental design principles demonstrates their commitment to innovation and their ability to work at the cutting edge of technology.Shadow has positioned themselves as a research and hardware company that is capable of tackling any robotics challenge. They have established a unique niche in the industry, bridging academic research and practical, real-world applications.<h2 dir="ltr">Leading Innovation in Custom Robotics Solutions</h2>Shadow Robot stands out as a leader in collaborative robotics and demonstrates a commitment to addressing practical needs through innovative design and partnership. Shadow’s work on DEX-EE, developed alongside Google DeepMind, emphasizes the company’s ability to meet the changing demands of cutting-edge research. Shadow continues to push the boundaries of dexterity and manipulation by fostering collaboration and by focusing on tailored solutions that meet practical needs. In the evolving field of robotics, Shadow’s dedication to understanding and responding to the unique challenges positions the company as an important player in robotics research.<blockquote>DEX-EE marks a significant milestone for Shadow Robot. While designed for our core research market, it embodies our vision of reaching beyond this to real-world application research. We remain committed to our fundamental research customers, but DEX-EE represents our expanding horizons. It's a step towards a future where our technology not only advances scientific understanding but also addresses practical challenges across industries, allowing us to make a broader impact while continuing to innovate within the research community.</blockquote>Gavin Cassidy, Head of Product Development at Shadow Robot.For more information about Shadow’s collaborative problem solving and innovative robotics, check out the <a href="https://www.shadowrobot.com/">Shadow Robot website</a>. The <a href="https://www.wevolver.com/article/dexterous-robotic-hands-part-1-unraveling-the-history-technology-and-applications-of-dexterous-robotic-hands">Shadow Robot Dexterous Hand series</a> also provides an in-depth look at Shadow’s commitment to advancing the field of robotics.<h3 dir="ltr">References</h3><ol><li>&nbsp;Shadow Robot. “Collaborative Working: A Case Study”. June 13, 2024. Accessed from <a href="https://www.shadowrobot.com/blog/collaboration-case-study/">https://www.shadowrobot.com/blog/collaboration-case-study/</a>&nbsp;</li><li>&nbsp;Shadow Robot. “DEX-EE Overview” The most robust dexterous robot hand on the market”. June 6, 2024. Accessed from <a href="https://www.shadowrobot.com/blog/shadow-robot-hand-overview/">https://www.shadowrobot.com/blog/shadow-robot-hand-overview/</a>&nbsp;</li><li>&nbsp;Shadow Robot. “Introducing DEX-EE the most robust dexterous robot hand on the market’. Accessed from <a href="https://www.shadowrobot.com/dex-ee/">https://www.shadowrobot.com/dex-ee/</a>&nbsp;</li><li>&nbsp;Shadow Robot. “DEX-EE Overview: The most dexterous robot hand on the market”. <a href="https://www.shadowrobot.com/blog/shadow-robot-hand-overview/">https://www.shadowrobot.com/blog/shadow-robot-hand-overview/</a>&nbsp;</li><li>&nbsp;Google DeepMind. “Our latest advances in robot dexterity”. 12 September 2024. Accessed from <a href="https://deepmind.google/discover/blog/advances-in-robot-dexterity/">https://deepmind.google/discover/blog/advances-in-robot-dexterity/</a>&nbsp;</li></ol>

Discover how Shadow Robot’s DEX-EE gripper is advancing robotic dexterity with AI-driven precision.

Gripping the Future: Shadow Robot's DEX-EE and the Next Generation of Dexterous Manipulation

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries. There are also full-sequence diffusion models like Sora, which convert words into dazzling, realistic visuals by successively “denoising” an entire video sequence.Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have proposed a simple change to the diffusion training scheme that makes this sequence denoising considerably more flexible.When applied to fields like computer vision and robotics, the next-token and full-sequence diffusion models have capability trade-offs. Next-token models can spit out sequences that vary in length. However, they make these generations while being unaware of desirable states in the far future — such as steering its sequence generation toward a certain goal 10 tokens away — and thus require additional mechanisms for long-horizon (long-term) planning. Diffusion models can perform such future-conditioned sampling, but lack the ability of next-token models to generate variable-length sequences.Researchers from CSAIL want to combine the strengths of both models, so they created a sequence model training technique called “Diffusion Forcing.” The name comes from “Teacher Forcing,” the conventional training scheme that breaks down full sequence generation into the smaller, easier steps of next-token generation (much like a good teacher simplifying a complex concept).<iframe width="640" height="360" src="https://www.youtube.com/embed/m4rhzR-h-JU?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Diffusion Forcing found common ground between diffusion models and teacher forcing: They both use training schemes that involve predicting masked (noisy) tokens from unmasked ones. In the case of diffusion models, they gradually add noise to data, which can be viewed as fractional masking. The MIT researchers’ Diffusion Forcing method trains neural networks to cleanse a collection of tokens, removing different amounts of noise within each one while simultaneously predicting the next few tokens. The result: a flexible, reliable sequence model that resulted in higher-quality artificial videos and more precise decision-making for robots and AI agents.By sorting through noisy data and reliably predicting the next steps in a task, Diffusion Forcing can aid a robot in ignoring visual distractions to complete manipulation tasks. It can also generate stable and consistent video sequences and even guide an AI agent through digital mazes. This method could potentially enable household and factory robots to generalize to new tasks and improve AI-generated entertainment.“Sequence models aim to condition on the known past and predict the unknown future, a type of binary masking. However, masking doesn’t need to be binary,” says lead author, MIT electrical engineering and computer science (EECS) PhD student, and CSAIL member Boyuan Chen. “With Diffusion Forcing, we add different levels of noise to each token, effectively serving as a type of fractional masking. At test time, our system can “unmask” a collection of tokens and diffuse a sequence in the near future at a lower noise level. It knows what to trust within its data to overcome out-of-distribution inputs.”In several experiments, Diffusion Forcing thrived at ignoring misleading data to execute tasks while anticipating future actions.When implemented into a robotic arm, for example, it helped swap two toy fruits across three circular mats, a minimal example of a family of long-horizon tasks that require memories. The researchers trained the robot by controlling it from a distance (or teleoperating it) in virtual reality. The robot is trained to mimic the user’s movements from its camera. Despite starting from random positions and seeing distractions like a shopping bag blocking the markers, it placed the objects into its target spots.To generate videos, they trained Diffusion Forcing on “Minecraft” game play and colorful digital environments created within Google’s <a href="https://github.com/google-deepmind/lab">DeepMind Lab Simulator</a>. When given a single frame of footage, the method produced more stable, higher-resolution videos than comparable baselines like a Sora-like full-sequence diffusion model and ChatGPT-like next-token models. These approaches created videos that appeared inconsistent, with the latter sometimes failing to generate working video past just 72 frames.Diffusion Forcing not only generates fancy videos, but can also serve as a motion planner that steers toward desired outcomes or rewards. Thanks to its flexibility, Diffusion Forcing can uniquely generate plans with varying horizon, perform tree search, and incorporate the intuition that the distant future is more uncertain than the near future. In the task of solving a 2D maze, Diffusion Forcing outperformed six baselines by generating faster plans leading to the goal location, indicating that it could be an effective planner for robots in the future.Across each demo, Diffusion Forcing acted as a full sequence model, a next-token prediction model, or both. According to Chen, this versatile approach could potentially serve as a powerful backbone for a “world model,” an AI system that can simulate the dynamics of the world by training on billions of internet videos. This would allow robots to perform novel tasks by imagining what they need to do based on their surroundings. For example, if you asked a robot to open a door without being trained on how to do it, the model could produce a video that’ll show the machine how to do it.The team is currently looking to scale up their method to larger datasets and the latest transformer models to improve performance. They intend to broaden their work to build a ChatGPT-like robot brain that helps robots perform tasks in new environments without human demonstration.“With Diffusion Forcing, we are taking a step to bringing video generation and robotics closer together,” says senior author Vincent Sitzmann, MIT assistant professor and member of CSAIL, where he leads the Scene Representation group. “In the end, we hope that we can use all the knowledge stored in videos on the internet to enable robots to help in everyday life. Many more exciting research challenges remain, like how robots can learn to imitate humans by watching them even when their own bodies are so different from our own!”<hr>Chen and Sitzmann wrote the paper alongside recent MIT visiting researcher Diego Martí Monsó, and CSAIL affiliates: Yilun Du, a EECS graduate student; Max Simchowitz, former postdoc and incoming Carnegie Mellon University assistant professor; and Russ Tedrake, the Toyota Professor of EECS, Aeronautics and Astronautics, and Mechanical Engineering at MIT, vice president of robotics research at the Toyota Research Institute, and CSAIL member. Their work was supported, in part, by the U.S. National Science Foundation, the Singapore Defence Science and Technology Agency, Intelligence Advanced Research Projects Activity via the U.S. Department of the Interior, and the Amazon Science Hub. They will present their research at NeurIPS in December.

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries.

Combining next-token prediction and video diffusion in computer vision and robotics

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/b921c890-089c-4f02-b05d-59a0c9850785?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><hr><h3 id="isPasted">Episode Notes</h3>(0:50) - <a href="https://www.wevolver.com/article/ai-helps-detect-and-monitor-infrastructure-defects">Infrastructure Defect Detecting AI</a>Become a founding reader of our newsletter: <a href="http://read.thenextbyte.com/">http://read.thenextbyte.com/</a><hr><h2 data-attrid="title" data-local-attribute="d3bn" data-ved="2ahUKEwiBhLnmu4aAAxVmwjgGHcDWC38Q3B0oAXoECEgQEQ" id="isPasted">Transcript</h2>What's up, folks? As you know, cracks in buildings and roads and bridges can cause some serious damage if they're not found early enough. Today, we're talking about a team from EPFL that's gonna fix this problem using AI that can help detect cracks automatically and let us know when we need to do repairs. I think it's interesting, so let's jump into it.I'm Daniel, and I'm Farbod. And this is the NextByte Podcast. Every week, we explore interesting and impactful tech and engineering content from Wevolver.com and deliver it to you in bite sized episodes that are easy to understand, regardless of your background.&nbsp;Daniel:&nbsp;What's up friends? Like we said, today we're talking about the future of crack detection in infrastructure and why that's so important. We're not just talking about like cracks and buildings and roads and, sometimes it makes them look ugly or ivy can grow in them. But the problem is that they can cause serious damage if not found early enough. The traditional methods we have for inspecting and looking for these cracks are really slow, really costly, really hard to do in dangerous areas. And there's a lot of human error involved with it as well. So, we're talking about some awesome research here where they're using AI to try and help find and monitor cracks, but they're even taking kind of a unique AI approach as well that doesn't require detailed pixel by pixel labeling either. So, I'm pretty stoked about this one.Farbod:&nbsp;Dude, same. I feel like infrastructure improvement has been talked about for so long. I remember being in middle school and politicians were promising the next infrastructure build and how like, our infrastructure is aging and we need to replace it. Well, before this part, I was just kind of curious. I'm like, let's pull up some stats just about the US alone, right? I don't know if you knew this, but a freeway bridge inspection can take, can cost about $5,000 and take about eight hours to do. And that doesn't take into account the detour traffic causing people pains to get to work, et cetera, et cetera. And on top of that, the way we do grading of these cracks tends to be kind of subjective. So, the person that's doing it might grade it more, I don't know, liberally than the person that did last year. So that person might've thought things were going bad. This person says everything's good. So that kind of creates a weird situation where we can't track how bad it's getting super well.Daniel:&nbsp;Or can you trust the data that comes from these expectations? When you spend this much money and you spend this much time, you may not be even able to trust the data that comes out of it.Farbod: And to just make matters worse, there's something like one out of every three bridges in the US needs repair or replacement. So, one, you don't know if things are actually bad. Two, it seems like at least one third are kind of bad, maybe more. So not super comfortable about driving over a bridge at this point, but I saw something interesting about drones. Drones have had great improvement over the past couple of years. People were saying on average it's about 40% cheaper. So that makes it more affordable for local governments to take them on. But still, I'm guessing no one is super excited about shelling out over $3,000 to inspect every bridge they got. So yeah, sticky situation to be in.Daniel:&nbsp;Well, and just a quick throwback here. I would say like in the broader realm of infrastructure detection, both of our capstone projects, when we were graduating our undergrad program was related to using robotics and automation to try and detect issues with infrastructure before they progress to something that becomes a catastrophic failure. And that's not to say that we're experts in this topic at all. I think we were just at the tip of the iceberg here, but it is interesting to note that in separate graduating classes for about, and I both had this capstone project because it's such a big problem and we're still sitting here saying, maybe now we finally got some AI solution, which can help with this in a more robust manner, which is exciting for both of us, I think on a personal note, to see this field start to progress.Farbod: Dude, absolutely. Back when we were young men, which we're not anymore, at least my team, we were toying around with the idea of using AI, but even then, there wasn't a whole lot of excitement around it. Specifically, we were looking at corrosion, but now it looks like the game has completely shifted. We're going to EPFL for this episode, but if we can transition to the secret sauce, these folks are talking about how AI can actually completely remove the human element here and in the best way possible. We've been talking about subjectivity, how that can result in variance from person to person, and if a crack is getting better or worse. These folks are saying, hey, we can just train a very simple model that can tell us if there's a crack or if there's no crack, like binary. Once it learns how to detect the crack well, then it's just a matter of saying “cool, where are you seeing the crack and how is it changing over time?” Which in the world of like, you know, computer vision is a fairly simple and straightforward analysis. And the main application they're looking at for this is railroads, right? They're saying, railroads are used very often if there's cracks that grow that can cause catastrophic failures. So how do we maintain this and like monitor it on a regular basis? Well, they're like trains are passing through this bridges and stuff. What if they were the ones that were doing the analysis completely autonomously, no added costs, no added time, no requirement for shut down.Daniel: It's not sending people on harnesses to rappel from these. Honestly, you should just check on the link in the show notes, click for no other reason than to see the beautiful photo of this train on a railroad bridge going across a canyon in Switzerland where, and I wanted to confirm this, I wasn't 100% sure. The Matterhorn-Gothard bond network. I'm pretty sure Nellie and I took this train on our honeymoon, which is pretty cool.Farbod: Wow. That's special.Daniel:&nbsp;If not something very similar because all of Switzerland is full of beautiful trains and bridges and mountains, but I'm pretty sure we went on this exact railroad section where they're starting testing, but yeah, if for no other reason, check out this photo, but for me to get back to my main point, this is a bridge suspended several stories above the bottom of a canyon. It would cost a lot and you'd have to pay someone a lot to be able to visually inspect all of that manually, like as a person by hand. And what they're trying to do now is instead send a drone, but not in the form of the quadcopter drone that you think of probably when we say drone. The solution is, send a drone rail car, right? An instrumented rail car down this railroad and try and collect a bunch of data on, where are their cracks? Are they propagating? Are they staying the same? Which ones need to be addressed? Continuing to understand and map the ever evolving, let's say ecosystem around all these bridges and this infrastructure and understand when there's issues and indicate whether they need further inspection or whether they need immediate repair, et cetera.Farbod:&nbsp;Dude, totally agree. And the solution seems so simple when you say it out loud, but it really is, I don't know, I guess the best solutions are the simple ones. And what it made me think of is obviously this is great for the railroad system. But then we've been talking about like as a commuter that's going in a tunnel. That one tunnel in Boston I always hate it gives me the creeps when I'm going through it or when I'm going through bridges to Maryland and stuff like that. It would be great if there was something that could do the analysis there and in my mind, I'm like we're definitely a long way away from having autonomous vehicles doing these trips for us. But Uber and Lyft, they're drivers that are operating every single day that are doing these normal routes that people take to go to work and whatnot. I wonder if there was an opportunity there to say, hey, while you're doing this, mount this device on the back of your car so that whenever your GPS detects you're going from the beginning of the tunnel to the back of the tunnel, you do a scan and then that data uploads to a cloud somewhere and we can do analysis over time to see if we need to patch any cracks or what?Daniel: I'm like wondering if you read my mind here. I went to buses, because I'm like, the department of transportation in a city may also own the buses, right? So, they're in charge of the bridges. They're also in charge of the buses. Wonder if they can just put a bunch of instruments on top of a bus. In my mind, it's like the closest analogy to a rail car. Put a bunch of this infrastructure on top of a bus and this instrumentation, see if you can analyze the infrastructure around us using that same method as well. But I also want to jump a little bit into the specific AI methods they're using. And I'm not an expert here by any means, but I took some notes from the paper that I think are interesting. That there have previously tried to do AI classification of images, but it's too computationally intensive to justify the cost. So, if you were to use AI to try and evaluate millions of photos of an entire bridge. It may actually cost more than actually sending a human down there to do it. Because one of the things that I had to do is classify each pixel by pixel, trying to understand whether it was light or if it's dark and look at contrast to understand what is a crack, then it classifies a crack and that took a lot of computational energy. One of the things that they did is they, they train their AI ahead of time with images labeled as crack or no crack. And then it uses explainable AI is what they say is their unique method here to try and look at a new photo and circle, basically highlight a cracked area and create a map showing where cracks are and where they aren't. And essentially what it's doing is it's just taking photo after photo after photo and trying to trace the area around where it thinks the crack is. And over a period of time, it's trying to look at the same photo from the same location the last time this train came through and look at whether this crack got bigger or if it stayed the same. And if cracks are serially getting bigger and bigger and bigger from these images, it warrants sending someone down there, an expert down there to take a look at it. And one of the things that they mentioned is this cut the processing and labeling effort, as opposed to previous AI methods by about 90%, which makes it cheaper and faster and worthwhile to implement in an autonomous manner. It's not quite as accurate as these full AI models, but they said it can still help directionally track crack growth. And they think that they can hone it essentially and get rid of more of the area that they're experiencing right now. But essentially the game plan I guess is to go test this out in the field, get more learnings, understand more but in practice even if it has the same performance that it has today you can use this as kind of like a non-invasive constant monitoring where it looks at the rails, it looks at the ties, it looks at the retaining walls and tries to understand if there's anything wrong with the railroad. And you could have this on every train or one of every 10 trains passing through this railroad. Even though I think right now it's got 35% error, which again, it sounds high compared to most AI models, but if you're telling me, you can tell me 65% of the time, whether a crack's growing or not. If I've got two or three trains that pass through over a series of a week and show that this crack indicates that this crack may be growing, the playbook here is not to say, all right, let's shut down the bridge and tear it down. The playbook here could be, let's go send one of those expert inspectors that we used to have doing this job full time and had to shut down the railroad to do now that we know that there's something that's worth further evaluation, let's send one of those crack experts down there to understand what's going on and see whether it needs repair.Farbod: And this is a great example, at least in my opinion of complimentary automation, right? Like here's a task that was bad for people to do manually. And some places are like in America, it's just straight up not done at times. Now you have automation that's taking care of the tedious part. And then you have the human that's taking on the challenging part, which is, hey, we're like 90% sure there is a crack, please go out there and now characterize it and give us feedback on if it's catastrophic or if, you know, it's a simple crack that needs to be fixed, etc, etc. So, I feel like this is the best use of every technology or every resource, think of the human expert as a resource. And this is kind of going against the AI is going to get rid of all jobs mentality.Daniel: Well, and I think what we, the intended end here is to be able to do continuous monitoring, collect data all the time to understand which regions are susceptible to cracks, and then use that as a heat map essentially to show someone who's again, got expertise in this field to go, this is the area that you need to inspect. This is the area that you need to monitor. This is the area that may need potential repair as opposed to the other portions of track that maybe didn't need as much attention and they were getting it before because you had to go check every single mile of this thing manually. And maybe you used to only check one, every single mile of this thing manually, one out of every five years. And now we could have a train passing on it five times a day that tells us how the railroad's doing. So, I appreciate here, like you said, it's a complimentary solution between automation and humans. But it also gets us an immediate benefit, which is a lot more data, a lot more consistently on how our infrastructure is doing and when it needs to be repaired.Farbod: Totally agree, man. You want to do us a favor and wrap up the episode?Daniel:&nbsp;I will. What if we could stop big cracks in buildings and roads before they cause big problems, I'm thinking of bridges collapsing or bridges needing to be shut down because they need repair or rip, cars getting derailed. These are big problems and we're talking today about a new AI method that helps find and track these cracks automatically without needing a ton of expensive detailed data. Essentially, they're getting the job done with 90% less effort. That's what their new tech promises. And they use this system called explainable AI that spots cracks with simple image labels, saving time and money, keeping an eye on crack growth to let experts know where they need to address cracks and where we might need repairs in the future. I think it's a lot like, I don't know, Google Maps navigation or some sort of traffic heat map, but it tells us where most areas are prone to cracks and where we need repairs so that folks know where they need to focus their efforts.Farbod:&nbsp;Money.Daniel:&nbsp;Thanks, my dude.Farbod:&nbsp;You killed it, dude. All right, folks, thank you so much for listening. And as always, we'll catch you in the next one.Daniel:&nbsp;Peace.<hr>As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.To learn more about this show, please visit our<a href="https://www.wevolver.com/profile/the.next.byte">&nbsp;shows page</a>. By following the page, you will get automatic updates by email when a new show is published. Be sure to give us a follow and review on<a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" target="_blank" rel="nofollow">&nbsp;Apple</a> podcasts,<a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" target="_blank" rel="nofollow">&nbsp;Spotify</a>, and most of your favorite podcast platforms!--The Next Byte: We're two engineers on a mission to simplify complex science &amp; technology, making it easy to understand. In each episode of our show, we dive into world-changing tech (such as AI, robotics, 3D printing, IoT, &amp; much more), all while keeping it entertaining &amp; engaging along the way.

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems

Podcast: Saving America's Collapsing Infrastructure

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/35ecb899-3a2c-45e8-9d17-7a8c87d944ba?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Most doctors go into medicine because they want to help patients. But today’s health care system requires that doctors spend hours each day on other work — searching through electronic health records (EHRs), writing documentation, coding and billing, prior authorization, and utilization management — often surpassing the time they spend caring for patients. The situation leads to physician burnout, administrative inefficiencies, and worse overall care for patients.Ambience Healthcare is working to change that with an AI-powered platform that automates routine&nbsp;tasks for clinicians before, during, and after patient visits."We build co-pilots to give clinicians AI superpowers," says Ambience CEO Mike Ng MBA ’16, who co-founded the company with Nikhil Buduma ’17. "Our platform is embedded directly into&nbsp;EHRs to free up clinicians to&nbsp;focus on what matters most, which is providing the best possible patient care."Ambience’s suite of products handles pre-charting and real-time AI scribing, and assists with navigating the thousands of rules to select the right insurance billing codes. The platform can also send after-visit summaries to patients and their families in different languages to keep everyone informed and on the same page.Ambience is already being used across roughly 40 large institutions such as UCSF Health, the Memorial Hermann Health System, St. Luke’s Health System, John Muir Health, and more.&nbsp;Clinicians leverage Ambience in dozens of languages and more than&nbsp;100 specialties and subspecialties, in settings&nbsp;like the emergency department, hospital inpatient settings,&nbsp;and the oncology ward.The founders say clinicians using Ambience save two to three hours per day on documentation,&nbsp;report lower levels of burnout, and develop higher-quality relationships with their patients.<h2>From problem to product to platform</h2>Ng worked in finance until getting an up-close look at the health care system after he fractured his back in 2012. He was initially misdiagnosed and put on the wrong care plan, but he learned a lot about the U.S. health system in the process, including how the majority of clinicians’ days are spent documenting visits, selecting billing codes, and completing other administrative tasks. The average clinician only spends 27 percent of their time on direct patient care.In 2014, Ng decided to enter the MIT Sloan School of Management. During his first week, he attended the “t=0” celebration of entrepreneurship hosted by the Martin Trust Center for MIT Entrepreneurship, where he met Buduma.&nbsp;The pair became fast friends, and they ended up taking classes together including 15.378 (Building an Entrepreneurial Venture) and 15.392 (Scaling Entrepreneurial Ventures).“MIT was an incredible training ground to evaluate what makes a great company and learn the foundations of building a successful company,” Ng says.Buduma had gone through his own journey to discover problems with the health care system. After immigrating to the U.S. from India as a child and battling persistent health issues, he had watched his parents struggle to navigate the U.S. medical&nbsp;system. While completing his bachelor’s degree at MIT, he was also steeped in the AI research community and wrote an early textbook on modern AI and deep learning.In 2016, Ng and Buduma&nbsp;founded&nbsp;their first company in San Francisco — Remedy Health — which operated its own AI-powered health care platform. In the process of hiring clinicians, taking care of patients, and implementing technology themselves, they developed an even deeper appreciation for the challenges that health care organizations face.During that time, they also got an inside look at advances in AI. Google’s Chief Scientist Jeff Dean, a major investor in Remedy and now in Ambience, led a research group inside of Google Brain to invent the transformer architecture. Ng and Buduma say they were among the first to put transformers into production to support their own clinicians at Remedy. Subsequently, several of their friends and housemates went on to start the large language model group within OpenAI. Their friends’ work formed the research foundations that ultimately led to ChatGPT. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;“It was very clear that we were at this inflection point where we were going to have this class of general-purpose models that were going to get exponentially better,” Buduma says. “But I think we also noticed a big gap between those general-purpose models versus what actually would be robust enough to work in a clinic. Mike and I decided in 2020 that there should be a team that specifically focused on fine-tuning these models for health&nbsp;care and medicine.”The founders started Ambience by building an AI-powered scribe that works on phones and laptops to record the details of doctor-patient visits in a HIPAA-compliant system that preserves patient privacy. They quickly saw that the models needed to be fine-tuned for each area&nbsp;of medicine, and they slowly expanded specialty coverage one by one in a multiyear process.The founders also realized their scribes needed to fit within back-office operations like insurance coding and billing.“Documentation isn’t just for the clinician, it's also for the revenue cycle team,” Buduma says. “We had to go back and rewrite all of our algorithms to be coding-aware. There are literally tens of thousands of coding rules that change every year and differ by specialty and contract type.”From there, the founders built out models for clinicians to make referrals and to send comprehensive summaries of visits to patients.“In most care settings before Ambience, when a patient and their family left the clinic, whatever the patient and their family wrote down was what they remembered from the visit,” Buduma says. “That’s one of the features that physicians love most, because they are trying to create the best experience for patients and their families. By the time that patient is in the parking lot, they already have a really robust, high-quality summary of exactly what you talked about and all the shared decision-making around your visit in their portal.”<h2>Democratizing health care</h2>By improving physician productivity, the founders believe they’re helping the health care&nbsp;system manage a chronic shortage of clinicians that’s expected to grow in coming years.“In health care, access is still a huge problem,” Ng says. “Rural Americans have a 40 percent higher risk of preventable hospitalization, and half of that is attributed to a lack of access to specialty care.”With Ambience already helping health systems manage razor-thin margins by streamlining administrative tasks,&nbsp;the founders have a longer-term vision to help&nbsp;increase access to the best clinical information across the country.“There’s a really exciting opportunity to make expertise at some of the major academic medical centers more democratized across the U.S.,” Ng says. “Right now, there’s just not enough specialists in the U.S. to support our rural populations. We hope to help scale the knowledge of the leading specialists in the country through an AI infrastructure layer, especially as these models become more clinically intelligent.”

Ambience is being used across roughly 40 health systems in the U.S. by clinicians in over 100 subspecialties. Image: iStock

Alumni-founded Ambience Healthcare automates routine tasks for clinicians before, during, and after patient visits.

Equipping doctors with AI co-pilots

Robots, combined with artificial intelligence (AI), are transforming industries such as manufacturing, healthcare, and research. 

AI-enabled robots and the importance of force sensing

‘Industry 4.0’—the next&nbsp;stage of the Industrial Revolution—leans heavily on the Industrial Internet of Things (IIoT) to support large scale automation of traditional manufacturing practices. In turn, today’s IIoT is founded on wireless machine-to-machine (M2M) communication,&nbsp;<a href="https://www.nordicsemi.com/Products/Technologies/Edge-AI" target="_blank" rel="noreferrer noopener">edge computing</a>, and Machine Learning (ML).&nbsp;Wireless networks&nbsp;using these technologies&nbsp;link machine tools,&nbsp;programmable logic controllers (PLCs), sensors, gateways, and the Cloud, enabling every part of the factory to gather and process data and share information with every other part as well as the Internet. This&nbsp;deep&nbsp;mine of information is enabling engineers to revolutionize how everything is manufactured - from cans to cars, screws to smartphones, and jigsaws to jet turbines.&nbsp;But there is always room for improvement. For example, machines can be further enhanced to improve yield and quality while using less material, while engineers can design products for greater sustainability. A new generation of IIoT solutions promises to make this happen.&nbsp;<h2>IIoT&nbsp;and M2M&nbsp;increase manufacturing flexibility&nbsp;</h2>Wireless technology not only cuts the cost of implementing IIoT connectivity it also makes it easy to reconfigure networks as the factory adapts and expands. Such technology not only changes the way products are made but also how they’re designed.&nbsp;&nbsp;Now, connectivity links the front office to the factory floor using M2M communications, allowing computer aided design (CAD) tools to talk to machine tools to directly program them to make parts. And machine tools can speak to CAD to let them know where the bottlenecks are in the manufacturing process such that products can be redesigned for simpler manufacture without compromising function.&nbsp;The results are an increase in productivity and reduction in product failures - bringing sustained cost savings and environmental benefits.&nbsp;<h2>Edge computing and&nbsp;ML&nbsp;optimize&nbsp;production in&nbsp;smart factories&nbsp;</h2>Once mass production is up-and-running, wireless technology makes the modern factory a highly controlled and optimized environment. Things rarely go wrong. But rarely doesn’t mean never. Today’s wireless&nbsp;<a href="https://www.nordicsemi.com/Products/Wireless/Bluetooth-Low-Energy" target="_blank" rel="noreferrer noopener">Bluetooth LE</a>&nbsp;Systems-on-Chip (SoCs) from Nordic incorporate powerful embedded processors to perform edge processing, looking for the infrequent trends that indicate things might be changing for the worse. This edge computing is increasingly reliant on ML models constantly adapting to spot deviations in sensor trends and forwarding the key information for further action.&nbsp;&nbsp;Such models can help to pre-empt issues that might arise due to these external factors—for instance, an increase in humidity caused by workers arriving for the day, air flow from open windows and doors, and changes in temperature throughout the day and night—and adjust machine settings in advance of those factors&nbsp;impacting&nbsp;manufacturing processes. And dedicated vibration and acoustic sensors can&nbsp;monitor&nbsp;the machine tools to make sure they are in the best of health. Any unusual vibration, temperature increase or increase in power consumption can be reported ahead of a breakdown for early maintenance – thus preventing an unscheduled and expensive hiatus in production.&nbsp;<h2>Tech to&nbsp;power the industrial future&nbsp;</h2>But while modern factories are impressive examples of the power of connectivity, standing still is not an option. The Japanese concept of Kaizen (“improvement”) epitomizes this strategy. Kaizen guides continuous effort across the factory to improve programs and processes while eliminating waste and redundancy. Kaizen has proved so successful that it has spread throughout the world.&nbsp;To continue the improvement process, factories will need even better wireless technology than they have today. Nordic is working hard and investing deeply to build and supply the next generation of IIoT technology. Several multiprotocol SoCs and Systems-in-Package (SiPs) from Nordic’s product range already support TinyML, a framework for running embedded ML on constrained devices. But there’s much more to come.&nbsp;The company’s new&nbsp;<a href="https://www.nordicsemi.com/Products/nRF54H20">nRF54H20 SoC</a>, for example, integrates multiple Arm Cortex-M33 processors, clocked up to 320 MHz, together with multiple RISC-V coprocessors. The higher clock speeds of the SoC’s application processors allow for reduced processing latency for AI operations, when compared to Nordic’s nRF5340 SoC.&nbsp;&nbsp;Better yet the nRF54H20 brings increased efficiency—cutting the power consumption required to run equivalent operations—because it is fabricated using the cutting-edge 22-nm process node. In addition, Nordic’s next-generation multiprotocol radio offers improvements in both TX power and RX sensitivity, enabling efficient and reliable transfer of information from embedded ML applications to Cloud connected devices, or between different nodes in a network. This makes the SoC ideal for next generation IIoT applications.&nbsp;&nbsp;The nRF54H20 will be supported in Nordic’s&nbsp;<a href="https://www.nordicsemi.com/Products/Technologies/Edge-AI/ML-Studio?lang=en#infotabs">ML Studio</a>&nbsp;alongside the nRF52, nRF53, nRF91 Series products already supported.&nbsp;&nbsp;<h2>The beginning of a revolution&nbsp;</h2>Wireless tech is&nbsp;booming&nbsp;in&nbsp;IIoT&nbsp;applications.&nbsp;According to&nbsp;ABI Research, the Bluetooth-enabled industrial devices market&nbsp;alone&nbsp;is expected to grow from 143 million annual unit shipments in 2023 to over 611 million by 2028, achieving a CAGR of 34 percent over the five-year forecast&nbsp;period[1]. But even with this impressive growth things are just getting&nbsp;started; Industry 4.0 is set to have a larger impact on global manufacturing than all three&nbsp;previous&nbsp;major leaps in the industrial revolution combined.&nbsp;<hr><h3>References </h3>1. https://www.abiresearch.com/market-research/product/7783366-how-can-bluetooth-technology-enable-digita/&nbsp;

‘Industry 4.0’—the next stage of the Industrial Revolution—leans heavily on the Industrial Internet of Things (IIoT) to support large scale automation of traditional manufacturing practices.

Machine Learning is transforming the Industrial Internet of Things

Simulating digital actions to teach robots new tasks is also time-consuming for humans, though. Cutting those minutes in half, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) PhD student Lirui Wang and his colleagues’ new “GenSim2” framework uses multimodal and reasoning large language models (LLMs that process and produce text, images, and other media) to supersize training data for robots. The researchers combined the powers of multimodal LLM GPT-4V (which can draw better inferences about text and images) and reasoning LLM OpenAI o1 (which can “think” before answering) to take ten real-world videos of tasks and generate 100 new, simulated action videos.GenSim2 can then convert task names into task descriptions and then to task code, which can be simulated into a sequence of actions for a robot to execute. The approach could eventually assist home robots with tasks like figuring out each step needed to reheat your breakfast, including opening a microwave and placing bread in a toaster. It could also help in manufacturing and logistics settings one day, where a machine may need to transport new materials in several steps. This framework is a sequel to Wang’s earlier work,&nbsp;<a href="https://www.csail.mit.edu/news/using-llms-code-new-tasks-robots" target="_blank">“GenSim,”</a> which used LLMs to encode new pick-and-place tasks for robots. He wanted to expand his approach to more dexterous activities with more complex object categories, like opening a box or closing a safe.&nbsp; “To plan these more complicated chores in robotics, we need to figure out how to solve them,” says Wang. “This planning problem was not present in GenSim, since the tasks were much simpler, so we only needed “blind” LLMs. With GenSim2, we integrated the logic model GPT-4V, which teaches multimodal models to “see” by analyzing image inputs with better reasoning skills. Now, we can code the simulation task, and then generate plans in seconds.”<h2 dir="ltr">The nuts and bolts of GenSim2</h2>First, you prompt an LLM like GPT-4 to generate a novel task plan like “place a ball in a box,” including images, assets, and keypoints (or specific points in an image). From there, GPT-4V reviews these details and concisely encodes which poses and actions are needed to execute the task. Humans can provide feedback about this plan to GPT-4V, and then it will refine its outline. Finally, a motion planner simulates those actions into videos, generating new training data for the robot.To convert these plans into actions, the researchers also designed a new architecture called the “proprioceptive point-cloud transformer” (PPT). PPT converts language, point cloud (data points within a 3D space), and proprioception inputs into a final action sequence. This allows a robot to learn to imitate video simulations and generalize to objects it hasn’t seen before.<h2 dir="ltr">Lights, camera, action plan!</h2>GenSim2’s souped-up approach generated data for 100 articulated tasks with 200 objects. Among these, the system simulated 50 long-horizon tasks, such as securing gold in a safe and preparing breakfast. Compared to the generative robotic agent and baseline “<a href="https://arxiv.org/abs/2311.01455" target="_blank">RoboGen</a>,” GenSim2 had a 20% better success rate with generating and planning primitive tasks, while also being more reliable with long-horizon ones. The researchers note that having multimodal models that can reason about visual inputs gave them the edge.Another intriguing finding: it only took humans about four minutes on average to verify robotic plans — half of how long it took them to design a task manually. Human efforts included labeling keypoints in the motion planner and giving feedback to help the multimodal language model improve its plans.In real-world experiments, GenSim2 successfully helped plan tasks for a robot, like opening a laptop and closing a drawer. When it trained on both simulation and real data to develop its robotic policy, the framework had a better success rate than either one standalone. This reduces the required effort to collect large amounts of data in the real world. While GenSim2 is a more intricate, advanced follow-up to its predecessor, the researchers note that they’d like it to plan and simulate robotic tasks with even less human intervention. Currently, it struggles to reliably create and code meaningful tasks on its own.&nbsp;Wang also notes that while it’s a step forward in achieving automated task generation, the researchers intend to make the system more advanced. To do this, they plan to increase task complexity and diversity through advanced multimodal agents and generate 3D assets.“Scaling up robot data has been a major challenge in creating generalizable robot foundation models,” says Yunzhu Li, Assistant Professor of Computer Science at Columbia University, who wasn’t involved in the paper. “GenSim2 addresses this by developing a scalable framework for data and action generation, using a combination of simulation, GPT-4, and sim-to-real transfer. I’m excited to see how this work could spark a “GPT moment” for robotics by effectively expanding the data available for robots.” Wang wrote the paper with Tsinghua University assistant professor Huazhe Xu and PhD student Pu Hua, Shanghai Jiao Tong University professor Weinan Zhang and PhD students Minghuan Liu and Yunfeng Lin, and University of California at San Diego student researcher Annabella Macaluso. Their work was supported, in part, by Amazon and the company’s Greater Boston Tech Initiative. The researchers will present their work at the Conference on Robot Learning next month in Munich, Germany.

For robots, simulation is a great teacher for learning long-horizon (multi-step) tasks — especially compared to how long it takes to collect real-world training data.

Multimodal and reasoning LLMs supersize training data for dexterous robotic tasks

Imagine you’re tasked with sending a team of football players onto a field to assess the condition of the grass (a likely task for them, of course). If you pick their positions randomly, they might cluster together in some areas while completely neglecting others. But if you give them a strategy, like spreading out uniformly across the field, you might get a far more accurate picture of the grass condition.Now, imagine needing to spread out not just in two dimensions, but across tens or even hundreds. That's the challenge MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers are getting ahead of. They've developed an AI-driven approach to “low-discrepancy sampling,” a method that improves simulation accuracy by distributing data points more uniformly across space.A key novelty lies in using graph neural networks (GNNs), which allow points to “communicate” and self-optimize for better uniformity. Their approach marks a pivotal enhancement for simulations in fields like robotics, finance, and computational science, particularly in handling complex, multidimensional problems critical for accurate simulations and numerical computations.“In many problems, the more uniformly you can spread out points, the more accurately you can simulate complex systems,” says T. Konstantin Rusch, lead author of the new paper and MIT CSAIL postdoc. “We've developed a method called Message-Passing Monte Carlo (MPMC) to generate uniformly spaced points, using geometric deep learning techniques. This further allows us to generate points that emphasize dimensions which are particularly important for a problem at hand, a property that is highly important in many applications. The model’s underlying graph neural networks lets the points 'talk' with each other, achieving far better uniformity than previous methods.”Their work was&nbsp;<a href="https://www.pnas.org/doi/full/10.1073/pnas.2409913121" target="_blank">published in the September issue of the&nbsp;Proceedings of the National Academy of Sciences</a>.<h2 dir="ltr">Take me to Monte Carlo</h2>The idea of Monte Carlo methods is to learn about a system by simulating it with random sampling. Sampling is the selection of a subset of a population to estimate characteristics of the whole population. Historically, it was already used in the 18th century, &nbsp;when mathematician Pierre-Simon Laplace employed it to estimate the population of France without having to count each individual.Low-discrepancy sequences, which are sequences with low discrepancy, i.e., high uniformity, such as Sobol’, Halton, and Niederreiter, have long been the gold standard for quasi-random sampling, which exchanges random sampling with low-discrepancy sampling. They are widely used in fields like computer graphics and computational finance, for everything from pricing options to risk assessment, where uniformly filling spaces with points can lead to more accurate results.&nbsp; The MPMC framework suggested by the team transforms random samples into points with high uniformity. This is done by processing the random samples with a GNN that minimizes a specific discrepancy measure.One big challenge of using AI for generating highly uniform points is that the usual way to measure point uniformity is very slow to compute and hard to work with. To solve this, the team switched to a quicker and more flexible uniformity measure called L2-discrepancy. For high-dimensional problems, where this method isn’t enough on its own, they use a novel technique that focuses on important lower-dimensional projections of the points. This way, they can create point sets that are better suited for specific applications.The implications extend far beyond academia, the team says. In computational finance, for example, simulations rely heavily on the quality of the sampling points. “With these types of methods, random points are often inefficient, but our GNN-generated low-discrepancy points lead to higher precision,” says Rusch. “For instance, we considered a classical problem from computational finance in 32 dimensions, where our MPMC points beat previous state-of-the-art quasi-random sampling methods by a factor of four to 24.”<h2 dir="ltr">Robots in Monte Carlo</h2>In robotics, path and motion planning often rely on sampling-based algorithms, which guide robots through real-time decision-making processes. The improved uniformity of MPMC could lead to more efficient robotic navigation and real-time adaptations for things like autonomous driving or drone technology. “In fact, in a recent preprint, we demonstrated that our MPMC points achieve a fourfold improvement over previous low-discrepancy methods when applied to real-world robotics motion planning problems,” says Rusch.“Traditional low-discrepancy sequences were a major advancement in their time, but the world has become more complex, and the problems we're solving now often exist in 10, 20, or even 100-dimensional spaces,” says Daniela Rus, CSAIL director and MIT professor of electrical engineering and computer science. “We needed something smarter, something that adapts as the dimensionality grows. GNNs are a paradigm shift in how we generate low-discrepancy point sets. Unlike traditional methods, where points are generated independently, GNNs allow points to 'chat' with one another so the network learns to place points in a way that reduces clustering and gaps — common issues with typical approaches.”Going forward, the team plans to make MPMC points even more accessible to everyone, addressing the current limitation of training a new GNN for every fixed number of points and dimensions.“Much of applied mathematics uses continuously varying quantities, but computation typically allows us to only use a finite number of points,” says Art B. Owen, Stanford University professor of statistics, who wasn’t involved in the research. “The century-plus-old field of discrepancy uses abstract algebra and number theory to define effective sampling points. This paper uses graph neural networks to find input points with low discrepancy compared to a continuous distribution. That approach already comes very close to the best-known low-discrepancy point sets in small problems and is showing great promise for a 32-dimensional integral from computational finance. We can expect this to be the first of many efforts to use neural methods to find good input points for numerical computation.”Rusch and Rus wrote the paper with University of Waterloo researcher Nathan Kirk, Oxford University’s DeepMind Professor of AI and former CSAIL affiliate Michael Bronstein, and University of Waterloo Statistics and Actuarial Science Professor Christiane Lemieux. Their research was supported, in part, by the AI2050 program at Schmidt Futures, Boeing, the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator, the Swiss National Science Foundation, Natural Science and Engineering Research Council of Canada, and an EPSRC Turing AI World-Leading Research Fellowship.

MIT CSAIL researchers developed an AI-driven approach to “low-discrepancy sampling,” a method that improves simulation accuracy by distributing data points more uniformly across space.

How AI is improving simulations with smarter sampling techniques

Imagine having to straighten up a messy kitchen, starting with a counter littered with sauce packets. If your goal is to wipe the counter clean, you might sweep up the packets as a group. If, however, you wanted to first pick out the mustard packets before throwing the rest away, you would sort more discriminately, by sauce type. And if, among the mustards, you had a hankering for Grey Poupon, finding this specific brand would entail a more careful search.MIT engineers have developed a method that enables robots to make similarly intuitive, task-relevant decisions.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727764894016-MIT-Scene-Tasks-01-press_0.jpg" style="width: 100%px;" class="fr-fic fr-dib">From left to right: team members Lukas Schmid, Nathan Hughes, Dominic Maggio, Yun Chang, and Luca Carlone. Credit: Andy RyanThe team’s new approach, named Clio, enables a robot to identify the parts of a scene that matter, given the tasks at hand. With Clio, a robot takes in a list of tasks described in natural language and, based on those tasks, it then determines the level of granularity required to interpret its surroundings and “remember” only the parts of a scene that are relevant.In real experiments ranging from a cluttered cubicle to a five-story building on MIT’s campus, the team used Clio to automatically segment a scene at different levels of granularity, based on a set of tasks specified in natural-language prompts such as “move rack of magazines” and “get first aid kit.”The team also ran Clio in real-time on a quadruped robot. As the robot explored an office building, Clio identified and mapped only those parts of the scene that related to the robot’s tasks (such as retrieving a dog toy while ignoring piles of office supplies), allowing the robot to grasp the objects of interest.Clio is named after the Greek muse of history, for its ability to identify and remember only the elements that matter for a given task. The researchers envision that Clio would be useful in many situations and environments in which a robot would have to quickly survey and make sense of its surroundings in the context of its given task.“Search and rescue is the motivating application for this work, but Clio can also power domestic robots and robots working on a factory floor alongside humans,” says Luca Carlone, associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro), principal investigator in the Laboratory for Information and Decision Systems (LIDS), and director of the MIT SPARK Laboratory. “It’s really about helping the robot understand the environment and what it has to remember in order to carry out its mission.”The team details their results in a <a href="https://dspace.mit.edu/handle/1721.1/157072" target="_blank">study appearing today</a> in the journal Robotics and Automation Letters. Carlone’s co-authors include members of the SPARK Lab: Dominic Maggio, Yun Chang, Nathan Hughes, and Lukas Schmid; and members of MIT Lincoln Laboratory: Matthew Trang, Dan Griffith, Carlyn Dougherty, and Eric Cristofalo.<h2>Open fields</h2>Huge advances in the fields of computer vision and natural language processing have enabled robots to identify objects in their surroundings. But until recently, robots were only able to do so in “closed-set” scenarios, where they are programmed to work in a carefully curated and controlled environment, with a finite number of objects that the robot has been pretrained to recognize.In recent years, researchers have taken a more “open” approach to enable robots to recognize objects in more realistic settings. In the field of open-set recognition, researchers have leveraged deep-learning tools to build neural networks that can process billions of images from the internet, along with each image’s associated text (such as a friend’s Facebook picture of a dog, captioned “Meet my new puppy!”).From millions of image-text pairs, a neural network learns from, then identifies, those segments in a scene that are characteristic of certain terms, such as a dog. A robot can then apply that neural network to spot a dog in a totally new scene.But a challenge still remains as to how to parse a scene in a useful way that is relevant for a particular task.“Typical methods will pick some arbitrary, fixed level of granularity for determining how to fuse segments of a scene into what you can consider as one ‘object,’” Maggio says. “However, the granularity of what you call an ‘object’ is actually related to what the robot has to do. If that granularity is fixed without considering the tasks, then the robot may end up with a map that isn’t useful for its tasks.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727764931766-MIT-Scene-Tasks-03-press.jpg" style="width: 100%px;" class="fr-fic fr-dib">Clio quickly segments a scene such as a cluttered cubicle (pictured), and feeds the segments through an “information bottleneck” algorithm that focuses on the segments that only represent target objects relevant to the task at hand. Colored cubes represent segment regions that Clio has picked identified as task-relevant objects that a robot can focus on while ignoring the rest of the scene. Credit: Courtesy of the researchers<h2>Information bottleneck</h2>With Clio, the MIT team aimed to enable robots to interpret their surroundings with a level of granularity that can be automatically tuned to the tasks at hand.For instance, given a task of moving a stack of books to a shelf, the robot should be able to &nbsp;determine that the entire stack of books is the task-relevant object. Likewise, if the task were to move only the green book from the rest of the stack, the robot should distinguish the green book as a single target object and disregard the rest of the scene — including the other books in the stack.The team’s approach combines state-of-the-art computer vision and large language models comprising neural networks that make connections among millions of open-source images and semantic text. They also incorporate mapping tools that automatically split an image into many small segments, which can be fed into the neural network to determine if certain segments are semantically similar. The researchers then leverage an idea from classic information theory called the “information bottleneck,” which they use to compress a number of image segments in a way that picks out and stores segments that are semantically most relevant to a given task.“For example, say there is a pile of books in the scene and my task is just to get the green book. In that case we push all this information about the scene through this bottleneck and end up with a cluster of segments that represent the green book,” Maggio explains. “All the other segments that are not relevant just get grouped in a cluster which we can simply remove. And we’re left with an object at the right granularity that is needed to support my task.”The researchers demonstrated Clio in different real-world environments.“What we thought would be a really no-nonsense experiment would be to run Clio in my apartment, where I didn’t do any cleaning beforehand,” Maggio says.The team drew up a list of natural-language tasks, such as “move pile of clothes” and then applied Clio to images of Maggio’s cluttered apartment. In these cases, Clio was able to quickly segment scenes of the apartment and feed the segments through the Information Bottleneck algorithm to identify those segments that made up the pile of clothes.They also ran Clio on Boston Dynamic’s quadruped robot, Spot. They gave the robot a list of tasks to complete, and as the robot explored and mapped the inside of an office building, Clio ran in real-time on an on-board computer mounted to Spot, to pick out segments in the mapped scenes that visually relate to the given task. The method generated an overlaying map showing just the target objects, which the robot then used to approach the identified objects and physically complete the task.“Running Clio in real-time was a big accomplishment for the team,” Maggio says. “A lot of prior work can take several hours to run.”Going forward, the team plans to adapt Clio to be able to handle higher-level tasks and build upon recent advances in photorealistic visual scene representations.“We’re still giving Clio tasks that are somewhat specific, like ‘find deck of cards,’” Maggio says. “For search and rescue, you need to give it more high-level tasks, like ‘find survivors,’ or ‘get power back on.’ So, we want to get to a more human-level understanding of how to accomplish more complex tasks.”This research was supported, in part, by the U.S. National Science Foundation, the Swiss National Science Foundation, MIT Lincoln Laboratory, the U.S. Office of Naval Research, and the U.S. Army Research Lab Distributed and Collaborative Intelligent Systems and Technology Collaborative Research Alliance.

MIT’s Clio runs in real-time to map task-relevant objects in a robot’s surroundings, allowing the bot (Boston Dynamic’s quadruped robot Spot, pictured) carry out a natural language task (“pick up orange backpack”). Credit: Courtesy of the researchers

A new method called Clio enables robots to quickly map a scene and identify the items they need to complete a given set of tasks.

Helping robots zero in on the objects that matter

<section id="isPasted">Robots are widely used in the automotive industry and have started entering new application domains such as logistics in the last few years. However, current robots still face many limitations. They typically perform a single action or a fixed sequence of actions, repeating them the same way each time. To achieve greater efficiency and open new possibilities, robots must develop more human-like skills, such as fast physical interaction, spatial understanding, and fast adaptation to changes. We spoke with Alessandro Saccon, Associate Professor in nonlinear control and robotics at the department of Mechanical Engineering at the TU/e. He recently completed the<a href="http://i.am/" target="_blank" rel="noreferrer">&nbsp;</a><a href="http://i.am/" target="_blank" rel="noreferrer">I.AM project</a> that explicitly focuses on the advancement of fast physical interactions.Source:<a href="https://innovationorigins.com/en/" target="_blank" rel="noreferrer">&nbsp;Innovation Origins</a></section><section id="c393798" tabindex="-1"><h2>Why are impact-aware robots so important for humanity?</h2>“Certain jobs are not particularly suited for humans from a safety or ergonomic perspective. For instance, when handling 20-kilogram luggage at airports, working in unsafe areas of a nuclear plant, or dealing with disaster scenarios, you might prefer a machine instead. There are also various plans to send them to space for planet exploration. However, robots still statically interact with the environments when compared to us: the execution of certain key tasks is not yet possible or the execution is too slow. That’s why, in our project, we aimed to develop impact-aware robots. That means a robot has to learn to predict and react to what happens when it comes into fast contact with heavy objects in the environment.”<h2>What makes those robots different from the traditional robots we've known forever?</h2>“Typical robots are not designed to interact dynamically with their environment; making fast contact with surroundings is generally avoided at all costs. There is a very large number of scientific papers in the robotics literature whose focus is&nbsp;collision avoidance. In the I.AM project, we targeted instead&nbsp;collision exploitation. We looked into how the robots can, for example, pick up heavy objects quickly while ensuring that the execution of this type of motion remains reliable, despite disturbances and perception inaccuracies. An object might be heavier than the robot anticipated, or it assumes that an object is at a certain location, but it’s slightly off—maybe even by a few centimeters. How do you make these movements robust despite such uncertainties? That’s one of the things that we have been researching in depth.”<h2>Practically speaking, what main activities were involved in your project?</h2>“The project employed first-principle physics calculations, using basic concepts such as mass and friction, along with software simulations to identify discrepancies between mathematical models and real-world events. Although simulations can never perfectly replicate robot behavior, we improved and most importantly understood how to still use these algorithms for controlling the robots. We did this by taking real-time measurements of robots interacting with various objects in different scenarios. It’s an iterative cycle where you develop and implement a theoretical robot control algorithm in a simulation, evaluate the results, and compare them with real-world outcomes.”<h2>Can you highlight some key findings from this project?</h2>“We discovered how we can make a robot reliably and swiftly grab a heavy object with two arms, by developing a new control algorithm that respects the natural impact dynamics. We also understood how to use software simulations to obtain predictions that can be used for this purpose or other impact tasks.”While working on this project, I also further appreciated how complex movements and spatial perception come so naturally to us humans. We academic researchers are now working very hard on hardware development, spatial perception, and planning—especially the ability to understand the environment in real time and quickly decide what to do next, also in case of failure. This is one of the grand challenges in modern robotics. These actions are natural and intuitive for us, yet we don’t fully understand how we do them nor how we should build a machine with similar abilities.”<h2>Were there also companies involved in this project?</h2>Yes, for example, one of our partners was logistic process automation specialist VanderLande. This is a well-known and large company from the Netherlands that operates worldwide in various fields, such as airports, warehousing, and parcel handling. They provided a lot of real-life use cases and insights on what are the current bottlenecks in the field, the so-called market ``pains’’. One of the great aspects of this collaboration was having a shared lab on TU/e campus, which facilitated hands-on testing and close cooperation. Students and researchers loved it. We have done various comparisons between real and software-simulated impact experiments as well as developed new models of suction grippers to be used for motion control and planning.”<h2>Would you say that the Netherlands is specialized in robotics?</h2>“The Netherlands as a whole is definitely making significant strides when it comes to robotics. The country has long been known for contributing to medical robotics, robot learning, and mobile robotics, to cite a few. I like to think that via this project and its international collaborations, we have also made significant strides in impact-aware robotics. This area of research, which we could say originated here, has garnered global attention and recognition. Our project has played a key role in bringing the subject to the forefront, and it's gratifying to see that our work has been well-received and acknowledged, including in recent publications."<h2>The project is done. What does the future look like for you?</h2>“I will continue investigating and exploring new opportunities, including national and European funding. I'm considering follow-up projects focusing on areas that we couldn’t address during this one, like fast planning and perception. I’m also still in close contact with local and international companies to explore further collaboration opportunities. Some of the many students involved in the project –they are the real heroes!– have been hired in companies that were partners in the project: things are growing in the right direction. The visibility this project has generated is both challenging and exciting at the same time. While it means juggling more tasks, I’m also looking forward to seeing what the future holds.”<h3>European research project I.AM</h3>Europe is at the forefront of the market for torque-controlled robots designed to handle physical interactions with their environment. The recently completed I.AM project, in which Alessandro Saccon and his colleagues were involved, built on this technology and reinforced European leadership by enabling robots to utilize intentional impacts for manipulation tasks. I.AM specifically targeted impact-aware manipulation in logistics—a rapidly growing field in robotics expected to expand significantly in the coming years. Three other research institutes and four companies from the Netherlands, Germany, Switzerland, Sweden, and France collaborated on this four-year project.</section>

To achieve greater efficiency, robots must develop more human-like skills, such as fast physical interaction. That was the focus of the I.AM project.

Revolutionizing robotics: teaching robots to touch and interact like humans

Curious about how AI predicts trends or creates new content? In this piece, we dive into Predictive AI vs. Generative AI, exploring their core differences and real-world impact. Discover how these technologies shape everything from recommendations to creativity.

Generative AI vs Predictive AI: Unveiling the Titans of Artificial Intelligence

Machina Labs-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Machina Labs-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Machina Labs-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Machina Labs-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Machina Labs-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Dessa-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Dessa-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Dessa-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Dessa-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Dessa-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Edge Impulse-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Edge Impulse-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Mon Oct 30 2023 14:34:39 GMT+0200 (Eastern European Standard Time)

Edge Impulse-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Plainsight-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Synopsys Inc

Voltera

BERHALTER Swiss Die-Cutting-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

BERHALTER Swiss Die-Cutting-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

BERHALTER Swiss Die-Cutting-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

BERHALTER Swiss Die-Cutting-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

BERHALTER Swiss Die-Cutting-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Addcomposites-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Addcomposites-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Addcomposites-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Addcomposites-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Addcomposites-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

machine learning

machine learning

Profiles