At least some of you will remember The Jetsons.The television series, created by Hanna-Barbera Cartoons Inc., was a space-age version of The Flintstones (another Hanna-Barbera production). It originally aired in 1962-1963 with later episodes created in a reboot from 1985 to 1987.But while Fred Flintstone drove a stone-age car (complete with stone wheels) that he powered by pushing his feet along the ground, George Jetson and his family lived in Orbit City, where Jetson commuted to his two-hour per week job via a flying car with a bubble top. And instead of having dinosaurs (including pterodactyls) help carry out tasks, The Jetsons live in a future where they’re surrounded by automated devices. You could think of their surroundings as the 1960s vision of the Smart Home.And an integral part of that home? Well, that would be Rosey (later changed to ‘Rosie’) the robot.Rosey was the family’s robotic maid. She carried out tasks that weren’t performed by the many other automatic conveniences that filled the Jetson’s home. She had two manipulator arms and an internally stored vacuum that be deployed on demand.She was very useful around the house, carrying out tasks to save the family time.And this story? Well, it’s about our own Rosie – which is also very space-age.Below: A Rosie the robot publicity cel, signed by show creators William Hanna and Joseph Barbera. The cel was auctioned in 2018; <a href="https://comics.ha.com/itm/animation-art/presentation-cel/the-jetsons-rosie-the-robot-publicity-cel-hanna-barbera-1985-/a/7193-97570.s#" target="_blank">image by Heritage Auctions</a><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI4NDAzNDIwMTI1LVNjcmVlbnNob3QtMjAyNC0xMC0wMi1hdC0xLjUxLjIwLVBNLnBuZy53ZWJwIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib" alt="Rosie the Robot"> <h3 id="isPasted">The Rosie Story</h3>So. What is Rosie? We asked Head of R&amp;D Sales Luke Corbeth for a snapshot.“Rosie is a dual arm mobile manipulation robot designed for pick and place in an industry 4.0 setting,” he says. In other words, it has two arms and manoeuvres on a wheeled platform, and is capable of moving objects from one location to another or even manipulating a single object with both end effectors.And Rosie has a few tricks up her sleeve. Or, more accurately, sleeves.“The actual robot is very unique because it has six mounting points for the arms. So you can mount the arms on top, high on the side or low on the side to access shelving of different heights. In fact,&nbsp;you could actually mount one arm directly on the top right, for example, and then mount the second one on the bottom left. So you could grab something from the top of the shelf and from the floor at the same time, which is kind of cool, right?”Yes, indeed.Rosie’s home is not with the Jetsons (she has no vacuum cleaner) but in a new lab that hasn’t yet been officially launched at <a href="https://www.polymtl.ca/futur/en" target="_blank">Polytechnique Montréal</a>. It’s called the Intelligent-Cyber Physical System Lab, or I-CPS. So we contacted <a href="https://www.polymtl.ca/expertises/en/birglen-lionel" target="_blank">Lionel Birglen</a>, a professor with the Department of Mechanical Engineering. We wanted to learn more about what the lab does, what he does – and what plans he has for Rosie (which InDro built and shipped in 2023).Dr. Birglen is a PhD Mechanical Engineer, with a specialisation in robotics. He’s particularly interested in – and an expert on – manipulators and end effectors and has designed and built them. He’s written two books, holds three patents, and is the author or contributing author of at least <a href="https://www.researchgate.net/scientific-contributions/Lionel-Birglen-69852833" target="_blank">94 research papers</a>. He’s also – get this – been listed among the <a href="https://www.polymtl.ca/carrefour-actualite/en/news/50-polytechnique-montreal-researchers-among-top-2-most-cited-their-respective-fields" target="_blank">top two per cent&nbsp;most-cited scientists in the world</a> in his area of specialisation.So it kinda goes without saying, but he’s a pretty big deal in this field.Dr. Birglen has a deep interest in the role robotics will play in the future of industry. And, within that realm, he’s intensely interested in ensuring that robots, particularly those that will be sharing space with human beings on a factory or warehouse floor, will be safe.And – he emphasises – he doesn’t trust simulations for important work like this.“Because simulations lie. They lie all the time,” he says. “You have to understand that reality is infinitely more complex than anything you can have in simulation – so actual experiments are absolutely essential to me. They are essential to my work,&nbsp;to my understanding of what robotic manipulation is.”“I believe in math, but I know that reality is different.&nbsp;It’s more complex, more complicated, and includes so many un-modelled phenomena.”&nbsp;<h3>Rosie's Journey</h3>Dr. Birglen knew he wanted a new robot for use in the new lab (which we’ll get to shortly). And he knew he wanted a robot with two manipulator arms.“Dual-arm robots are, in my opinion, the future for industry applications,” he says.And while humanoid bipeds grab a lot of attention, they’re far more complex (and expensive) than wheeled robots. Plus, he says, most factory applications take place on a single level and don’t require climbing stairs.“From a factory perspective, a wheeled platform makes a lot of sense because typically in factories you don’t have, say, five levels connected by stairs.”So he knew he wanted an autonomous, wheeled, dual-arm robot. And he started, initially, to think of a company other than InDro for the build.“I came across InDro almost by accident,” he explains. “<a href="https://www.polymtl.ca/expertises/en/beltrame-giovanni" target="_blank">Giovanni Beltrame</a> told me about you because he has purchased many, many robots from you. He said: ‘Those guys can build and assemble the robot for you. They’re close and they do a great job.’ So that’s how I came in contact with you.” (We’ve written previously about the amazing work Dr. Beltrame is working on involving robots and space. You can find that <a href="https://indrorobotics.ca/robots-on-earth-help-prepare-for-research-on-the-moon/" target="_blank">here.</a>)And so, after a number of calls with Luke Corbeth and the engineering team to settle on design and performance parameters, work on Rosie began.Below: Technologist Tirth Gajera (‘T’) puts the finishing touches on Rosie in 2023<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI4NDAzNDY1MDg1LURTQzAxNTE4LXNjYWxlZC5qcGcud2VicCIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib" alt="InDro's Roșie the Robot"> <h3 id="isPasted">The Lab</h3>Polytechnique Montréal’s&nbsp;Intelligent-Cyber Physical System Lab (I-CPS) is set up as a highly connected Industry 4.0 factory. Faculty from four different departments – computer engineering, electrical engineering, industrial engineering and mechanical engineering (Dr. Birglen) – are involved with the lab. Interns and students, under supervision, also work in the facility.“So we have four&nbsp;departments involved in this lab and the idea is to build a small scale factory of the future, meaning that everything is connected. We are building a mini-factory inside this lab,” he says.So think of cameras that can track objects on shelves – and people and robots within the environment. Think of smart tools like a CNC machine, which will eventually be operated by Rosie. And, perhaps just as important as the connectivity within the lab, is connectivity to other research institutes in Quebec, including <a href="https://www.ulaval.ca/en" target="_blank">Université Laval</a>, <a href="https://www.usherbrooke.ca/about/" target="_blank">Université de Sherbrooke</a> and <a href="https://www.etsmtl.ca/en" target="_blank">École de Technologié Supérieure</a> (ÉTS). All of those institutes are working with similar mini-factories, and they’re all connected. There’s even a relationship (and connectivity) with manipulator manufacturer <a href="https://www.kinovarobotics.com/" target="_blank">Kinova</a>. Funding came via a significant grant from the <a href="https://www.innovation.ca/" target="_blank">Canada Foundation for Innovation</a>, or CFI.“So think of our lab as like one node of this network of mini-factories around Quebec,” explains Dr. Birglen. That connectivity of all components is still a work-in-progress, but “ultimately the goal is that there is a cyber-connection between these different mini-factories, these different laboratories around Quebec, so that one part of one node can work in collaboration with another node in realtime.”Plus, of course, a lot of learnings will take place within the individual labs themselves.“We want to bring collaborative robots to work in tandem with humans,” he says. “We want our robots to safely move around people, we want robots to help people. And we also want robots to learn how to work from people.”&nbsp;<h3>Safety, Safety, Safety</h3>As mentioned earlier, there’s a huge emphasis on safety. And while there are international safety standards for collaborative robots, even a ‘safe’ cobot can pose a threat.“All the collaborative robots that you have currently on the market more or less follow this technical standard and they are more or less safe, but they’re still dangerous,” explains Dr. Birglen. “And the classical example that we’ve all heard, and which is true, is that if a safe cobot has a knife in its hand and is moving around – it is very dangerous.”So safety in the lab(s) is paramount – and that means safety at multiple levels. There must be safety:<ul><li>At the task level; you must not have tasks that could endanger people</li><li>Safety at the control level</li><li>Safety in terms of collision detection, mitigation, obstacle avoidance</li><li>Safety at the data security level</li></ul>Plus – and this really interests Dr. Birglen – you must ensure safety with any additional mechanical innovations that are introduced.“What you develop, any mechanical system you develop, must be as much as possible intrinsically safe. And actually that’s one of the topics I’m currently working on is to develop end effectors and tooling that is intrinsically safe.”<a href="https://www.linkedin.com/embed/feed/update/urn:li:ugcPost:7186438904722472960" target="_blank" rel="noopener noreferrer">A LinkedIn post from Luke Corbeth shows Rosie, using both arms, inside the I-CPS lab</a><h3> </h3><h3 id="isPasted">The Future&nbsp;</h3>And why is research like this so important? What difference will it make to have robots and humans working safely together, with safe manipulators and end effectors that might even be able to, for example, lift an object in concert with a human being? And why the focus on interconnectedness between all of these facilities?Well, there’s obviously the value of the research itself – which will lead to greater efficiencies, improved manipulators, gripping technologies, new algorithms and AI enhancements – as well as enhanced safety down the road. But there’s a much bigger picture, says Dr. Birglen, especially if you can get your head around thinking about the future from a global perspective.China, he says, is no longer a developing nation. The days when the words “Made in China” meant poor quality are – with rare exceptions – gone. The country is, in fact, highly developed – and working at breakneck speed when it comes to innovation and adoption of robotics at scale. A revolution is underway that has massive implications for competitive advantage that simply cannot be ignored. So the research at &nbsp;I-CPS is not merely important from an academic perspective, it’s strategic when viewed through a global economic lens.“We as a country – meaning Canada – are in competition with other countries for manufacturing, for producing goods and services.&nbsp;China is a developed country and it is very, very, very good in robotics,” he states. “You know how in the past&nbsp;we saw China as producing low quality goods, low quality robots? That’s over, man. That’s finished.”And?“If they are investing in robotics like mad and we are not, we’re going to be a leftover –&nbsp;Canada is going to sink as a rich country. If you want to produce wealth in the 21st Century, you need robots, you need automation, you need integration. In short, you need to be the leader of the pack or you’re going to be eaten.”It’s a stark warning – and it’s true.I step outside as author and state this having lived in China back when it was still a developing country in the late 1980s – and having returned several times since then. The transformation has been nothing short of astonishing. How, you might ask, did it achieve all this?The answer has its genesis with former Chinese leader Deng Xiaoping. who led the country from 1978 to 1989. He didn’t merely open the door to reform; he created policies that began sending waves of students from what had been a xenophobic country abroad to study. There was an emphasis on careers that could help modernise the nation, including all aspects of engineering, aerospace, construction, transportation, architecture, etc. That’s where all this began.Thankfully (and with credit to federal funding agencies like CFI), there are projects like I-CPS underway – and academics like Dr. Lionel Birglen with the vision to push the needle safely forward.Below: “Baxter” – the original dual-arm robot. Baxter is still at Polytechnique Montréal, but Rosie is the mobile future. Photo by Luke Corbeth<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1728403885667-Image-22.jpeg.webp" style="width: 100%px;" class="fr-fic fr-dib" alt="Baxter"><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1728403913708-Rosie.png.webp" style="width: 100%px;" class="fr-fic fr-dib" alt="Rosie Details"><h3 id="isPasted">InDro's Take&nbsp;</h3>We’re obviously pleased Polytechnique Montréal selected InDro to build Rosie. And we’re particularly pleased to see that she’s being deployed at I-CPS, as part of an integrated and networked research project that has such potentially profound implications for the future.“I believe Dr. Birglen is correct in his assessment of the importance of robotics and automation in the future,” says InDro Robotics Founder and CEO Philip Reece. “And when you throw innovations with drones and even autonomous Uncrewed Aerial Vehicles capable of carrying large cargo loads and passengers into the mix, we are actually heading into a Jetsons-like future,” he adds.“I think there’s a growing understanding of the implications of this kind of future from not only the private sector, but also federal regulators and funding agencies. At InDro our mission will always focus on continued innovation. Sometimes those innovations are our own inventions, but a key piece of the puzzle is R&amp;D work carried out by academics like Lionel Birglen. We’re confident that Rosie’s arms are in the right hands.”Interested in learning more about a custom robotics solution? Feel free to <a href="https://indrorobotics.ca/connectwithanexpert/" target="_blank">contact us here</a>.

Rosie is a dual arm mobile manipulation robot designed for pick and place in an industry 4.0 setting.

Dual Manipulator Rosie The Robot Used for Industry 4.0 Research

<h2 id="isPasted">At the top of many automation wish lists is a particularly time-consuming task: chores.</h2>The moonshot of many roboticists is cooking up the proper hardware and software combination so that a machine can learn “generalist” policies (the rules and strategies that guide robot behavior) that work everywhere, under all conditions. Realistically, though, if you have a home robot, you probably don’t care much about it working for your neighbors. MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers decided, with that in mind, to attempt to find a solution to easily train robust robot policies for very specific environments.“We aim for robots to perform exceptionally well under disturbances, distractions, varying lighting conditions, and changes in object poses, all within a single environment,” says Marcel Torne Villasevil, MIT CSAIL research assistant in the Improbable AI lab and lead author on a recent paper about the work. “We propose a method to create digital twins on the fly using the latest advances in computer vision. With just their phones, anyone can capture a digital replica of the real world, and the robots can train in a simulated environment much faster than the real world, thanks to GPU parallelization. Our approach eliminates the need for extensive reward engineering by leveraging a few real-world demonstrations to jump-start the training process.”<iframe width="640" height="360" src="https://www.youtube.com/embed/w-L90BhfDFo?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h2 id="isPasted">Taking your robot home</h2>RialTo, of course, is a little more complicated than just a simple wave of a phone and (boom!) home bot at your service. It begins by using your device to scan the target environment using tools like NeRFStudio, ARCode, or Polycam. Once the scene is reconstructed, users can upload it to RialTo’s interface to make detailed adjustments, add necessary joints to the robots, and more.The refined scene is exported and brought into the simulator. Here, the aim is to develop a policy based on real-world actions and observations, such as one for grabbing a cup on a counter. These real-world demonstrations are replicated in the simulation, providing some valuable data for reinforcement learning. “This helps in creating a strong policy that works well in both the simulation and the real world. An enhanced algorithm using reinforcement learning helps guide this process, to ensure the policy is effective when applied outside of the simulator,” says Torne.Testing showed that RialTo created strong policies for a variety of tasks, whether in controlled lab settings or more unpredictable real-world environments, improving 67 percent over imitation learning with the same number of demonstrations. The tasks involved opening a toaster, placing a book on a shelf, putting a plate on a rack, placing a mug on a shelf, opening a drawer, and opening a cabinet. For each task, the researchers tested the system’s performance under three increasing levels of difficulty: randomizing object poses, adding visual distractors, and applying physical disturbances during task executions. When paired with real-world data, the system outperformed traditional imitation-learning methods, especially in situations with lots of visual distractions or physical disruptions.“These experiments show that if we care about being very robust to one particular environment, the best idea is to leverage digital twins instead of trying to obtain robustness with large-scale data collection in diverse environments,” says Pulkit Agrawal, director of Improbable AI Lab, MIT electrical engineering and computer science (EECS) associate professor, MIT CSAIL principal investigator, and senior author on the work.As far as limitations, RialTo currently takes three days to be fully trained. To speed this up, the team mentions improving the underlying algorithms and using foundation models. Training in simulation also has its limitations, and currently it’s difficult to do effortless sim-to-real transfer and simulate deformable objects or liquids.<h2>The next level</h2>So what’s next for RialTo’s journey? Building on previous efforts, the scientists are working on preserving robustness against various disturbances while improving the model’s adaptability to new environments. “Our next endeavor is this approach to using pre-trained models, accelerating the learning process, minimizing human input, and achieving broader generalization capabilities,” says Torne.“We’re incredibly enthusiastic about our 'on-the-fly' robot programming concept, where robots can autonomously scan their environment and learn how to solve specific tasks in simulation. While our current method has limitations — such as requiring a few initial demonstrations by a human and significant compute time for training these policies (up to three days) — we see it as a significant step towards achieving 'on-the-fly' robot learning and deployment,” says Torne. “This approach moves us closer to a future where robots won’t need a preexisting policy that covers every scenario. Instead, they can rapidly learn new tasks without extensive real-world interaction. In my view, this advancement could expedite the practical application of robotics far sooner than relying solely on a universal, all-encompassing policy.”“To deploy robots in the real world, researchers have traditionally relied on methods such as imitation learning from expert data, which can be expensive, or reinforcement learning, which can be unsafe,” says Zoey Chen, a computer science PhD student at the University of Washington who wasn’t involved in the paper. “RialTo directly addresses both the safety constraints of real-world RL [robot learning], and efficient data constraints for data-driven learning methods, with its novel real-to-sim-to-real pipeline. This novel pipeline not only ensures safe and robust training in simulation before real-world deployment, but also significantly improves the efficiency of data collection. RialTo has the potential to significantly scale up robot learning and allows robots to adapt to complex real-world scenarios much more effectively.”"Simulation has shown impressive capabilities on real robots by providing inexpensive, possibly infinite data for policy learning,” adds Marius Memmel, a computer science PhD student at the University of Washington who wasn’t involved in the work. “However, these methods are limited to a few specific scenarios, and constructing the corresponding simulations is expensive and laborious. RialTo provides an easy-to-use tool to reconstruct real-world environments in minutes instead of hours. Furthermore, it makes extensive use of collected demonstrations during policy learning, minimizing the burden on the operator and reducing the sim2real gap. RialTo demonstrates robustness to object poses and disturbances, showing incredible real-world performance without requiring extensive simulator construction and data collection.”Torne wrote this paper alongside senior authors Abhishek Gupta, assistant professor at the University of Washington, and Agrawal. Four other CSAIL members are also credited: EECS PhD student Anthony Simeonov SM ’22, research assistant Zechu Li, undergraduate student April Chan, and Tao Chen PhD ’24. Improbable AI Lab and WEIRD Lab members also contributed valuable feedback and support in developing this project.This work was supported, in part, by the Sony Research Award, the U.S. government, and Hyundai Motor Co., with assistance from the WEIRD (Washington Embodied Intelligence and Robotics Development) Lab.

The moonshot of many roboticists is cooking up the proper hardware and software combination so that a machine can learn “generalist” policies (the rules and strategies that guide robot behavior) that work everywhere, under all conditions.

Precision home robots learn with real-to-sim-to-real

In this episode, we discuss how a talented team of researchers from Stanford university have created a robotic platform capable of imitation learning for conducting over 850 common tasks, including searing shrimp!

Podcast: Did A Shrimp Fry This Rice? No, A Robot Did

Robotic systems automate repetitive tasks, handling complex, powerful motions and working in environments that are dangerous or harmful to humans. More integrated, higher-performance microcontrollers (MCUs) enable higher power efficiency and smoother and safer motion with much higher accuracy, therefore increasing productivity and automation. For example, higher accuracy – sometimes within 0.1 mm – is important for applications that handle laser welding, precision coatings, or inkjet or 3D printing.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1717222891211-GUID-20240130-SS0I-FNRF-GSFH-MVQTTTVX1WBR-low.jpg" style="width: 100%px;" class="fr-fic fr-dib">The number of axes in a robot arm and the type of control architecture needed (either centralized or distributed, also known as decentralized) will ultimately determine the right MCU or motor-control integrated circuit (IC) for that system. Modern factories use a mix of robots with varying numbers of axes, defined as the degree of movement and rotation across x, y or z planes, in order to handle different phases of manufacturing; therefore, there are various control architectures throughout the factory floor.It’s important to choose MCUs with additional performance headroom to provide future scalability, as well as the capacity to support add-on features. Planning for scalability and additional features can reduce cost, time and complexity during the design process.In this article, I’ll explore centralized and distributed motor-control architectures and the design considerations for the integrated, real-time MCUs enabling them. To read about key motor drive design challenges for robotics, read our technical article, “<a href="https://e2e.ti.com/blogs_/b/industrial_strength/posts/how-motor-drive-innovations-are-helping-solve-robotic-movement-design-challenges" target="_blank">How motor drive innovations are helping solve robotic movement design challenges</a>.”<h2>A centralized architecture</h2>In a centralized system (<a href="https://www.ti.com/document-viewer/lit/html/SSZTD23#FIG_ZVY_HCV_G1C" target="_blank">Figure 1</a>), one MCU controls multiple axes. This approach is effective at handling heat dissipation in higher-power motor drivers (typically above 2-3 kW) that require large heat sinks and cooling fans. In this architecture, position data is usually acquired externally through a resolver board or aggregator connected to the encoders.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1717222997634-GUID-20240130-SS0I-NM5K-FRBN-8SQTBFPXBCCR-low.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 1 Block diagram of a centralized motor-control architecture for multiple-axis systemsTypically, in this architecture, multiple power stages are on the same printed circuit board or in close proximity; thus, one MCU can control multiple axes. This approach simplifies the real-time control and synchronization between multiple axes, as there is no need for longer communication lines between multiple motor-control MCUs.The motor-control MCU/microprocessor in a centralized architecture mainly requires high-performing real-time processing cores such as Arm® Cortex®-R5F cores or digital signal processors, real time communication interfaces such as EtherCAT, and enough pulse-width modulation (PMW) channels and peripherals for voltage and current sensing. MCUs such as the AM2432 can enable scalable multiple-axis systems and offer real-time control peripherals for as many as six axes and real-time communications in a single chip.In the past, field-programmable gate arrays or application-specific integrated circuits were primarily used for centralized motor control in automated systems. However, the high level of integration and cost-effectiveness of Arm Cortex MCUs can help you meet your system’s performance requirements while also enabling design scalability and flexibility.While centralized control architectures can meet the performance and efficiency design requirements of high-powered automated systems such as heavy-payload industrial robots, they do require additional cables from the cabinet to the mechanical motor in the joint, as well as from the position sensors to the aggregators. Those wires do not only cost more, they also wear out and require maintenance.<h2>A decentralized architecture</h2>Decentralized architectures (<a href="https://www.ti.com/document-viewer/lit/html/SSZTD23#FIG_N21_HCV_G1C" target="_blank">Figure 2</a>) have recently grown in popularity for systems with lower power requirements, and have become the standard approach for collaborative robot (cobot) manipulators.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1717223024230-GUID-20240130-SS0I-W9T6-X4MK-G7DDQDDXB8LN-low.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 2 Block diagram of a decentralized motor control architecture for single-axis systemsDecentralized architectures integrate multiple single-axis motor drives into each joint within the robot, connected and synchronized through a real-time communication interface such as EtherCAT. Each drive usually controls one single axis and handles certain safety functions locally. Therefore, each MCU needs real-time control and communication capabilities, motor-control peripherals for a single axis, three to six PWM channels, on-chip successive approximation register analog-to-digital converters, or delta-sigma modulator inputs.In these applications, the position sensor usually sits next to the MCU, so these MCUs need a digital or analog interface to read data from the position sensor. While this architecture requires more MCUs, cost reductions at the system level can be significant, since you’ll need less cabling between the power bus and communication interface. Modern real-time MCUs such as the TMS320F28P650DK integrate all necessary peripherals including safety peripherals, which can enable single- or two-chip solutions for an integrated axis in decentralized architectures with high performance at a small size.<h2>Conclusion</h2>While motors may not be the trendiest system in a robot – especially when compared to artificial intelligence-enabled systems – they are the “muscles” that keep factories moving. They might be the most crucial part of modern manufacturing, so selecting the right control device deserves a lot of consideration. As these devices become more integrated, additional capabilities such as edge computing and wireless connectivity may make their way into motor-control designs.

More integrated, higher-performance microcontrollers (MCUs) enable higher power efficiency and smoother and safer motion with much higher accuracy, therefore increasing productivity and automation.

How MCUs increase system performance in robot motor control designs

This article was written by Adam Taylor for Mouser Electronics<h2 id="isPasted">Introduction</h2>In this project, we will configure the AMD Kria™ KR260 Robotics Starter Kit to control a Trossen Robotics ReactorX 150 robot arm. This complex robot arm uses <a href="https://www.mouser.com/new/robotis/robotis-dynamixel-x-modules/" rel="nofollow" target="_blank" data-internal="false">ROBOTIS DYNAMIXEL</a> servos, which contain not only motors but also microcontrollers and networking capability. While this enables the complex motor drives needed for robotics applications, it also means that each servo has several different drive and control mechanisms. Typically, when working with servos and robotics applications of this complexity, significant processing is required to plan and address the robotic movement.To interact with and control such robots, engineers often use the <a href="https://www.ros.org/" rel="nofollow" target="_blank" data-internal="false">Robot Operating System (ROS)</a>, which runs on Linux. The most used version of ROS is ROS 2, which updated the ROS frameworks and tools to work with a wider variety of environments, provide support for real-time environments, and use significantly updated APIs. ROS 2 provides developers with functionalities such as hardware drivers, robot models, datatypes, and support for perception and simultaneous localization and mapping (SLAM). ROS 2 also provides a series of tools that aid in the development or operation of systems such as RViz, which provides a 3D visualization, and Gazebo, a simulator.ROS 2 is built around a graph architecture; within this architecture, processing takes place in nodes that can receive and post data about the node, such as sensor, control, planning, actuator positioning, or current state. Nodes are connected on the ROS graph by topics, which are communication pipelines to which nodes can publish data and receive information. Along with nodes and topics, a node may also advertise services. The services have a single result, such as capturing a frame of video, sampling a sensor, or opening an actuator.<h2>Bill of Materials</h2><ul><li><a href="https://www.mouser.com/new/xilinx/xilinx-kria-kr260-kit/" rel="nofollow" target="_blank" data-internal="false">AMD Kria™ KR260 Robotics Starter Kit</a></li><li>Monitor with DisplayPort™ compatibility</li><li>USB keyboard and mouse</li><li><a href="https://www.trossenrobotics.com/reactorx-150-robot-arm.aspx" rel="nofollow" target="_blank" data-internal="false">ROBOTIS ReactorX In150</a> robotic arm</li></ul><h2>Resources</h2><ul><li><a href="https://ubuntu.com/certified/202202-29985" rel="nofollow" target="_blank" data-internal="false">Ubuntu Linux distribution image for the AMD Kria™ KR260 Robotics Starter Kit</a> (Ubuntu)</li><li><a href="https://www.xilinx.com/products/som/kria/kr260-robotics-starter-kit/kr260-getting-started/setting-up-the-sd-card-image.html" rel="nofollow" target="_blank" data-internal="false">Getting Started with the AMD Kria KR260 Robotics Starter Kit</a> (AMD)</li><li><a href="https://sourceforge.net/projects/vcxsrv/files/vcxsrv/1.20.14.0/" rel="nofollow" target="_blank" data-internal="false">VcXsrv Windows X Server Files</a></li></ul><h2>Software Setup</h2>In this project, we will install the Trossen Robotics packages and ROS 2 on the AMD Kria™ KR260 Robotics Starter Kit, which will enable you to control the robot arm.<h3>Installing Ubuntu</h3>Start by downloading and installing the Ubuntu Linux distribution image for the AMD Kria™ KR260 kit, available in the Resources section. Once the image is available, flash the image to an SD card following the instructions provided by AMD in Getting Started with Kria KR260 Robotics Starter Kit in the Resources section.Once you have successfully booted the AMD Kria KR260 kit, you will need to update the Ubuntu installation to ensure we can install ROS 2 correctly.<ol><li>Install resolvconf.</li></ol><code>&nbsp; &nbsp;sudo apt update</code><code>&nbsp; &nbsp;sudo apt install resolvconf</code><ol start="2"><li>With resolvconf installed, ensure it is running.</li></ol><code>&nbsp; &nbsp;sudo systemctl status resolvconf.service</code><ol start="3"><li>Once you have confirmed the service is running, add your preferred DNS server. In this example, we use Googles DNS.</li></ol><code>&nbsp; &nbsp;echo "nameserver 8.8.8.8" | sudo tee -a /etc/resolvconf/resolv.conf.d/head</code><code>&nbsp; &nbsp;echo "nameserver 8.8.4.4" | sudo tee -a /etc/resolvconf/resolv.conf.d/head</code><ol start="4"><li>Because we made changes, we need to restart the services.</li></ol><code>&nbsp; &nbsp;sudo systemctl restart resolvconf.service</code><code>&nbsp; &nbsp;sudo systemctl restart systemd-resolved.service</code><ol start="5"><li>Prevent Ubuntu on the KR260 from going to sleep.</li></ol><code>&nbsp; &nbsp;sudo gsettings set org.gnome.desktop.session idle-delay 0</code><code>&nbsp; &nbsp;sudo systemctl mask suspend.target</code><h3>Installing ROS 2</h3>To install ROS 2 and the packages to control the Trossen Robotics X Series arm, we will modify the instructions slightly to those used for the Raspberry Pi, which, like the AMD Kria™ KR260 kit, uses the Arm64 (aarch64) instruction set architecture.The latest version of ROS 2 supported by Trossen’s libraries is ROS 2 Humble. Use the following commands to install ROS 2 and the Trossen libraries:<code>cd ~</code><code>sudo apt install curl</code><code>curl 'https://raw.githubusercontent.com/Interbotix/interbotix_ros_manipulators/main/interbotix_ros_xsarms/install/rpi4/xsarm_rpi4_install.sh' &gt; xsarm_rpi4_install.sh</code><code>sed -i 's/sudo apt-get update &amp;&amp; sudo apt -y upgrade/sudo apt-get update/g' xsarm_rpi4_install.sh</code><code>chmod +x xsarm_rpi4_install.sh</code><code>./xsarm_rpi4_install.sh -d humble -j rx150</code><h2>Demo Application</h2>With ROS 2 installed on the AMD Kria™ KR260 kit, we will run one of the provided demo applications. Open two terminal windows on the KR260.<ol><li>In the first terminal window, enter the following command:</li></ol><code>&nbsp; &nbsp;ros2 launch interbotix_xsarm_control xsarm_control.launch.py robot_model:=rx150</code><ol start="2"><li>In the second terminal, enter the following command:</li></ol><code>&nbsp; &nbsp;python3 /home/ubuntu/interbotix_ws/src/interbotix_ros_manipulators/interbotix_ros_xsarms/interbotix_xsarm_control/demos/python_ros2_api/bartender.py</code>This will run an application that demonstrates the control of the robot arm.<h3>Demo Breakdown</h3>By examining the code in this application, we can see how we can control the arm using the Python programming language. First, we need to import the InterbotixManipulatorXS arm package from the InterbotiX libraries we installed. To do this, enter the following command: <code>from interbotix_xs_modules.arm import InterbotixManipulatorXS</code>With these packages installed, we can start creating applications that manipulate the arm using a Python application.Looking at the example code, we can see the first thing we do is initialize it with the parameters of the robot, the model, type, and its end effector.With the robot (shown as <code>bot</code> in the code) defined, the application first checks the robot has the correct number of joints for the demo. Once it has been confirmed as being suitable for the demo, the robot is controlled by using several Python commands that set the pose, or single positions, as required. The gripper can also be opened and closed using Python calls.<code>bot = InterbotixManipulatorXS("rx150", "arm", "gripper")</code><code>&nbsp; &nbsp; if (bot.arm.group_info.num_joints &lt; 5):</code><code>&nbsp; &nbsp; &nbsp; &nbsp; print('This demo requires the robot to have at least 5 joints!')</code><code>&nbsp; &nbsp; &nbsp; &nbsp; sys.exit()</code><code>&nbsp; &nbsp; bot.arm.set_ee_pose_components(x=0.3, z=0.2)</code><code>&nbsp; &nbsp; bot.arm.set_single_joint_position("waist", np.pi/2.0)</code><code>&nbsp; &nbsp; bot.gripper.open()</code><code>&nbsp; &nbsp; bot.arm.set_ee_cartesian_trajectory(x=0.1, z=-0.16)</code><code>&nbsp; &nbsp; bot.gripper.close()</code>Finally, the demo ends by safely positioning the robot to its home position.<code>&nbsp; &nbsp; bot.arm.go_to_home_pose()</code><code>&nbsp; &nbsp; bot.arm.go_to_sleep_pose()</code>Knowing this, we can start creating own custom applications, as shown in <a href="https://youtu.be/g4xX4aowwI8" rel="nofollow" target="_blank" data-internal="false">this video</a>. But first, we need to be able to remotely develop applications on the AMD Kria™ KR260 board.<h2>Remote Development</h2>To develop our remote application, we will use Visual Studio Code on the development machine. The Python application we create using Visual Studio Code on our development machine will be in the file system of the AMD Kria™ KR260 kit. This also has the benefit of requiring no connected display, keyboard, or mouse, as we can access everything from the development environment.We also want to be able to remotely view the visualization of the robot arm as we develop our application (in simulation mode or actual execution).To enable remote development and visualization, we must establish SSH connection and enable X11 forwarding.<ol><li>Use Windows Powershell to install OpenSSH using the following commands.</li></ol><code>&nbsp; &nbsp;Get-WindowsCapability -Online | Where-Object Name -like 'OpenSSH*'</code><code>&nbsp; &nbsp;# Install the OpenSSH Client</code><code>&nbsp; &nbsp;Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0</code><code>&nbsp; &nbsp;# Install the OpenSSH Server</code><code>&nbsp; &nbsp;Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0</code><code>&nbsp; &nbsp;# Start the sshd service</code><code>&nbsp; &nbsp;Start-Service sshd</code><code>&nbsp; &nbsp;# OPTIONAL but recommended:/code&gt;</code><code>&nbsp; &nbsp;Set-Service -Name sshd -StartupType 'Automatic'</code><code>&nbsp; &nbsp;# Confirm the Firewall rule is configured. It should be created automatically by setup. Run the following to verify</code><code>&nbsp; &nbsp;if (!(Get-NetFirewallRule -Name "OpenSSH-Server-In-TCP" -ErrorAction SilentlyContinue | Select-Object Name, Enabled)) {</code><code>&nbsp; &nbsp; &nbsp; &nbsp; Write-Output "Firewall Rule 'OpenSSH-Server-In-TCP' does not exist, creating it..."</code><code>&nbsp; &nbsp; &nbsp; &nbsp; New-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22</code><code>&nbsp; &nbsp;} else {</code><code>&nbsp; &nbsp; &nbsp; &nbsp; Write-Output "Firewall rule 'OpenSSH-Server-In-TCP' has been created and exists."</code><code>&nbsp; &nbsp;}</code><ol start="2"><li>With OpenSSH installed, create a key using the following command.</li></ol><code>&nbsp; &nbsp;ssh-keygen</code>By default, the system will save the keys to <code>C:\Users\&lt;user&gt;/.ssh/id_rsa.</code><ol start="3"><li>Exchange keys using the following command.</li></ol><code>&nbsp; &nbsp;type $env:USERPROFILE\.ssh\id_rsa.pub | ssh ubuntu@&lt;board_ip&gt; "cat &gt;&gt; .ssh/authorized_keys"</code>To establish a SSH connection from Visual Studio Code, we must first install the remote SSH option from the extensions (Figure 1).<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE3MTQ3NTU5NTE3LUFNRHBfZmlnMS53ZWJwIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">Figure 1: SSH remote connection application. (Source: Mouser Electronics)Once this is installed, we can connect to the AMD Kria KR260 board (Figure 2), which will allow us to develop our applications remotely in the KR260 file system.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE3MTQ3NTc0MDA4LUFNRHBfZmlnMi53ZWJwIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">Figure 2: Establishing an SSH remote connection. (Source: Mouser Electronics)To be able to use X11 forwarding, we must first install cXsrv Windows X Server (available in the Resources section) on our development machine. Run X Server and configure it. For screenshots fo this setup see <a href="https://resources.mouser.com/robotics/mastering-multi-axis-robotics-guide" target="_blank" rel="noopener noreferrer">here</a>. &nbsp;Connect to the display by running the following command in a command window on the development machine:<code>set DISPLAY=127.0.0.1:0.0</code>We can then initiate an SSH connection with the AMD Kria KR260 board and start developing our robot application.<code>ssh -Y &lt;user&gt;@&lt;board_ip&gt;</code>As this runs, the robot arm will move and you will see a visualization of this movement on X Server (Figure 7).<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE3MTQ3Njc4MjM0LUFNRHBfZmlnNy53ZWJwIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">Figure 7: X Server robotic arm visualization (Source: Mouser Electronics)<h2 id="isPasted">Wrap Up</h2>Robotics can be complicated. In this project, we have seen that the AMD Kria™ KR260 kit can enable rapid development of robotic solutions using ROS 2. The KR260 provides the ability to accelerate features within the programmable logic element or to integrate further system control elements such as time-sensitive networking within the programmable logic.<hr>AMD, and the AMD Arrow logo, Kria, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used herein are for identification purposes and may be trademarks of their respective owners.

In this project, we will configure the AMD Kria™ KR260 Robotics Starter Kit to control a Trossen Robotics ReactorX 150 robot arm. This complex robot arm uses ROBOTIS DYNAMIXEL servos, which contain not only motors but also microcontrollers and networking capability.

Mastering Multi-Axis Robotics: A Step-by-Step Guide

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/eb21608e-dd64-4028-89db-ee2a5395b3ae?dark=false" class="fr-draggable" data-gtm-yt-inspected-23="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Earlier this year, a robot took the internet by storm by sautéing shrimp. In addition to this culinary feat, <a href="https://mobile-aloha.github.io/" data-extlink="" target="_blank">Mobile ALOHA</a> was shown performing other acts associated with the long-held dream of general-purpose household robots, such as vacuuming, doing laundry, and watering plants.While some of the tasks Mobile ALOHA has performed on video weren’t autonomous, its ability to someday perform nearly any daily chore is not far-fetched. Behind the dazzling demonstrations is a unique teleoperation structure that allows Mobile ALOHA to learn many complex activities, impressively fast.“We’re providing a general framework that allows people to show robots how to do a task and then allows robots to learn from what they were shown,” said <a href="https://profiles.stanford.edu/chelsea-finn" data-extlink="" target="_blank">Chelsea Finn</a>, assistant professor of computer science and of electrical engineering at Stanford. Finn advises the Mobile ALOHA team, which is led by computer science graduate students Zipeng Fu and Tony Z. Zhao. &nbsp;<iframe width="640" height="360" src="https://www.youtube.com/embed/P7FRO_6L-iA?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-23="true" id="118103598" title="Stanford robot learns from humans">&nbsp;</iframe>Some general-use robots are programmed step-by-step and others, including Mobile ALOHA, learn through demonstration. But the Mobile ALOHA researchers took imitation learning a step further: The researchers strap themselves into a teleoperation system directly behind the robot’s arms and puppeteer the robot through the desired actions.Once Mobile ALOHA is operated through a task in a set environment about 50 times, powerful imitation learning algorithms help it make the leap to doing that task independently. (These algorithms are similar to the large language models behind popular chatbots, but for physical action instead of words.)So far, the researchers have taught Mobile ALOHA to autonomously put away a cooking pot in a cabinet, call an elevator, push in chairs, sauté shrimp, clean up a wine spill, and give high-fives. Future versions of the robot may be smaller with greater freedom of movement, and be easier for non-experts to operate.“The robot isn’t just a cool machine, we also wanted it to be fun and appeal to what people think a future robot should look like,” said Zhao.“Helping robots is a very promising future of the field where we – as AI researchers and roboticists – can make a positive impact in society,” said Fu.

What's the best way to train robots to perform everyday tasks? | Image courtesy of Kurt Hickman

Stanford engineers created a low-cost, mobile robot that quickly learned to sautée shrimp, put away dishes, and clean up spills. Other tasks aren't far behind.

Meet the robot that learned to saute shrimp

In this episode, we discuss how a team from Carnegie Mellon University spearheading non-invasive brain computer interface solutions has had a significant breakthrough in improving their accuracy.

Podcast: All The Brain Chip Implant Benefits & None of The Risks

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/96f146e3-d84e-4bc9-8fa4-af61e7ce87ed?dark=false" class="fr-draggable" data-gtm-yt-inspected-9="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>The full article will continue below.<hr>Pursuing a viable alternative to invasive brain-computer interfaces (BCIs) has been a continued research focus of Carnegie Mellon University’s <a href="https://www.cmu.edu/bme/helab/index.html" rel="noopener" target="_blank">He Lab</a>. In 2019, the group used a noninvasive BCI to successfully demonstrate, for the first time, that a mind-controlled robotic arm had the ability to continuously track and follow a computer cursor. As technology has improved, their AI-powered deep learning approach has become more robust and effective. In new work published in PNAS Nexus, the group demonstrates that humans can control continuous tracking of a moving object all by thinking about it, with unmatched performance.Noninvasive BCIs bring a host of advantages, in contrast to their invasive counterparts (think Neuralink or Synchron). These include increased safety, cost-effectiveness, and an ability to be used by numerous patients, as well as the general population. Amid these benefits, however, noninvasive BCIs face challenges because their recordings are less accurate and difficult to interpret.<iframe width="640" height="360" src="https://www.youtube.com/embed/EoKHw6bdoZs?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-9="true" id="917213689" title="Bin He: Neuroengineering and Dynamic Brain Mapping">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</iframe>In He’s <a href="https://academic.oup.com/pnasnexus/article-lookup/doi/10.1093/pnasnexus/pgae145" rel="noopener" target="_blank" id="isPasted">recent study</a>, a group of 28 human participants were given a complex BCI task to track an object in a two-dimensional space all by thinking about it. During the task, an electroencephalography (EEG) method recorded their activity, from outside the brain. Using AI to train a deep neural network, the He group then directly decoded and interpreted human intentions for continuous object movement using the BCI sensor data. Overall, the work demonstrates the excellent performance of non-invasive BCI for a brain-controlled computerized device.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE0NTE3ODg4OTAzLTA0MzAtZW0tbm9uaW52YXNpdmUtYmNpLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Source:&nbsp;He Lab“The innovation in AI technology has enabled us to greatly improve the performance versus conventional techniques, and shed light for wide human application in the future,” expressed <a href="https://engineering.cmu.edu/directory/bios/he-bin.html" rel="noopener" target="_blank" id="isPasted">Bin He</a>, professor of <a href="https://www.cmu.edu/bme/" rel="noopener" target="_blank">biomedical engineering</a> at Carnegie Mellon University.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1714517922793-0430-em-2-noninvasive-bci.gif" style="width: 100%px;" class="fr-fic fr-dib">This video shows the cursor and target trajectories throughout a single 60-second trial with lag adjusted. The randomly moving target is represented by a yellow circle, while the cursor that the subject controlled using the BCI system is shown as a blue circle. A human subject controlled the blue cursor using the deep learning-based BCI decoder just by thinking about it. Source:&nbsp;He LabMoreover, the capability of the group’s AI-powered BCI suggests a direct application to continuously controlling a robotic device.“We are currently testing this AI-powered noninvasive BCI technology to control sophisticated tasks of a robotic arm,” said He. “Also, we are further testing its applicability to not only able-body subjects, but also stroke patients suffering motor impairments.” In a few years, this may lead to AI-powered assistive robots becoming available to a broad range of potential users.To this end, motor-impaired patients who are suffering from spinal cord injury, stroke, or other movement impairment, but do not want to receive an implant, stand to benefit immensely from research in this vein. “We keep pushing noninvasive neuroengineering solutions that can help everybody,” added He.This work was supported in part by the National Institute of Neurological Disorders and Stroke, National Center for Complementary and Integrative Health, and National Institute of Biomedical Imaging and Bioengineering.Other collaborators on the PNAS Nexus paper include the first authors Dylan Forenzo, BME Ph.D. student, and Hao Zhu, former BME Masters student; Jenn Shanahan, former lab technician; and Jaehyun Lim, BME undergraduate student.

Achieving a noteworthy milestone to advance noninvasive brain-controlled interfaces, researchers used AI technology to improve the decoding of human intention and control a continuously moving virtual object all by thinking about it, with unmatched performance.

Refined AI approach improves noninvasive BCI performance

In this episode, we talk about how a fashion company spun-off from MIT has developed the world’s first truly one-size-fits-all garment.

Podcast: Novel Fabrics + Robots -- Sustainable One-size-fits-all Fashion Revolution

A pick and place robot in an industrial setup

Understanding the inception, evolution, composition, and types of pick and place robots and their impact on various industrial sectors 

Pick and Place Robots: An In-Depth Guide to Their Functionality and Applications

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/7a6ed573-f46a-40bb-b396-d38d1207aa2e?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Until recently, bespoke tailoring — clothing made to a customer’s individual specifications — was the only way to have garments that provided the perfect fit for your physique. For most people, the cost of custom tailoring is prohibitive. But the invention of active fibers and innovative knitting processes is changing the textile industry.“We all wear clothes and shoes,” says Sasha MicKinlay MArch ’23, a recent graduate of the MIT Department of Architecture. “It’s a human need. But there’s also the human need to express oneself. I like the idea of customizing clothes in a sustainable way. This dress promises to be more sustainable than traditional fashion to both the consumer and the producer.”McKinlay is a textile designer and researcher at the Self-Assembly Lab who designed the <a href="https://selfassemblylab.mit.edu/4d-knit-dress" target="_blank">4D Knit Dress</a> with Ministry of Supply, a fashion company specializing in high-tech apparel. The dress combines several technologies to create personalized fit and style. Heat-activated yarns, computerized knitting, and robotic activation around each garment generates the sculpted fit. A team at Ministry of Supply led the decisions on the stable yarns, color, original size, and overall design.“Everyone’s body is different,” says Skylar Tibbits, associate professor in the Department of Architecture and founder of the Self-Assembly Lab. “Even if you wear the same size as another person, you're not actually the same.”<iframe width="640" height="360" src="https://player.vimeo.com/video/893959788" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;&nbsp;&nbsp;</iframe>4D Knit Dress: Transforming Style. Video: Self-Assembly Lab<h2 id="isPasted">Active textiles</h2>Students in the Self-Assembly Lab have been working with dynamic textiles for several years. The yarns they create can change shape, change property, change insulation, or become breathable. Previous applications to tailor garments include <a href="https://selfassemblylab.mit.edu/active-textile-tailoring" target="_blank">making sweaters</a> and <a href="https://news.mit.edu/2022/form-fitting-face-mask-lavender-tessmer-0718" target="_blank">face masks</a>. Tibbits says the 4D Knit Dress is a culmination of everything the students have learned from working with active textiles.McKinlay helped produce the active yarns, created the concept design, developed the knitting technique, and programmed the lab’s industrial knitting machine. Once the garment design is programmed into the machine, it can quickly produce multiple dresses. Where the active yarns are placed in the design allows for the dress to take on a variety of styles such as pintucks, pleats, an empire waist, or a cinched waist.“The styling is important,” McKinlay says. “Most people focus on the size, but I think styling is what sets clothes apart. We’re all evolving as people, and I think our style evolves as well. After fit, people focus on personal expression.”<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzEwMTM1NjY2MDgwLU1pbmlzdHJ5X0ZJTkFMX2RyZXNzZXMuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">Designers explored this series of possible styles for the robotic heat activation of the 4D Knit Dress. Image courtesy of the Self-Assembly Lab.Danny Griffin MArch ’22, a current graduate student in architectural design, doesn’t have a background in garment making or the fashion industry. Tibbits asked Griffin to join the team due to his experience with robotics projects in construction. Griffin translated the heat activation process into a programmable robotic procedure that would precisely control its application.“When we apply heat, the fibers shorten, causing the textile to bunch up in a specific zone, effectively tightening the shape as if we’re tailoring the garment,” says Griffin. “There was a lot of trial and error to figure out how to orient the robot and the heat gun. The heat needs to be applied in precise locations to activate the fibers on each garment. Another challenge was setting the temperature and the timing for the heat to be applied.”It took a while to determine how the robot could&nbsp;reach all areas of the dress.“We couldn’t use a commercial heat gun — which is like a handheld hair dryer — because they’re too large,” says Griffin. “We needed a more compact design. Once we figured it out, it was a lot of fun to write the script for the robot to follow.”A dress can begin with one design — pintucks across the chest, for example — and be worn for months before having heat re-applied to alter its look. Subsequent applications of heat can tailor the dress further.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzEwMTM1NjgyOTQyLVJvYm90aWNfYXJtX2F0X3dvcmsuanBlZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Robotic heat activation can be seen during a demonstration at Ministry of Supply in Boston. Photo: Olivia Mintz<h2>Beyond fit and fashion</h2>Efficiently producing garments is a “big challenge” in the fashion industry, according to Gihan Amarasiriwardena ’11, the co-founder and president of Ministry of Supply.“A lot of times you'll be guessing what a season's style is,” he says. “Sometimes the style doesn't do well, or some sizes don’t sell out. They may get discounted very heavily or eventually they end up going to a landfill.”“Fast fashion” is a term that describes clothes that are inexpensive, trendy, and easily disposed of by the consumer. They are designed and produced quickly to keep pace with current trends. The 4D Knit Dress, says Tibbits, is the opposite of fast fashion. Unlike the traditional “cut-and-sew” process in the fashion industry, the 4D Knit Dress is made entirely in one piece, which virtually eliminates waste.“From a global standpoint, you don’t have tons of excess inventory because the dress is customized to your size,” says Tibbits.McKinlay says she hopes use of this new technology will reduce the amount of waste in inventory that retailers usually have at the end of each season.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzEwMTM1NzEwMDIxLU1pbmlzdHJ5X0ZJTkFMX0RyYWZ0MDUgY29weS5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">A complete 4D Knit Dress comes out of a Stoll industrial knitting machine after being produced in one piece, without any cutting or sewing. Photo courtesy of the Self-Assembly Lab.“The dress could be tailored in order to adapt to these changes in styles and tastes,” she says. “It may also be able to absorb some of the size variations that retailers need to stock. Instead of extra-small, small, medium, large, and extra-large sizes, retailers may be able to have one dress for the smaller sizes and one for the larger sizes. Of course, these are the same sustainability points that would benefit the consumer.”The Self-Assembly Lab has collaborated with Ministry of Supply on projects with active textiles for several years. Late last year, the team debuted the <a href="https://selfassemblylab.mit.edu/4d-knit-dress" target="_blank">4D Knit Dress</a> at the company’s flagship store in Boston, complete with a robotic arm working its way around a dress as customers watched. For Amarasiriwardena, it was an opportunity to gauge interest and receive feedback from customers interested in trying the dress on.“If the demand is there, this is something we can create quickly” unlike the usual design and manufacturing process, which can take years, says Amarasiriwardena.Griffin and McKinlay were on hand for the demonstration and pleased with the results. For Griffin, with the “technical barriers” overcome, he sees many different avenues for the project.“This experience leaves me wanting to try more,” he says.McKinlay too would love to work on more styles.“I&nbsp;hope this research project helps people rethink or reevaluate their relationship with clothes,” says McKinlay. “Right now when people purchase a piece of clothing it has only one ‘look.’ But, how exciting would it be to purchase one garment and reinvent it to change and evolve as you change or as the seasons or styles change? I'm hoping that's the takeaway that people will have.”

Caption:In late 2023 the MIT Self Assembly Lab and the high-tech fashion company Ministry of Supply debuted their 4D Knit Dress at the latter's store Boston, complete with a robotic arm working its way around a dress as customers watched. Credits:Image courtesy of MIT Self Assembly Lab

Developed by the Self-Assembly Lab, the 4D Knit Dress uses several technologies to create a custom design and a custom fit, while addressing sustainability concerns.

Is this the future of fashion?

In this episode, we delve into the cognitive strategies employed by researchers at EPFL to augment the human body with an additional robotic arm and learn about the profound impact of cognitive enhancements on the integration of advanced robotic limbs

Podcast: Why Robotic Limbs Are Critical To The Human Race

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/85f51311-32bd-4a6e-953c-0d6777ee40fc?dark=false" class="fr-draggable" data-gtm-yt-inspected-9="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Neuroengineer Silvestro Micera develops advanced technological solutions to help people regain sensory and motor functions that have been lost due to traumatic events or neurological disorders. Until now, he had never before worked on enhancing the human body and cognition with the help of technology.<iframe width="640" height="360" src="https://www.youtube.com/embed/w78rRbaH7Wg?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-gtm-yt-inspected-8="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-9="true" id="720627293" title="Cognitive strategies for augmenting the body with a wearable, robotic arm">&nbsp;</iframe>Now in a study published in Science Robotics, Micera and his team report on how diaphragm movement can be monitored for successful control of an extra arm, essentially augmenting a healthy individual with a third – robotic – arm.“This study opens up new and exciting opportunities, showing that extra arms can be extensively controlled and that simultaneous control with both natural arms is possible,” says Micera, Bertarelli Foundation Chair in Translational Neuroengineering at EPFL, and professor of Bioelectronics at Scuola Superiore Sant’Anna.The study is part of the <a href="https://nccr-robotics.ch/research/wearable-robotics/third-arm/" target="_blank" rel="noopener">Third-Arm project</a>, previously funded by the Swiss National Science Foundation (NCCR Robotics), that aims to provide a wearable robotic arm to assist in daily tasks or to help in search and rescue. Micera believes that exploring the cognitive limitations of third-arm control may actually provide gateways towards better understanding of the human brain.Micera continues, “The main motivation of this third arm control is to understand the nervous system. If you challenge the brain to do something that is completely new, you can learn if the brain has the capacity to do it and if it’s possible to facilitate this learning. We can then transfer this knowledge to develop, for example, assistive devices for people with disabilities, or rehabilitation protocols after stroke.”“We want to understand if our brains are hardwired to control what nature has given us, and we’ve shown that the human brain can adapt to coordinate new limbs in tandem with our biological ones,” explains Solaiman Shokur, co-PI of the study and EPFL Senior Scientist at the Neuro-X Institute. “It’s about acquiring new motor functions, enhancement beyond the existing functions of a given user, be it a healthy individual or a disabled one. From a nervous system perspective, it’s a continuum between rehabilitation and augmentation.”To explore the cognitive constraints of augmentation, the researchers first built a virtual environment to test a healthy user’s capacity to control a virtual arm using movement of his or her diaphragm. They found that diaphragm control does not interfere with actions like controlling one’s physiological arms, one’s speech or gaze.In this virtual reality setup, the user is equipped with a belt that measures diaphragm movement. Wearing a virtual reality headset, the user sees three arms: the right arm and hand, the left arm and hand, and a third arm between the two with a symmetric, six-fingered hand.“We made this hand symmetric to avoid any bias towards either the left or the right hand,” explains Giulia Dominijanni, PhD student at EPFL’s Neuro-X Institute.In the virtual environment, the user is then prompted to reach out with either the left hand, the right hand, or in the middle with the symmetric hand. In the real environment, the user holds onto an exoskeleton with both arms, which allows for control of the virtual left and right arms. Movement detected by the belt around the diaphragm is used for controlling the virtual middle, symmetric arm. The setup was tested on 61 healthy subjects in over 150 sessions.“Diaphragm control of the third arm is actually very intuitive, with participants learning to control the extra limb very quickly,” explains Dominijanni. “Moreover, our control strategy is inherently independent from the biological limbs and we show that diaphragm control does not impact a user’s ability to speak coherently.”The researchers also successfully tested diaphragm control with an actual robotic arm, a simplified one that consists of a rod that can be extended out, and back in. When the user contracts the diaphragm, the rod is extended out. In an experiment similar to the VR environment, the user is asked to reach and hover over target circles with her left or right hand, or with the robotic rod.Besides the diaphragm, but not reported in the study, vestigial ear muscles have also been tested for feasibility in performing new tasks. In this approach, a user is equipped with ear sensors and trained to use fine ear muscle movement to control the displacement of a computer mouse.“Users could potentially use these ear muscles to control an extra limb,” says Shokur, emphasizing that these alternative control strategies may help one day for the development of rehabilitation protocols for people with motor deficiencies.Part of the third arm project, previous studies regarding the <a href="https://actu.epfl.ch/news/a-smart-artificial-hand-for-amputees-merges-user-a/" target="_blank">control of robotic arms</a> have been focused on helping amputees. The latest Science Robotics study is a step beyond repairing the human body towards augmentation.“Our next step is to explore the use of more complex robotic devices using our various control strategies, to perform real-life tasks, both inside and outside of the laboratory. Only then will we be able to grasp the real potential of this approach,” concludes Micera.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTY4NDIyNzU0LTgxOTJ4NTQ2NC5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib">Martina Gini controls a simplified robotic arm with breathing. © 2023 EPFL / Alain Herzog, CC-BY-SA<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTY4NTQ4Mjg5LWV6Z2lmLTUtNDVjNDI4M2Y2Zi5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib">Silvestro Micera (left) and Leonardo Pollina (right) prepare for the third arm virtual reality experiment. © 2023 EPFL / Alain Herzog, CC-BY-SA

EPFL scientists show that breathing may be used to control a wearable extra robotic arm in healthy individuals, without hindering control of other parts of the body.

Cognitive strategies to augment the body with an extra robotic arm

Imagine that dinner could be prepared by a robot in your own home with just the push of a button. It sounds like something straight out of a far-flung future. Now, researchers at CMU are laying the groundwork for this technology with artificial intelligence.&nbsp;Alison Bartch, Atharva Dikshit, and Abraham George, under the guidance of <a href="https://engineering.cmu.edu/directory/bios/barati-farimani-amir.html" target="_blank">Amir Barati Farimani</a>, assistant professor of <a href="https://www.meche.engineering.cmu.edu/index.html" rel="noopener" target="_blank">mechanical engineeringOpens in new window</a>, trained a robot arm on different fruit slices and shapes to help improve its accuracy with cutting and moving real pieces of food on a cutting board.&nbsp;“So I’m cooking and have a cluttered cutting board. I’m reasoning about how to chop and how to select the objects that I want to cut, without cutting things that I don’t want,” said Bartch, a Ph.D. candidate in mechanical engineering. “It’s something that humans are pretty good at. And it’s a surprisingly difficult problem for robots.”<blockquote>It’s something that humans are pretty good at. And it’s a surprisingly difficult problem for robots.<cite>Alison Bartch<cite id="isPasted">,&nbsp;Ph.D. student<cite id="isPasted">, Mechanical Engineering</cite></cite> </cite></blockquote>The researchers combined two vision foundational models—models trained on large visual data sets—to help the robot arm recognize the shape and the type of fruit and vegetable slices. The AI would then determine what motion to use on these slices to produce the correct results with 70% accuracy.One of the models was only trained on whole fruits and vegetables, so the researchers needed to fine-tune the AI to more accurately recognize slices with a relatively small amount of newly collected data. According to Bartch, their fine-tuned framework can potentially be used for other related research; the combined foundational models, however, can be used for a variety of other AI projects in the future.“That’s just one of the things in a lot of robotics and machine learning—a lot of the data sets are already collected,” said George, also a Ph.D. candidate in mechanical engineering.&nbsp;Bartch said that the research focuses more on the “practicalities” of cutting food, such as producing the correct number of slices and moving pieces out of the way, making this research more applicable to a real kitchen than other research.“For this work, we focused more on the pipeline surrounding it, as opposed to the actual chop action,” Bartch said. “You want to pick which object to cut. You want to clear the scene before you cut. After you cut, detecting where the objects are then and continuing to plan. A lot of existing work focused on cutting assumed the position of the vegetable and that there is nothing else in the environment.”&nbsp;However, it will still be a while before AI will be chopping vegetables in homes. According to Bartch, there is still a lot of research and fine-tuning needed, and their robot still needs some human input. It’s difficult to replicate human reasoning in machines, even with tasks many humans find simple.“I think there is definitely a really long way to go before it’s in a person’s house.” Bartch said. “Because reasoning about cooking is really difficult. When you want to go into more practical situations, you need the system to be able to reason when the food is done.Atharva Dikshit, a master’s student in mechanical engineering, also contributed to this research. This research was submitted to the Journal of Intelligent and Robotic Systems.

Researchers at CMU combined two vision foundational models—models trained on large visual data sets—to help a robot arm recognize the shape and the type of fruit and vegetable slices.

Chop, chop: Improving food prep with the power of AI

Engineers at Princeton University and Google have come up with a new way to teach robots to know when they don’t know. The technique involves quantifying the fuzziness of human language and using that measurement to tell robots when to ask for further directions. Telling a robot to pick up a bowl from a table with only one bowl is fairly clear. But telling a robot to pick up a bowl when there are five bowls on the table generates a much higher degree of uncertainty — and triggers the robot to ask for clarification.Because tasks are typically more complex than a simple “pick up a bowl” command, the engineers use large language models (LLMs) — the technology behind tools such as ChatGPT — to gauge uncertainty in complex environments. LLMs are bringing robots powerful capabilities to follow human language, but LLM outputs are still frequently unreliable, said <a href="https://engineering.princeton.edu/faculty/anirudha-majumdar" target="_blank">Anirudha Majumdar</a>, an assistant professor of <a href="https://mae.princeton.edu/" target="_blank">mechanical and aerospace engineering</a> at Princeton and the senior author of a <a href="https://openreview.net/forum?id=4ZK8ODNyFXx" target="_blank">study</a> outlining the new method.“Blindly following plans generated by an LLM could cause robots to act in an unsafe or untrustworthy manner, and so we need our LLM-based robots to know when they don’t know,” said Majumdar.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1701898003603-knowno-ambiguities_WEB-2048x871.jpg" style="width: 766px;" class="fr-fic fr-dib">The researchers tested their method on a simulated robotic arm and on two types of robot hardware. The most complex experiments (left) involved a robotic arm mounted on a wheeled platform and placed in an office kitchen with a microwave and a set of recycling, compost and trash bins. Another set of experiments used a tabletop robotic arm (right) tasked with sorting a set of toy food items into two different categories; a setup with a left and right arm (center) added an additional layer of ambiguity. Images by the researchersThe system also allows a robot’s user to set a target degree of success, which is tied to a particular uncertainty threshold that will lead a robot to ask for help. For example, a user would set a surgical robot to have a much lower error tolerance than a robot that’s cleaning up a living room.“We want the robot to ask for enough help such that we reach the level of success that the user wants. But meanwhile, we want to minimize the overall amount of help that the robot needs,” said&nbsp;<a href="https://allenzren.github.io/" target="_blank">Allen Ren</a>, a graduate student in mechanical and aerospace engineering at Princeton and the study’s lead author. Ren received a&nbsp;<a href="https://www.corl2023.org/awards" target="_blank">best student paper award</a> for his Nov. 8 presentation at the Conference on Robot Learning in Atlanta. The new method produces high accuracy while reducing the amount of help required by a robot compared to other methods of tackling this issue.The researchers tested their method on a simulated robotic arm and on two types of robots at Google facilities in New York City and Mountain View, California, where Ren was working as a student research intern. One set of hardware experiments used a tabletop robotic arm tasked with sorting a set of toy food items into two different categories; a setup with a left and right arm added an additional layer of ambiguity.The most complex experiments involved a robotic arm mounted on a wheeled platform and placed in an office kitchen with a microwave and a set of recycling, compost and trash bins. In one example, a human asks the robot to “place the bowl in the microwave,” but there are two bowls on the counter — a metal one and a plastic one.The robot’s LLM-based planner generates four possible actions to carry out based on this instruction, like multiple-choice answers, and each option is assigned a probability. Using a statistical approach called conformal prediction and a user-specified guaranteed success rate, the researchers designed their algorithm to trigger a request for human help when the options meet a certain probability threshold. In this case, the top two options — place the plastic bowl in the microwave or place the metal bowl in the microwave — meet this threshold, and the robot asks the human which bowl to place in the microwave.In another example, a person tells the robot, “There is an apple and a dirty sponge … It is rotten. Can you dispose of it?” This does not trigger a question from the robot, since the action “put the apple in the compost” has a sufficiently higher probability of being correct than any other option.“Using the technique of conformal prediction, which quantifies the language model’s uncertainty in a more rigorous way than prior methods, allows us to get to a higher level of success” while minimizing the frequency of triggering help, said the study’s senior author&nbsp;<a href="https://engineering.princeton.edu/faculty/anirudha-majumdar" target="_blank">Anirudha Majumdar</a>, an assistant professor of mechanical and aerospace engineering at Princeton.Robots’ physical limitations often give designers insights not readily available from abstract systems. Large language models “might talk their way out of a conversation, but they can’t skip gravity,” said coauthor&nbsp;<a href="https://andyzeng.github.io/" target="_blank">Andy Zeng</a>, a research scientist at Google DeepMind. “I’m always keen on seeing what we can do on robots first, because it often sheds light on the core challenges behind building generally intelligent machines.”Ren and Majumdar began collaborating with Zeng after he gave a talk as part of the&nbsp;<a href="https://robo.princeton.edu/seminar" target="_blank">Princeton Robotics Seminar</a> series, said Majumdar. Zeng, who earned a computer science Ph.D. from Princeton in 2019, outlined Google’s efforts in using LLMs for robotics, and brought up some open challenges. Ren’s enthusiasm for the problem of calibrating the level of help a robot should ask for led to his internship and the creation of the new method.“We enjoyed being able to leverage the scale that Google has” in terms of access to large language models and different hardware platforms, said Majumdar.Ren is now extending this work to problems of active perception for robots: For instance, a robot may need to use predictions to determine the location of a television, table or chair within a house, when the robot itself is in a different part of the house. This requires a planner based on a model that combines vision and language information, bringing up a new set of challenges in estimating uncertainty and determining when to trigger help, said Ren.The paper, “<a href="https://openreview.net/forum?id=4ZK8ODNyFXx" target="_blank">Robots That Ask for Help: Uncertainty Alignment for Large Language Model Planners</a>,” was presented Nov. 8 at the Conference on Robot Learning. In addition to Ren, Zeng and Majumdar, coauthors include Anushri Dixit and Alexandra Bodrova of Princeton; and Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu and Dorsa Sadigh of Google DeepMind. The research was supported in part by the U.S. National Science Foundation and the Office of Naval Research.

Engineers at Princeton University and Google have come up with a new way to teach robots to know when they don’t know and ask for clarification from a human. Photo by the researchers

Modern robots know how to sense their environment and respond to language, but what they don’t know is often more important than what they do know. Teaching robots to ask for help is key to making them safer and more efficient.

How do you make a robot smarter? Program it to know what it doesn't know.

Soft Robotics Podcast-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Soft Robotics Podcast-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Soft Robotics Podcast-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Soft Robotics Podcast-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Soft Robotics Podcast-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NEURA Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

NEURA Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NEURA Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NEURA Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

NEURA Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Rozum Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Rozum Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Rozum Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Rozum Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Rozum Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

READY Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

READY Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

READY Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

READY Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

READY Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

YellowFIN Robotic Solutions-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

YellowFIN Robotic Solutions-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

YellowFIN Robotic Solutions-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

YellowFIN Robotic Solutions-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

YellowFIN Robotic Solutions-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Adapta Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Adapta Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Adapta Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Adapta Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Adapta Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Inovo Robotics Ltd-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Inovo Robotics Ltd-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Inovo Robotics Ltd-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Inovo Robotics Ltd-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Inovo Robotics Ltd-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

United Robotics Group-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

United Robotics Group-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

United Robotics Group-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

United Robotics Group-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

United Robotics Group-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NODE Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

NODE Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NODE Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NODE Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

NODE Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

robotic arms

robotic arms

Profiles