<section id="isPasted">Even the most advanced robots struggle to sense touch as effectively as humans do -- just think of all the “fail” videos online where robots grasp eggs and accidentally crack them, or a robot arm moves too fast and smashes a bottle to pieces. Touch has been a huge challenge for roboticists who are working to develop more sophisticated systems that enable robots to accomplish tasks with human-like precision and adaptability.<h2>New tool gives machines a new level of dexterity and precision</h2>To address this challenge,&nbsp;<a href="https://yunzhuli.github.io/">Yunzhu Li</a>, an assistant professor of&nbsp;<a href="https://www.cs.columbia.edu/">computer science</a> at&nbsp;<a href="https://www.engineering.columbia.edu/">Columbia Engineering</a>, has developed a robot learning system that fuses the sense of touch with sight, giving machines a new level of dexterity and precision. His latest paper introduces&nbsp;<a href="https://arxiv.org/pdf/2410.24091">3D-ViTac</a>, a multi-modal robot manipulation system that is the first to integrate visual and tactile data into a unified 3D point cloud space for imitation learning. This system allows robots to “see” and “feel” objects, handle fragile objects, and perform long-horizon, precise tasks far more effectively than vision-only systems. Li and collaborators presented the paper Nov. 6 at the&nbsp;<a href="https://www.corl.org/">Conference on Robot Learning (CoRL)</a> in Munich, Germany.“Our tactile sensors bring robots' sensing capabilities a step closer to human levels,” said Li, who joined Columbia this year. This research builds on his previous work on a&nbsp;<a href="https://stag.csail.mit.edu/">tactile glove for understanding how humans use their hands</a>. “The sensors are flexible, scalable, cost-effective, and able to detect detailed contact patterns, making them superior to previous tactile sensors used by robots.”<h2>New approach&nbsp;</h2>In this work, the researchers introduce a high-resolution force sensor, made from force-sensitive materials (specifically, piezoresistive materials), to the robot learning community to mimic human tactile sensing capabilities. The piezoresistive tactile sensor converts applied pressure into changes in electrical resistance. This change in resistance enables the robot to “feel” the objects it interacts with, allowing for precise grip strength and accurate in-hand object manipulation.</section><section><h2>3D-ViTac in Action</h2><iframe width="640" height="360" src="https://www.youtube.com/embed/ZN3mHMNZC7Q?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>This contact-rich data provides a clear demonstration of how touch can enhance manipulation and how combining touch with vision can bring robots closer to human-level dexterity. The system was tested on tasks like grasping fragile items, such as eggs and grapes, and performing in-hand manipulation, like reorienting a hex key between fingers or adjusting the grip on a spatula. The tactile and visual fusion significantly outperformed systems relying solely on visual inputs, which was especially useful when handling fragile items or doing tasks with limited visibility.<h2>Thin, flexible tactile sensors transform robots from clunky tools into ones capable of precise, fluid manipulation&nbsp;</h2>The researchers developed a dense, flexible tactile sensor array integrated into a soft robotic gripper. The data from the sensors, combined with visual data, generate a 3D-point cloud, like a visual representation or a scene, that enables the robot to both “see” and “feel” its surroundings. The tactile feedback allows the robot to adjust its grip strength in real time, which is especially crucial when visual information is limited or occluded.&nbsp;Equipped with “fingers” capable of feeling the world around them, these robots can now handle fragile objects with care. Thin, flexible tactile sensors cover their hands, enabling them to perceive the slightest pressure and adjust their movements accordingly. This innovation has transformed the robots from clunky tools into ones capable of precise, fluid manipulation once thought impossible for machines.“This breakthrough also enables robots to handle occluded objects more reliably and effectively,” said&nbsp;<a href="https://binghao-huang.github.io/">Binghao Huang</a>, the project lead and a Columbia Engineering PhD student who works with Li. Occlusion occurs when an object is hidden from view, which is problematic for robots that rely on visual information to manipulate objects. “As the demand for humanoid robots to assist with household chores grows, our bimanual system equipped with tactile sensors presents a promising solution for achieving such tasks.”<h2>What’s next?</h2>With this leap forward in robotics, the line between human and machine skills begins to blur, opening the door to a future where robots cannot only see the world but feel it, too.&nbsp;The next step for the researchers is to further improve the system's scalability and to scale up data collection. They are also developing a tactile simulation and integrating it into the robot learning process. This will allow for larger-scale data collection and better generalization of the policy, enabling the system to perform well in new situations for which it was not explicitly trained.<hr><h2>About the Study</h2>Conference:&nbsp;<a href="https://www.corl.org/">Conference on Robot Learning (CoRL)</a> in Munich, Germany, November 6-9, 2024.Title:&nbsp;<a href="https://binghao-huang.github.io/3D-ViTac/">3D ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing</a>Authors: Binghao Huang1, Yixuan Wang1, Xinyi Yang2, Yiyue Luo3, Yunzhu Li1,&nbsp;<ol><li>Columbia University</li><li>University of Illinois Urbana-Champaign</li><li>University of Washington</li></ol>Funding: This work is partly funded by the Toyota Research Institute (TRI). This research solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity.</section>

Columbia Engineering computer scientists introduce a novel system that mimics human tactile sensing capabilities in robots.

Advancing Robotics with the Power of Touch

In a world where automation dominates the manufacturing sector, one question persists: why aren't robots omnipresent in our daily lives? The answer lies in the intricate challenges of bridging the gap between humans and machines. While robots thrive in structured environments, the unpredictability and complexity of human behavior pose significant hurdles to seamless integration.&nbsp;In Robotican's recent work, we have developed a new way for humans to interact with service robots. A person is talking to a robot and command it to perform series of tasks a spoken language. Such a command can be: "Go forward up until the end of the hallway, enter the room in front of you, ask the doctor to give you the examination files, and come back here". The robot uses a Speech-To-Text engine, to write the command in text, then it applies a specially trained LLM (Large Language Model) to divide the command into sub-tasks. In our example the sub-tasks will be: "move forward, enter the room, ask the doctor, go back".&nbsp; The robot start performing the sub-tasks according to the required order. &nbsp;While moving the robot uses it cameras to "see" the environment in front of it. The frame from this camera goes into an LVM (Large Visual Model) which receives a photo and output text. When the way is clear this LVM will output "move forward", and an action will be: send the robot a navigation way point 10 meters ahead". When say, in our example, a group of people are standing in front of the robot blocking the way. Then the LVM will output "People are blocking the way", and the action will be "ask them to let me pass through". &nbsp;Then a Text-To-Speech engine will say "Excuse me, I'm a robot with an important task. May I please pass through? Thank you.” Then the robot pass and continues its task. This process is repeated until the robot complete its task, or is unable to do it. <img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1731928237835-%D7%AA%D7%9E%D7%95%D7%A0%D7%94+2+%D7%9C%D7%9E%D7%90%D7%9E%D7%A8+2+%D7%95%D7%95%D7%99%D7%91%D7%95%D7%9C%D7%91%D7%A8.jpg" style="width: 100%px;" class="fr-fic fr-dib">"The answer lies in the intricate challenges of bridging the gap between humans and machines" This is a unique approach of combining TTS, STT, LLM and LVM to allow robot interact with human, and human interact with a robot in a spoken language, while understanding tasks, seeing and understanding the environment around it, and asking people to help the robot conducting its task successfully. Enter Robotican, a pioneering force at the forefront of autonomous robotics, spearheading the charge to revolutionize Human-Robot Interfaces. As a key player in the Israel Innovation Authority's groundbreaking consortium, Robotican harnesses its extensive expertise and recent advancement in AI and Deep Learning to develop innovative solutions that allows robots to navigate amongst people while interacting with them similarly to the way other peoples do.&nbsp;Championing the quest to teach robots to navigate unstructured environments effortlessly, Robotican embarks on a transformative journey empowered by cutting-edge Large Language and Vision Models (LLM &amp; LVM). Leveraging the revolutionary capabilities of these models, Robotican engineers have unlocked the potential to communicate with robots in a language they understand.&nbsp;Omri Eitan, Robotican's AI expert, sheds light on this groundbreaking advancement, stating, "The transformative power of the latest Language Model technology has ushered in a new era of complexity in decision-making and deep contextual reasoning. Tasks once deemed intensive engineering endeavors now undergo fine-tuning, and interactions once artificial now feel natural." &nbsp; With Robotican's innovative approach, robots decode textual or verbal instructions, process visual data using LVM to navigate crowded environments, and seamlessly engage in dialogue with individuals, courtesy of LLM. This paradigm shift not only enhances precision in task execution but also fosters meaningful human-like interactions, paving the way for unprecedented advancements in automation and artificial intelligence.&nbsp;Witness the dawn of a new era as Robotican redefines the landscape of human-robot interaction, propelling automation into realms previously deemed unimaginable. Join us as we embark on this transformative journey, shaping a future where robots seamlessly integrate into every facet of our lives. #RoboticanRevolution #HumanRobotInteraction #AIInnovation #AutomationAdvancements

Ghostrobotics V60 with Robotican's ARK (autonomous robotic kit)  

Pioneering AI-Driven Collaboration: Transforming Social Navigation into daily routine

Unveiling the Future: Human-Robot Interaction Revolutionizes Social Navigation Using AI and Deep Learning

In the international Cybathlon competition, people with physical disabilities undertake routine tasks with the aid of assistive technology that can be seamlessly integrated into everyday life. Below, we present three ETH teams that will be putting their innovative solutions to the test at this October’s event.<h2 id="isPasted">Help from a fourlegged friend</h2>Developed by the Robotics Systems Lab at ETH&nbsp;Zur­ich, ALMA is a quadrupedal robot equipped with a range of sensors and a robotic arm. Unlike other entries in the Cybathlon discipline for assistive robots, all of which are attached to a wheelchair, ALMA moves autonomously on its own four legs, controlled by paraplegic pilot Sammy Kunz via a quadstick – a type of mouth-operated joystick. Deploying the robot as a separate system provides greater mobility, say team managers Carmen Scheidemann and Andrei Cramariuc. “But our biggest asset is also our biggest liability,” Scheidemann explains. “Because we have to ensure that ALMA moves safely without bumping into any objects.” In competition, the team has ten minutes to complete ten everyday tasks. For example, the pilot must use the robot to open a door and then close it behind them, or to pick up an apple and hold it in such a way that the pilot can take a bite. The quadstick used with ALMA features various tubes. By blowing or sucking, the pilot is able to control the different axes of the robot: either its quadrupedal platform or its gripping arm.“At a preliminary event in February, we had eight minutes to complete four tasks,” says Scheidemann. “In the end, we needed just over six.” The team is therefore confident about achieving a good score at the upcoming championship. In preparation for October’s event, various refinements to ALMA have been made in close cooperation with Kunz. Most of these involved enhancing the controls so that Kunz can operate the robot with as little practice as possible. “We suggest an improvement, and Sammy gives us feedback on what works for him and what doesn’t,” explains Scheidemann. “Then we make the necessary modifications.” Whereas 15 hours of practice were required before the preliminary event in February, improvements to the control system have now drastically reduced preparation times.Both managers have hugely enjoyed working with Kunz. “It’s really fulfilling to be working on a project that has the potential to bring about such a major improvement in the day-to-day lives of people with disabilities,” says Cramariuc. ANYmal robots, which provide the quadrupedal platform for ALMA, are already in use for a variety of industrial jobs, including the inspection of oil rigs. The current challenge is to deploy assistive robots not only in controlled envir­onments but also in everyday situations. The team will continue to refine ALMA. “But it will be a good few years before the platform is ready for market,” Scheidemann cautions. “We’re still very much at the experimental stage.”<a href="https://cybathlon.ethz.ch/en/teams/rsl" target="_blank" id="isPasted">To the RSL team</a><h2 id="isPasted">Cameras and a vibrating belt</h2>Pressing the right doorbell, finding a free seat and navigating an area containing obstacles of various heights are just some of the challenges that blind pilot Lukas Hendry will face in the Vision Assist­ance Race at the Cybathlon. Technical support comes from the Sight Guide team, which will guide Hendry using data from two cameras strapped to his chest. Patrick Pfreundschuh, one of the team managers, describes how the technology works: “We use one camera for positioning – that is, to tell us where Lukas is and where he’s heading. The other is a depth camera that takes 3D measurements of the surrounding area.” There’s also a third camera which the pilot holds in his hand to assist with tasks that require objects to be examined in more detail. All three cameras transmit information to a small, lightweight computer that Hendry carries on his back. The computer crunches the data and sends Hendry corresponding instructions via an audio signal. Finding a system with enough computing power was surprisingly difficult, says Pfreundschuh: “When your pilot is in the race, you need to be processing the input as quickly as possible. Otherwise, he might collide with an obstacle, which is obviously something we want to avoid!” The Sight Guide team is hopeful about making further improvements before the Cybathlon, such as making the audio signals that convey the instructions more easily understandable. In addition to the computer and cameras, Hendry also wears a special vibrating belt that tells him in which direction to move. “We don’t want to say exactly how this belt works, because it’s what gives us our competitive edge!” says Pfreundschuh with a smile.The Sight Guide team consists of doctoral candidates and students from the Autonomous Systems Lab at ETH Zurich, the Robotics and Perception Group at the University of Zurich and the Institute for Mechatronic Systems at Zurich University of Applied Sciences (ZHAW). For the doctoral students, the Cybathlon is both inspiring and enjoyable, which is why they are happy to work on the project in their spare time. Every month or two, they meet up with Hendry to perform mock-ups of the tasks they will face in the Cybathlon. Hendry then supplies feedback – for example, on how fast he receives the instructions – and many of the tweaks he suggests can be applied on the spot. Working with the pilot is one of the things that makes this project so special, says Pfreundschuh: “It’s great to see Lukas sailing through tasks that would be impossible for him without our technology. It makes me happy and proud to see the benefits we can provide for blind people simply by putting theory into practice.” The goal is to continue improving the technology once the Cybathlon is over – and Pfreundschuh hopes to find a new generation of students willing to take the reins.<a href="https://sightguide.tech/" target="_blank" id="isPasted">To the sightguide team</a><h2 id="isPasted">Sensing the ground while wearing prosthesis</h2>A key drawback of a prosthetic leg is that wearers are unable to feel the ground beneath their foot. The absence of sensory feedback on where their foot is positioned and how firmly it is touching the floor makes it difficult to walk, especially on uneven terrain or when climbing stairs. This is the main reason many patients decide against a prosthesis. But it doesn’t have to be this way, as Stefan Poth has discovered since becoming a pilot in the Cybathlon. He is taking part in the competition with his own prosthetic leg, which has been fitted with a special insole in the shoe. Developed by the NeuroLegs team, this insole is equipped with sensors that gather information on what the researchers call the foot-ground interaction. This includes feedback on whether Poth’s weight is on the toe, middle part or heel of his prosthetic foot. Data on where he is stepping is sent to a microprocessor attached to his waist. The computer processes the data and converts it into electrical stimulation by activating electrodes on an elastic belt that Poth wears around his leg stump. “Depending on where Stefan is stepping, electrical stimulation is applied in real time to either the front, side or back of the stump to replicate the sensation that would normally be perceived in the foot,” says Noemi Gozzi, co-manager of the NeuroLegs team.Poth’s team for the leg prosthesis race is made up of students from the Neuroengineering Lab at ETH Zurich. Affiliated with the Department of Health Sciences and Technology, the lab is headed by ETH professor Stanisa Raspopovic. This year’s team is building on the valuable experience gained in the previous Cybathlon four years ago. “Instead of starting from scratch, we’ve actually been able to improve our technology this year. For example, we’ve made it a lot more user­friendly,” says Valerio Aurucci, the second of the team’s two managers. The belt Poth wears around his leg stump was developed by this year’s team to avoid having to attach electrodes to his skin one by one, which has also made the system more reliable.Gozzi emphasises how rewarding it is to see their research being put into practice: “We’re trying to develop systems that will make people’s day-to-day lives easier. Stefan’s reaction when he felt a sensation from his foot for the first time was something very special – and it certainly made up for all those hours in the lab!” The Neuroengineering Lab is one of the leading research groups working on peripheral nerve stimulation, and Aurucci sees major benefits in the innovative solution they have developed: “The biggest advantage is that our system can be used with any prosthesis.” Getting to a market-ready product will require more development and additional safety features. The two team managers – both of whom are doing their doctorate at ETH – are therefore carefully considering their next step. Should they stick to research and hand over NeuroLegs to the next generation, or take the plunge into business by setting up a spin-off? The future is theirs to decide.<a target="_blank"></a><a href="https://cybathlon.ethz.ch/en/teams/neuro-legs" target="_blank">To the NeuroLegs team</a>

Helping to open doors: Cybathlon pilot Sammy Kunz uses his mouth to steer the assistance robot, while Carmen Scheidemann from the Robotics Systems Lab monitors the test run. Come October, the pressure will be on. (Photograph: Markus Bertschi / ETH Zurich)

In the international Cybathlon competition, people with physical disabilities undertake routine tasks with the aid of assistive tech that can be seamlessly integrated into everyday life. Here are three ETH teams that will be putting their innovative solutions to the test at this October’s event.

How four-legged robots and smart belts help people overcome everyday barriers

<h2 id="isPasted">At the top of many automation wish lists is a particularly time-consuming task: chores.</h2>The moonshot of many roboticists is cooking up the proper hardware and software combination so that a machine can learn “generalist” policies (the rules and strategies that guide robot behavior) that work everywhere, under all conditions. Realistically, though, if you have a home robot, you probably don’t care much about it working for your neighbors. MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers decided, with that in mind, to attempt to find a solution to easily train robust robot policies for very specific environments.“We aim for robots to perform exceptionally well under disturbances, distractions, varying lighting conditions, and changes in object poses, all within a single environment,” says Marcel Torne Villasevil, MIT CSAIL research assistant in the Improbable AI lab and lead author on a recent paper about the work. “We propose a method to create digital twins on the fly using the latest advances in computer vision. With just their phones, anyone can capture a digital replica of the real world, and the robots can train in a simulated environment much faster than the real world, thanks to GPU parallelization. Our approach eliminates the need for extensive reward engineering by leveraging a few real-world demonstrations to jump-start the training process.”<iframe width="640" height="360" src="https://www.youtube.com/embed/w-L90BhfDFo?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h2 id="isPasted">Taking your robot home</h2>RialTo, of course, is a little more complicated than just a simple wave of a phone and (boom!) home bot at your service. It begins by using your device to scan the target environment using tools like NeRFStudio, ARCode, or Polycam. Once the scene is reconstructed, users can upload it to RialTo’s interface to make detailed adjustments, add necessary joints to the robots, and more.The refined scene is exported and brought into the simulator. Here, the aim is to develop a policy based on real-world actions and observations, such as one for grabbing a cup on a counter. These real-world demonstrations are replicated in the simulation, providing some valuable data for reinforcement learning. “This helps in creating a strong policy that works well in both the simulation and the real world. An enhanced algorithm using reinforcement learning helps guide this process, to ensure the policy is effective when applied outside of the simulator,” says Torne.Testing showed that RialTo created strong policies for a variety of tasks, whether in controlled lab settings or more unpredictable real-world environments, improving 67 percent over imitation learning with the same number of demonstrations. The tasks involved opening a toaster, placing a book on a shelf, putting a plate on a rack, placing a mug on a shelf, opening a drawer, and opening a cabinet. For each task, the researchers tested the system’s performance under three increasing levels of difficulty: randomizing object poses, adding visual distractors, and applying physical disturbances during task executions. When paired with real-world data, the system outperformed traditional imitation-learning methods, especially in situations with lots of visual distractions or physical disruptions.“These experiments show that if we care about being very robust to one particular environment, the best idea is to leverage digital twins instead of trying to obtain robustness with large-scale data collection in diverse environments,” says Pulkit Agrawal, director of Improbable AI Lab, MIT electrical engineering and computer science (EECS) associate professor, MIT CSAIL principal investigator, and senior author on the work.As far as limitations, RialTo currently takes three days to be fully trained. To speed this up, the team mentions improving the underlying algorithms and using foundation models. Training in simulation also has its limitations, and currently it’s difficult to do effortless sim-to-real transfer and simulate deformable objects or liquids.<h2>The next level</h2>So what’s next for RialTo’s journey? Building on previous efforts, the scientists are working on preserving robustness against various disturbances while improving the model’s adaptability to new environments. “Our next endeavor is this approach to using pre-trained models, accelerating the learning process, minimizing human input, and achieving broader generalization capabilities,” says Torne.“We’re incredibly enthusiastic about our 'on-the-fly' robot programming concept, where robots can autonomously scan their environment and learn how to solve specific tasks in simulation. While our current method has limitations — such as requiring a few initial demonstrations by a human and significant compute time for training these policies (up to three days) — we see it as a significant step towards achieving 'on-the-fly' robot learning and deployment,” says Torne. “This approach moves us closer to a future where robots won’t need a preexisting policy that covers every scenario. Instead, they can rapidly learn new tasks without extensive real-world interaction. In my view, this advancement could expedite the practical application of robotics far sooner than relying solely on a universal, all-encompassing policy.”“To deploy robots in the real world, researchers have traditionally relied on methods such as imitation learning from expert data, which can be expensive, or reinforcement learning, which can be unsafe,” says Zoey Chen, a computer science PhD student at the University of Washington who wasn’t involved in the paper. “RialTo directly addresses both the safety constraints of real-world RL [robot learning], and efficient data constraints for data-driven learning methods, with its novel real-to-sim-to-real pipeline. This novel pipeline not only ensures safe and robust training in simulation before real-world deployment, but also significantly improves the efficiency of data collection. RialTo has the potential to significantly scale up robot learning and allows robots to adapt to complex real-world scenarios much more effectively.”"Simulation has shown impressive capabilities on real robots by providing inexpensive, possibly infinite data for policy learning,” adds Marius Memmel, a computer science PhD student at the University of Washington who wasn’t involved in the work. “However, these methods are limited to a few specific scenarios, and constructing the corresponding simulations is expensive and laborious. RialTo provides an easy-to-use tool to reconstruct real-world environments in minutes instead of hours. Furthermore, it makes extensive use of collected demonstrations during policy learning, minimizing the burden on the operator and reducing the sim2real gap. RialTo demonstrates robustness to object poses and disturbances, showing incredible real-world performance without requiring extensive simulator construction and data collection.”Torne wrote this paper alongside senior authors Abhishek Gupta, assistant professor at the University of Washington, and Agrawal. Four other CSAIL members are also credited: EECS PhD student Anthony Simeonov SM ’22, research assistant Zechu Li, undergraduate student April Chan, and Tao Chen PhD ’24. Improbable AI Lab and WEIRD Lab members also contributed valuable feedback and support in developing this project.This work was supported, in part, by the Sony Research Award, the U.S. government, and Hyundai Motor Co., with assistance from the WEIRD (Washington Embodied Intelligence and Robotics Development) Lab.

The moonshot of many roboticists is cooking up the proper hardware and software combination so that a machine can learn “generalist” policies (the rules and strategies that guide robot behavior) that work everywhere, under all conditions.

Precision home robots learn with real-to-sim-to-real

<section id="isPasted">The European Championship is still in full swing but, at TU Eindhoven, they are already hard at work on the next ‘soccer tournament’, the RoboCup 2024 from July 17-21 in Eindhoven. This is the world championship for robotics and involves fully autonomous robots that play soccer matches, perform household tasks or locate difficult-to-reach places as part of rescue work.</section><section tabindex="-1">Thousands of people are expected to visit Eindhoven for RoboCup 2024, the world’s largest competition in robotics and AI. The best teams from around the world will compete for the world title in various robot competitions.Alongside René van de Molengraft, Professor of Robotics and co-founder of the successful TU/e team (and current robot soccer world champion) Tech United, we look ahead to the robot soccer tournament.A conversation about the latest wonders we can see there, the challenges on the way to achieving the grand goal of defeating human soccer players by 2050, and why soccer robots – and robots in general – perform even better when they are lazy.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721776291270-csm_51632710421_b5699b11ea_o_b7131df8d1.jpg" style="width: 100%px;" class="fr-fic fr-dib">Tech United’s soccer robots, with the goalkeeper on the left and a player on the right. In blue, you can see small omni wheels, which can roll both straight ahead and sideways. Photo: Bart van Overbeeke<section tabindex="-1" id="isPasted"><h3>Tech United team</h3>The soccer robots of TU Eindhoven’s team, Tech United, compete in the Middle Size League (MSL), the most spectacular robot soccer. Their healthcare robot, HERO, takes part in the @Home league.Soccer robots are attracting a lot of attention, especially in the MSL. Using a real soccer ball, they play five-on-five matches in two 15-minute halves on an 18 x 12-meter field. They are programmed in advance but, once the whistle blows, they play soccer completely autonomously and humans are simply spectators.They look like a sort of moving traffic cone on three omni wheels. These wheels cannot steer but can passively roll sideways. Thanks to these wheels, the robots are highly maneuverable and reliable.</section><section tabindex="-1"><h3>The latest wonders</h3>RoboCup primarily serves to stimulate the development of technology in mechatronics, computer science, electronics and AI, among other fields, with competition as the drive. After each tournament, the knowledge and progress are therefore shared with all participants. So, Tech United will have to come up with something new this year to go for its eighth (!) world title.“We’re introducing two new elements this year,” says Van de Molengraft. “The first improvement concerns the wheels. This year, we have a player with three steerable wheels. This allows the robot to accelerate and brake much faster. For now, it will be used as a substitute to see if the robot can handle an entire tournament.”“The other improvement is invisible as it relates to the software. We have completely recreated old code from the very beginning and have adapted it to the rest of the code. This code improves the robots’ movements and therefore basically all soccer actions.”<iframe width="640" height="360" src="https://www.youtube.com/embed/cfx4IONMwSU?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>In this episode of Tiki Taka Touzani, presentor Soufiane Touzani and soccer player Ismael Saibari (PSV) visit team Tech United and show that humans can still play the robots out now. Source: NPO3 / Tiki Taka Touzani<h3 id="isPasted">Defeating human soccer players</h3>Every few years, the RoboCup organization, in which Van de Molengraft is closely involved, draws up new rules for the various leagues. In doing so, they encourage innovations to get closer to the dot on the horizon: defeating a human soccer team by 2050. What will the next challenge be on the way to 2050?<blockquote>We need the switch from wheels to legs to win against people.René van de Molengraft, Professor Robotics</blockquote>“I’m thinking of the switch from wheels to legs. We will have to do that because we really cannot compete against humans using wheels yet. For that, we need to allow games with multi-legged robots in the MSL. That would be an enormous new challenge for Tech United,” he explains.“Of course, you will then see a huge drop-off in match results,” says Van de Molengraft. “But eventually, the robots on legs are going to win out over those on wheels. I’m really looking forward to that!”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721776377105-csm_51633359139_8465ebf61b_o_3b5396a5b7.jpg" style="width: 100%px;" class="fr-fic fr-dib">Tech United's @HOME care robot picks up a cup of tea. Photo: Bart van Overbeeke<h3 id="isPasted">Practical challenges</h3>In addition to the soccer leagues, the RoboCup has other categories in which to compete, such as autonomous healthcare robots and rescue robots. For example, Tech United is participating in the @Home league with their healthcare robot HERO.Within this competition, the focus lies in the development of robots that can provide service and assistance in home or care situations. In the future, the hope is that this will enable elderly people to live at home for longer, for example. Here too, the team is mainly a spectator during the matches.<blockquote>At the RoboCup, practical challenges are at the center of the care and rescue leagues.René van de Molengraft, Professor Robotics</blockquote>“You can see that highly practical challenges are at the center of the other leagues, such as picking up a cup of tea. The care robot has to know how big the cup is and how full it is, but also what material such a cup is made of. A plastic cup is easily crushed, resulting in hot tea everywhere. I see significant progress there, although the step to practice is still sizeable.”“It’s a different story for rescue robots. Robots are already being deployed in rescue work in which humans are difficult to reach or where it is too dangerous. Examples include the disaster at Fukushima or the earthquakes in Turkey and Syria.”“It’s really great to see rescue robots finding living people in incredibly difficult conditions, albeit usually still requiring a human operator to remotely control the robot.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721776418415-csm_51632932133_5d5131f4ce_o_f70bccc248.jpg" style="width: 100%px;" class="fr-fic fr-dib">Close-up of the part that the soccer robot uses to accept the ball and shoot it away. Photo: Bart van Overbeeke<h3 id="isPasted">At the heart of the research</h3>The TU/e team Tech United has been around for almost twenty years now. Van de Molengraft was involved in the foundation of the RoboCup soccer team, taking inspiration from Philips’ then-successful team. A little later, in 2009, followed the start of research into autonomous robotics at TU/e with the RoboEarth<a href="https://www.tue.nl/en/research/research-institutes/robotics-research/projects/roboearth" target="_blank">&nbsp;project</a>, which had European funding.“After that, we started carrying out more and more robotics projects and became a real research group. In 2019, as principal investigator,&nbsp;I was able to start my own robotics group,” he says.“We began with five people and are now a real section with about fifty employees and a number of robotics courses in the Mechanical Engineering bachelor’s program. And, starting next academic year, we will also have our own Robotics master’s track.”All of that research directly benefits the Tech United team as, in addition to students, the team consists of PhD candidates and staff.<h3 id="isPasted">Intelligent laziness</h3>As of this month, Van de Molengraft may call himself a Professor of Lazy Robotics. By ‘lazy robots’, he means robots that intelligently recognize which information they can safely ignore. “Normal robots, such as self-driving cars, consider all of the information around them to be equally important. It doesn’t work that way with humans,” explains Van de Molengraft.“Humans are perfectly capable of ignoring, for instance, visual information that is not important to a task.”“Imagine yourself on a bicycle. You perceive all of the information that you do not require for safe participation in traffic, but you don’t store that information. That person walking out of the supermarket you’re cycling past: unimportant!”“A self-driving car cannot do that yet. It has 20 cameras running simultaneously, viewing and analyzing each image. Everything is equally important, which takes a lot of computing power and a lot of energy. That needs to be improved.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721776461208-csm_52217918068_0eded10aec_k_e581fd5b89.jpg" style="width: 100%px;" class="fr-fic fr-dib">Tech United robots chase the ball. Photo: Tech UnitedHe gives another example: “We humans sit down or lean on a table when we don’t need to move. That costs us less energy. We want the same for robots. They have to become ‘intelligently lazy’, so to speak, and be able to assess which information is highly important for safety or their task and then focus on that.”“But we still have a lot of work to do to get robots to be as intelligently ‘lazy’ as humans are, both in terms of software and hardware. And that’s what we are working on at TU/e.”<h3>Lazy soccer robots</h3>Van de Molengraft laughs at the question of whether the best soccer robots should therefore be lazy. “Yes, that would certainly help. They wouldn’t always have to drive to the right spot down to the millimeter, for example. They only need to do that when receiving and shooting the ball.”“When a robot is simply moving freely, a few centimeters don’t really matter. We could make some gains there or do research on that. It does require some real rethinking on our part, but that’s what makes it such a great challenge that, to this day, never bores.”</section></section>

Tech United is the defending champion in the Middle Size League of the RoboCup, the most spectacular robot soccer. They have already been the world champion seven times. Photo: Bart van Overbeeke

Alongside René van de Molengraft, Professor of Lazy Robotics and founder of Tech United, we dive into the technology behind the RoboCup.

Why smart robots should be as lazy as possible

Humanoid robots are designed to resemble and mimic human actions and interactions, placing them as a bridge forming collaborations between humans and machines. Their human-like appearance and movements create a sense of relatability and approachability not typically seen in traditional industrial robots. This "personal appeal" has sparked increased interest in developing humanoid robots for diverse applications, such as healthcare, customer service, education, and even space exploration - where human characteristics such as empathy, decision-making making, and personality are deemed valuable.&nbsp;This article provides insights into four of the leading humanoid robot projects from around the world.&nbsp;<h2>Selection Rationale</h2>The humanoid robots selected in this comparison represent the pinnacle of the current state of humanoid robotic technology. Each robot was selected based on its unique contributions to its field, ranging from technical capabilities to broadest application.&nbsp;Each selected robot is also in full production and either available for purchase or being used by their creators in real-time applications.&nbsp;<h2>Optimus by Tesla</h2>Also known as Tesla Bot, this general-purpose, bipedal, autonomous humanoid robot was created to perform unsafe, repetitive, or boring tasks. The company announced the robot's development in August 2021 and showed the prototype in 2022. Tesla's CEO, Elon Musk, claimed that Optimus will eventually be able to do "anything that humans don’t want to do."Tesla's Optimus can sort colored blocks by color, locate its limbs in space, do some yoga, and even poach an egg. In December 2023, Musk released a video showing an Optimus Generation 2 walking and dancing. The new robot features a slimmer figure and sleeker, more precise movements.&nbsp;The robot is planned to measure 5 ft 8 in (173 cm) tall and weigh 125 lb (57 kg), and it will be controlled by the same AI system that Tesla uses in its cars. With a carrying capacity of 45 lb (20 kg), it can provide manufacturing assistance, and Musk has already commented that the robots will be utilized in Tesla’s car factories by the end of 2024 to enhance the factory's efficiency and significantly reduce production costs.&nbsp;The first commercial units of Tesla's Optimus will launch in 2025.<img width="624" src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1719036519097-1719036519097.png" class="fr-fic fr-dii" style="width: 778px;">Optimus, by Tesla, promises to be the future of automation in manufacturing. Credit:&nbsp;<a href="https://www.tesla.com/en_eu/AI" target="_blank">Tesla</a><h2>Atlas by Boston Dynamics</h2>Atlas, engineered by leading robotic company Boston Dynamics, represents a huge leap forward for robotic mobility and adaptability. Debuting on July 11, 2013, Atlas was designed to traverse and interact with its surroundings with a fluidity and precision reminiscent of human motion. Constructed with robust plastics, titanium, and aluminum 3D printed parts, Atlas was originally &nbsp; driven by powerful hydraulic actuators, but now, the company has announced a fully electric robot. In widely shared videos, Atlas has demonstrated a broad spectrum of physical prowess, including running, leaping, and executing backflips easily. These types of motion are difficult as they require the robot to self-correct and consider environmental factors.&nbsp;Atlas uses a comprehensive sensor network powered by AI to facilitate meticulous navigation and the proficient completion of multifaceted tasks across diverse landscapes. Such capabilities mean Atlas may have a future in scenarios such as search and rescue or fire management, where its agility and adaptability can be leveraged to great effect.&nbsp;After being acquired by &nbsp;Hyundai in 2021, the focus will be testing and iterating Atlas applications in manufacturing environments. The company promises that the electric version of Atlas is stronger and has a broader range of motion than previous models.&nbsp;<iframe width="640" height="360" src="https://www.youtube.com/embed/-e1_QhJ1EhQ?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> <h2>NAO by Aldebaran</h2>NAO is an autonomous, programmable humanoid robot originally developed by Aldebaran Robotics. SoftBank Robotics acquired the French company in 2015 and released NAO 3. In July 2022, the company was rebranded Aldebaran again.Nao is available as a research robot for schools, colleges and universities to teach programming as well as conducting research into human-robot interactions. Over 5,000 Nao robots are used in 200+ educational and research institutions in 70 countries. The robots are also used in healthcare scenarios, such as care homes.NAO6, the latest version, features 25 degrees of freedom and an inertial measurement unit with an accelerometer, gyroscope, and four ultrasonic sensors that support NAO with stability and movement. The robots are also equipped with seven touch sensors to perceive their environment and their speech recognition is available in 20 languages. The robot also has 5G capabilities.&nbsp;<img border="0" width="624" src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1719036519109-1719036519109.png" class="fr-fic fr-dii">From supporting the education of autistic children to helping train International Space Station astronauts and assisting elderly patients in care homes, NAO robots have been integrated into education and healthcare applications. Credit:&nbsp;<a href="https://www.aldebaran.com/de/nao" target="_blank">Aldebaran</a><h2>Ameca by Engineered Arts</h2>Ameca, first developed by Engineered Arts in February 2021, is currently the most advanced human-shaped robot in the market, developed specifically for human-robot interaction. They are modular, upgradable, and easy to develop upon.With embedded microphones, binocular eye-mounted cameras, a chest camera, &nbsp;and facial recognition software, Ameca's interactions with the public work by GPT-3 or human telepresence. It also has motorized arms, fingers, neck, and facial features on grey rubber skin, designed to appear genderless. Engineered Arts has developed their own hardware, based on proprietory research into humanoid robotics, and built with their Mesmer technology, a system focused on powerful, cost-effective, and realistic humanoid robots.Ameca is currently associated with the Museum of the Future in Dubai (UEA), where people can interact with it.&nbsp;<iframe width="640" height="360" src="https://player.vimeo.com/video/651929733" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> <h2>Technology Overview&nbsp;</h2>&nbsp;<table border="1" cellspacing="0" cellpadding="0" width="632"><tbody><tr><td width="12.341772151898734%" valign="bottom">Robot</td><td width="13.291139240506329%" valign="bottom">Company/Creator</td><td width="11.550632911392405%" valign="bottom">Vision &amp; Goal</td><td width="13.132911392405063%" valign="bottom">Materials</td><td width="17.088607594936708%" valign="bottom">AI Capabilities &amp; Technology</td><td width="16.139240506329113%" valign="bottom">Applications</td><td width="16.455696202531644%" valign="bottom">Command &amp; Agility</td></tr><tr><td width="12.341772151898734%">Optimus</td><td width="13.291139240506329%">Tesla&nbsp;</td><td width="11.550632911392405%">Replace humans in manufacturing operations for repetitive and dangerous tasks</td><td width="13.132911392405063%">Polyether Ether Ketone (PEEK)</td><td width="17.088607594936708%">Tesla-designed actuators and sensors</td><td width="16.139240506329113%">Manufacturing support and training</td><td width="16.455696202531644%">Carrying capacity of 20kg, can dance, boil eggs, do squats</td></tr><tr><td width="12.341772151898734%">Atlas</td><td width="13.291139240506329%">Boston Dynamics</td><td width="11.550632911392405%">Navigate and manipulate environments with human-like agility</td><td width="13.132911392405063%">Titanium and aluminum&nbsp;</td><td width="17.088607594936708%">Precise navigation, execution of complex tasks in varied terrains</td><td width="16.139240506329113%">Search and rescue, exploration, logistics</td><td width="16.455696202531644%">Responsive to commands, capable of running, jumping, performing backflips</td></tr><tr><td width="12.341772151898734%">NAO</td><td width="13.291139240506329%">Aldebaran</td><td width="11.550632911392405%">Human interaction and educational support</td><td width="13.132911392405063%">Polycarbonate-ABS plastic, polyamide, and carbon-fiber-reinforced thermoplastic</td><td width="17.088607594936708%">Conversational AI, personalized interaction based on facial recognition, collect information&nbsp;</td><td width="16.139240506329113%">Education, research, healthcare</td><td width="16.455696202531644%">Walk, dance, speak, and recognize faces and objects</td></tr><tr><td width="12.341772151898734%">Ameca</td><td width="13.291139240506329%">Engineered Arts</td><td width="11.550632911392405%">Development platform for testing AI and machine learning systems</td><td width="13.132911392405063%">Mesmer technology</td><td width="17.088607594936708%">Process complex data, recognize faces, comprehend speech, and adapt to different scenarios</td><td width="16.139240506329113%">Human interaction, a platform for developing AI</td><td width="16.455696202531644%">Advanced facial expression and human-like interactions</td></tr></tbody></table>&nbsp;<h2>Conclusion: The Pioneers of Humanoid Robotics - A Unified Perspective</h2>The exploration of Optimus, Atlas, NAO, and Ameca highlights the remarkable journey and achievements in humanoid robotics. Each robot, distinct in its capabilities and applications, contributes uniquely to the field. From Ameca's social interactions to Atlas's agility, NAO's educational support, and Optimus's manufacturing utility, these pioneers embody the diverse potential of robotics.Their materiality and design philosophies are tailored to specific applications, showcasing innovation in construction and functionality. The AI and technological advancements across these robots cover a broad spectrum, enabling them to perform tasks ranging from basic interactions to complex, autonomous problem-solving. Their deployment in healthcare, customer service, manufacturing exploration, and emergency response illustrates the expanding role of robotics in meeting intricate human needs.The integration of sophisticated command systems and remarkable agility demonstrates the evolving interface between humans and machines, pushing the boundaries of what humanoid robots can achieve. This unified perspective underlines the transformative impact of humanoid robotics, promising a future where their presence becomes an integral part of daily life and complex problem-solving across various sectors.

Exploring innovations, capabilities, and the future of robotic companions

A quick engineering comparison of cutting-edge humanoid robots

For the first time, the assistance robot GARMI demonstrates that it can directly combine specific skills and support seniors throughout the day. With the help of a digital twin, artificial intelligence and ChatGPT, the care assistant from the Technical University of Munich (TUM) conducts caregiving tasks such as bringing water and breakfast to the bedside, booking medical appointments and setting up and facilitating telemedical examinations. It also helps care recipients to get out of bed and do rehabilitation exercises.The GARMI assistance robot is becoming increasingly versatile and intelligent. As researchers from TUM's Munich Institute of Robotics and Machine Intelligence (MIRMI) demonstrated at the 2024 International Conference on Robotics and Automation (ICRA) in Yokohama, Japan, the robot not only understands various commands via ChatGPT. It also autonomously implements and executes a wide range of tasks and skills, such as grasping objects, maneuvering safely and communicating with patients. In addition, it&nbsp;books appointments with doctors for telemedical examinations. "GARMI is now able to perform the various individual&nbsp;skills that we have taught him over the last few years, securely and on-demand, via ChatGPT," explains Geriatronics&nbsp;project&nbsp;lead Dr.&nbsp;Abdeldjallil&nbsp;Naceri.<h2>Human-robot interaction: tools must be reliable and safe</h2>To achieve this, the researchers are combining various technological innovations. Before attempting real-world&nbsp;human interactions, a digital twin&nbsp;is used to avoid collisions and make sure that the robot's movements are safe. Artificial intelligence (AI) helps GARMI grasp and hand over cups and glasses without spilling liquid. And ChatGPT acts as a link between the robot, patients, physiotherapists and doctors. Prof.&nbsp;Naceri&nbsp;draws a parallel between innovations in GARMI and autonomous driving: "Before a new function like autonomous parking assistance is made available to real-world drivers, many development steps are necessary." says the researcher. "The same is true for care robotics. Because this technology will be used where people are present, it must be 100% safe and reliable."<h2>Care assistant with ChatGPT, Digital Twin,&nbsp;and human-like hands</h2>TUM researchers have made significant progress especially in three areas:&nbsp;<h3>1. Dexterity - gripping and moving precisely from a distance</h3>Gripping:&nbsp;The researchers have combined a camera, a robotic arm with seven joints, an artificial hand and artificial intelligence to enable GARMI to simulate the way humans grip objects. First, the camera takes a picture of the object to be grasped and identifies it as a cup, cylinder or ball using neural networks. As the camera only sees the object from one side, the system adds non-visible areas, such as a cup, by comparing what it sees with other images and reconstructing&nbsp;a complete 3D object.&nbsp;The researchers use a color-graded heat map indicating the probabilities of various representations of the object matching the way it actually looks in reality. This makes it possible to decide on the ideal position for the hand for gripping a cup, for example. The complex system now manages to do this correctly nine times out of ten.&nbsp;"After it works with one cup, our system can transfer the methodology to all other cup shapes," says Naceri.Moving objects&nbsp;from a distance:&nbsp;The researchers devised a special experimental setup to investigate whether doctors can work together with patients via telesurgery.&nbsp;To do this, they drew simple shapes on a digital graphics tablet.&nbsp;GARMI was equipped with a pen in one hand and a camera in the other.&nbsp;One room away, GARMI's task was to transfer the researchers'&nbsp;drawings onto a screen - in other words, project a simple drawing into a complex robotic system.&nbsp;It turned out that the best circles, squares, and triangles were created when GARMI used the camera autonomously. This finding will be incorporated into the collaboration between physicians and patients in the future. For example, it is essential to position ultrasound probes as precisely as possible and to perform movements correctly during rehabilitation exercises.Perceiving&nbsp;and navigating&nbsp;surroundings:&nbsp;In a new research paper, the researchers show how tools can be maneuvered&nbsp;around objects.&nbsp;The challenge is to keep an eye on distances while being able to correctly assess the mobility of the robot arm with all its joints. If this is successful, the robot can even evade balls thrown at it.<h3>2. Safety:&nbsp;The&nbsp;tactile&nbsp;robot&nbsp;has a 1 millisecond reaction time</h3>GARMI processes information at a cycle time of 1&nbsp;millisecond (ms). This applies equally to perception, interaction and navigation.&nbsp;The force sensors on the robot arms register the slightest contact and react immediately. If a human accidentally bumps into the robot's arm, it stops within a millisecond for safety reasons. Humans and robots initially meet as digital twins in a virtual environment to rule out accidents. This is essential, as the assistance robot can theoretically reach speeds up to 20 km/h in a care home. In the computer simulation, GARMI uses the Safety Motion Unit to register via sensors and slow down if a person is too near. When the person moves away, GARMI speeds up again.<h3>3. Language: ChatGPT uses a list of commands</h3>The AI tool ChatGPT functions as a translator between technology and humans. It has learned various commands such as "Start rehab," "Show me tomorrow's weather," or "Call the doctor." GARMI uses this tool to communicate with patients. The researchers currently have a list of 15 to 20 commands that trigger certain actions. "Potentially, we can expand it as much as we like," says robotics researcher Naceri. "This will make MIRMI one of the first institutes where robots and humans interact using ChatGPT."&nbsp;<h2>GARMI&nbsp;still learning&nbsp;to cope with the human environment</h2>The new universal GARMI is now active in a model apartment in Garmisch-Partenkirchen. The main field of research will be the further development of hands capable of performing even more refined tasks. It will be several years before GARMI is finally used in care homes. "It's like the autonomous car," says Naceri. "A lot of progress has already been made, but a few details are still missing before it is ready for the human environment."<hr>Publications<ul><li>RETOM: Leveraging Maneuverability for Reactive Tool Manipulation using Wrench-Fields; Felix Eberle, Riddhiman Laha, Haowen Yao, Abdesdjallil Naceri, Luis F.C. Figueredo, Sami Haddadin; ICRA 2024</li><li>Autonomous and Teleoperation Control of a Drawing Robot Avartar; Lingyun Chen, Abdeldjallil Naceri, Abdalla Swikir, Sandra Hirche, Sami Haddadin; ICRA 2024</li><li>Anthropomorphic Grasping with Neural Object Shape Completion; Diego Hidalgo-Carvajal,l Hanzhi Chen, Gemma C. Bettelani, Jaesug Jung, Melissa Zavaglia, Laura Busse, Abdeldjallil Naceri, Stefan Leuttenegger, Sami Haddadin; ICRA 202</li><li>Safe-By-Design Digital Twins for Human-Robot Interaction: A Use Case for Humanoid Service Robots. Jon Škerlj, Mazin Hamad, Jean Elsner, Abdeldjallil Naceri, Sami Haddadin; ICRA 2024</li><li>Video: <a href="https://youtu.be/mL-50vtXHdo" target="_blank" rel="noreferrer">A Day with GARMI: Robot Home Assistence in the Lives of Senior Citizens</a>; S. Bing, J. Skerlj, F. Eberle, A. Teimoorzadeh, X. Chen, M. Forouhar, H. Seghedian, A. Naceri, S. Haddadin; ICRA 2024</li></ul>

Care robot GARMI (front left) in the research laboratory at the Technical University of Munich (TUM) in Garmisch-Partenkirchen is becoming increasingly adept.

For the first time, the assistance robot GARMI demonstrates that it can directly combine specific skills and support seniors throughout the day.

GARMI care robot becomes a universal assistant

Parents spend roughly 300 hours each year cleaning up after their kids. That’s nearly two weeks that could be otherwise spent doing, well, anything else. Researchers in Carnegie Mellon University’s <a href="https://www.meche.engineering.cmu.edu/index.html" rel="noopener" target="_blank" id="isPasted">Department of Mechanical Engineering</a>, in collaboration with Google DeepMind and the University of Washington, are hopeful that parents will regain tidying time and then some with <a href="https://linchangyi1.github.io/LocoMan/" rel="noopener" target="_blank">LocoMan</a>, a four-legged version of the Jetsons’ beloved maid “Rosey the Robot,” that could lend a hand in the not-so-distant future.<iframe width="640" height="360" src="https://www.youtube.com/embed/d-GLiRPiHIE?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;&nbsp;</iframe>Quadrupedal robots, designed to move on four legs much like a dog, are already found leading search and rescue missions, surveying construction sites, and moving parts around factories. Their ability to navigate complex environments with a low center of gravity and light weight make quadrupedal robots a top contender for household use, too. Until now, the missing piece was the dexterity and versatility skills necessary for them to be efficient in daily life.By installing two custom-designed, lightweight manipulators (think robot arms) onto the front legs of a quadrupedal robot, <a href="https://engineering.cmu.edu/directory/bios/zhao-ding.html" target="_blank">Ding Zhao</a> has created a robot versatile enough to open doors, pour drinks, plug a phone in to charge, and—perhaps best of all—clean up after your on-the-go toddlers.<blockquote>While many researchers work on humanoid robots that look like humans, with potential to directly learn from humans or even replace humans in some tasks, we focus on robots that are complementary to humans to team up with them.<cite>Ding Zhao, Associate Professor, Mechanical Engineering</cite></blockquote>“While many researcher work on humanoid robots that look like humans, with potential to directly learn from humans or even replace humans in some tasks, we focus on robots that are complementary to humans to team up with them,” said Zhao, associate professor of mechanical engineering.Unlike humanoid robots that come at a high cost and hefty weight, Zhao’s manipulators can be added to existing lower-cost quadrupedal robots and are compact, lightweight, and easy to fabricate, making them an accessible technology.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE1NzM1MTQ3ODg3LTA1MDktZW0taG9tZS1oZWxwZXItcm9ib3QucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">The core idea is to leverage the leg as an integral component of the “robotic arm” for executing 6D pose manipulation.“We used four commercially available servos and 3D printed a few other parts to bring them to life,” Changyi Lin, a Ph.D. candidate in the CMU Safe AI Lab and the first author of the paper explained.Zhao’s team introduced the robot to a class of five preschool students at The Cyert Center for Early Education. After explaining to children how robots work and what we use them for, the children had the opportunity to interact with Zhao’s robot. The students were impressed with the robot’s capabilities and didn’t want it to leave their classroom.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE1NzM1MTc3MDc1LTA1MTMtZW0yLWhvbWUtaGVscGVyLXJvYm90LnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Ding Zhao (center), with children from the Cyert Center, interacting with the home-helper robot.The group continues to develop LocoMan to perform tasks autonomously by integrating advanced perception and intelligence planning capabilities into its existing whole-body controller.“We anticipate that robots like LocoMan will be ready to help out at home, or in daycares, within the next three years,” said Zhao. “It is a pleasant job just to imagine that I may no longer need to handle my kids’ messy fun.”This work will be presented at the loco-manipulation workshop and the manipulation skills workshop of the 2024 IEEE International Conference on Robotics and Automation.

New locomotion and manipulation technology from Ding Zhao’s lab will enable four-legged robots to lend a hand in the not-so-distant future.

Manipulation technology makes home-helper robot possible

In this episode, we discuss research coming out of Columbia University to better connect humans with robots by having them detect, process, and respond to facial expressions!

Podcast: Finally! A Humanoid Robot You Can Trust

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/f153920a-aca0-4937-863a-2cc925d11434?dark=false" class="fr-draggable" data-gtm-yt-inspected-9="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Columbia engineers build Emo, a silicon-clad robotic face that makes eye contact and uses two AI models to anticipate and replicate a person’s smile before the person actually smiles -- a major advance in robots predicting human facial expressions accurately, improving interactions, and building trust between humans and robots.<section id="isPasted">What would you do if you walked up to a robot with a human-like head and it smiled at you first? You’d likely smile back and perhaps feel the two of you were genuinely interacting. But how does a robot know how to do this? Or a better question, how does it know to get you to smile back?While we’re getting accustomed to robots that are adept at verbal communication, thanks in part to advancements in large language models like ChatGPT, their nonverbal communication skills, especially facial expressions, have lagged far behind. Designing a robot that can not only make a wide range of facial expressions but also know when to use them has been a daunting task.&nbsp;</section><section><h2>Tackling the challenge</h2>The <a href="http://engineering.columbia.edu/" rel="nofollow" target="_blank">Creative Machines Lab</a> at <a href="http://engineering.columbia.edu/" rel="nofollow" target="_blank">Columbia Engineering</a> has been working on this challenge for more than five years. In a new study published today in <a href="https://www.science.org/doi/10.1126/scirobotics.adi4724" rel="nofollow" target="_blank">Science Robotics</a>, the group unveils Emo, a robot that anticipates facial expressions and executes them simultaneously with a human. It has even learned to predict a forthcoming smile about 840 milliseconds before the person smiles, and to co-express the smile simultaneously with the person.The team, led by <a href="https://www.engineering.columbia.edu/faculty/hod-lipson" rel="nofollow" target="_blank">Hod Lipson</a>, a leading researcher in the fields of artificial intelligence (AI) and robotics, faced two challenges: how to mechanically design an expressively versatile robotic face which involves complex hardware and actuation mechanisms, and knowing which expression to generate so that they appear natural, timely, and genuine. The team proposed training a robot to anticipate future facial expressions in humans and execute them simultaneously with a person. The timing of these expressions was critical -- delayed facial mimicry looks disingenuous, but facial co-expression feels more genuine since it requires correctly inferring the human's emotional state for timely execution.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzExNTg1NTM0ODUxLXl1aGFuZy1odWEtb2YtY3JlYXRpdmUtbWFjaGluZXMtbGFiLWZhY2UtdG8tZmFjZS13aXRoLWVtby5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">Yuhang Hu of Creative Machines Lab face-to-face with Emo. Image: Courtesy of Creative Machines Lab<h2 id="isPasted">How Emo connects with you</h2>Emo is a human-like head with a face that is equipped with 26 actuators that enable a broad range of nuanced facial expressions. The head is covered with a soft silicone skin with a magnetic attachment system, allowing for easy customization and quick maintenance. For more lifelike interactions, the researchers integrated high-resolution cameras within the pupil of each eye, enabling Emo to make eye contact, crucial for nonverbal communication.The team developed two AI models: one that predicts human facial expressions by analyzing subtle changes in the target face and another that generates motor commands using the corresponding facial expressions.To train the robot how to make facial expressions, the researchers put Emo in front of the camera and let it do random movements. After a few hours, the robot learned the relationship between their facial expressions and the motor commands -- much the way humans practice facial expressions by looking in the mirror. This is what the team calls “self modeling” – similar to our human ability to imagine what we look like when we make certain expressions.Then the team ran videos of human facial expressions for Emo to observe them frame by frame. After training, which lasts a few hours, Emo could predict people’s facial expressions by observing tiny changes in their faces as they begin to form an intent to smile.“I think predicting human facial expressions accurately is a revolution in HRI. Traditionally, robots have not been designed to consider humans' expressions during interactions. Now, the robot can integrate human facial expressions as feedback,” said the study’s lead author Yuhang Hu, who is a PhD student at Columbia Engineering in Lipson’s lab. “When a robot makes co-expressions with people in real-time, it not only improves the interaction quality but also helps in building trust between humans and robots. In the future, when interacting with a robot, it will observe and interpret your facial expressions, just like a real person.”<iframe width="640" height="360" src="https://www.youtube.com/embed/pWTTzR_wXuQ?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-9="true" id="556492795" title="Human-Robot Facial Co-expression">&nbsp;</iframe>Go inside the Creative Machines Lab to watch Emo's facial co-expression. Video Credit: Yuhang Hu, Columbia Engineering<h2 id="isPasted">What’s next</h2>The researchers are now working to integrate verbal communication, using a large language model like ChatGPT into Emo. As robots become more capable of behaving like humans, Lipson is well aware of the ethical considerations associated with this new technology.“Although this capability heralds a plethora of positive applications, ranging from home assistants to educational aids, it is incumbent upon developers and users to exercise prudence and ethical considerations,” says Lipson, James and Sally Scapa Professor of Innovation in the Department of <a href="https://www.me.columbia.edu/" rel="nofollow" target="_blank">Mechanical Engineering</a> at Columbia Engineering, co-director of the <a href="https://make.columbia.edu/" rel="nofollow" target="_blank">Makerspace</a> at Columbia, and a member of the <a href="https://datascience.columbia.edu/" rel="nofollow" target="_blank">Data Science Institute</a>. “But it’s also very exciting -- by advancing robots that can interpret and mimic human expressions accurately, we're moving closer to a future where robots can seamlessly integrate into our daily lives, offering companionship, assistance, and even empathy. Imagine a world where interacting with a robot feels as natural and comfortable as talking to a friend.”&nbsp;</section>

The study's lead author, Yuhang Hu, a PhD student at Columbia Engineering in Hod Lipson’s lab. Credit: John Abbott/Columbia Engineering

Columbia engineers build Emo, a silicon-clad robotic face that makes eye contact and uses two AI models to anticipate and replicate a person’s smile before the person actually smiles 

Robot, can you say 'Cheese'?

In this compelling podcast, host Raymond Yin, explores the rapidly advancing realm of smart home connectivity, offering valuable insights for tech enthusiasts, homeowners, and those intrigued by the potential of interconnected living environments.<h2 dir="ltr">The Essence of Interoperability in Smart Homes</h2>A central theme of the episode is the critical importance of interoperability in smart homes. Yin and LaPré delve into how devices from various manufacturers can seamlessly communicate and collaborate, creating an integrated and efficient home environment. This aspect of smart home technology is not just a convenience but a necessity in our increasingly interconnected world.<a href="https://www.wevolver.com/article/matter-the-future-of-smart-home-connectivity-and-its-impact-on-product-development" target="_blank" rel="noopener noreferrer">Read more about the Matter protocol here. </a><h3 dir="ltr">Benefits of a Unified Ecosystem</h3>The podcast highlights the numerous advantages of a unified smart home ecosystem. The duo discusses how such integration leads to heightened convenience, bolstered security, and improved energy efficiency. These benefits underscore the transformative impact of smart home technologies on our daily lives.The conversation also acknowledges the challenges and complexities inherent in the smart home technology sector. As the field evolves rapidly, staying abreast of these changes and addressing the emerging issues is essential for both developers and consumers.<h3 dir="ltr">Insights from an Industry Expert</h3>Bringing a wealth of knowledge to the table, Chris LaPré shares his expertise on the current landscape of smart home connectivity. He sheds light on recent advancements in wireless technologies and the significance of open standards in promoting interoperability and shaping the future of connected homes.<h3 dir="ltr">Visioning the Future of Connected Homes</h3>The episode provides a glimpse into the future of smart homes, emphasizing the role of interoperability in its evolution. This forward-looking perspective is particularly engaging for listeners keen to understand where smart home technology is headed.In conclusion, "The Tech Between Us" episode with Raymond Yin and Chris LaPré offers a comprehensive and insightful exploration of the present and future of smart home connectivity. It's a must-listen for anyone interested in the intersection of technology and everyday living, providing a clear view of the technological landscape that lies between us and our homes. The podcast also encourages its audience to subscribe and stay updated on future episodes, ensuring they remain at the forefront of technological advancements in the field.

In a recent episode of the podcast "The Tech Between Us," host Raymond Yin engages in a compelling discussion with Chris LaPré, the Chief Technology Officer at the Connectivity Standards Alliance. 

The Tech Between Us Podcast: Exploring the Matter Protocol for a Smarter Home

Just a year ago, on November 29, 2022, Enchanted Tools announced its entrance into the world of robotics with the introduction of its humanoid robot, Mirokaï. With the goal of producing 100,000 robots by 2032, Enchanted Tools' engineers are working tirelessly to bring their dream to life. Miroki has traveled from Paris to Las Vegas to Tokyo, gaining popularity and acclaim along the way. Laurent Vasse, Enchanted Tools' Industrial Director, shared the resources and technological support provided by maxon teams to help meet these challenges.<h2 dir="ltr">Enchanted Tools: useful robots to re-enchant the world.</h2>Enchanted Tools is a start-up with the goal of transforming the field of robotics. To make this possible, they have developed robot-characters called Mirokaï, which weigh 28kg and stand at 1.23m tall. These robots are designed to be both endearing and useful, with a unique appearance that combines animal and manga characteristics. The Mirokaï serve three main purposes.<ul><li dir="ltr">Move autonomously, thanks to an orbital globe on which they are perched.</li><li dir="ltr">Grip objects, using a universal handle that gives them a 97% success rate.</li><li dir="ltr">Interact with humans, in a timely manner, by being able to understand and execute voice commands.&nbsp;</li></ul><blockquote><blockquote>"The Mirokaï have been designed with simplification in mind: simplifying input, simplifying navigation and simplifying interaction, to help those who help on a daily basis, while providing ethical answers to today's robotics and AI challenges." says Jérôme Monceaux, CEO of Enchanted Tools.</blockquote></blockquote>Mirokaï robots are capable of carrying loads up to 3kg, which makes them suitable for various sectors such as hospitals, hotels, airports, and restaurants. These robots can relieve staff from performing the most tedious and physically demanding logistical tasks. What sets these robots apart from others is their animation-inspired design, and they are programmed to interact with people emotionally to enhance their adaptability.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTAzNTczODU4LVNlzIFxdWVuY2UgMTEuMDBfMjhfMTNfMjcucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib">The Mirokaï robot character weighs 28kg and is 1.23m tall.<blockquote>“Since the first prototype was presented to the press, we have successfully developed a dozen demonstrators, which have enabled us to validate the robot’s impact, particularly with the help of healthcare professionals.” explains Laurent Vasse, Industrial Director at Enchanted Tools.</blockquote>Enchanted Tools' 70 engineers are now moving on to industrialization after a phase of concurrent engineering to strike the perfect balance between time-to-market, performance, and price.Open urban factory project to revolutionize robotic industrializationIn 2024, the Mirokaï will become a reality as the first pilot line is implemented. This will allow the production of approximately 50 robots by the end of the year. Previously, the R&amp;D engineers had designed and assembled demonstrators in the R&amp;D offices. However, with the introduction of the pilot line, the robot assembly process can be designed under conditions that closely mimic the industrial model. Laurent Vasse provided this information.Enchanted Tools is planning to open a factory in the heart of Paris. The company wants to show that robotics can re-enchant the world instead of dehumanizing it. The factory will not only produce robots but also introduce the world of robotics, mechatronics, and AI to people. According to Laurent Vasse, the company's goal is to change people's beliefs about this industry and democratize it. They hope to inspire people to pursue careers in this field as well.Within the factory Enchanted Tools there will be a broad range of activities and opportunities for engagement. Including:&nbsp;<ul><li dir="ltr">Industrial area,&nbsp;where assembly operations can be mastered.</li><li dir="ltr">Maintenance and assistance, where Enchanted Tools customers can benefit from the support they need once the robots are on the market.</li><li dir="ltr">R&amp;D&nbsp;Centre&nbsp;where Enchanted Tools engineers further develop the company's know-how.</li></ul>The factory is scheduled to open its doors in the summer of 2024, when all media attention will be focused on Paris, for the Olympic Games."We imagined this concept of an urban factory as one that could be replicated in other parts of the world, where we encounter the same issues of development, brand awareness, and robot maintenance. We therefore plan to duplicate this concept in North America, China, or Japan, to support our commercial development with this type of entity, which will combine all the skills we need to deploy in these regions" further explains Laurent Vasse.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTAzNzkwNTM4LW1heG9uIEVuY2hhbnRlZCBUb29scyBCdXJlYXUgZXR1ZGVzIDIucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib">maxon Engineers worked closely with the Enchanted Tools team to solve technical challenges.<h2 dir="ltr" id="isPasted">100,000 robots in 10 years</h2>The pilot-line aims to produce around 50 robots in 2024, then 200 - 300 units in 2025. From 2026 onwards, Enchanted Tools is set to use a larger factory to produce a thousand robots a year, then ramp up production to 100,000 robots by 2032. "From the start, our goal has been to produce robots in large quantities, on a massive scale. This means having working methods, in terms of development and industrialization, that are revolutionary for the world of robotics”, explains Laurent Vasse. For the teams at Enchanted Tools, the challenge is to find the right balance between improving control and retaining a form of agility. In other words, Enchanted Tools strives to retain the flexibility that is the strength of start-ups, while applying robust methodologies worthy of more mature industries, to successfully initiate the industrialization phase.&nbsp;<blockquote>"To succeed in this challenge, we needed partners like maxon, capable of following us in this way, with excellent technological mastery and the ability to adapt and buildup in industrialization,", says Laurent Vasse, Industrial Director of Enchanted Tools.</blockquote><h2 dir="ltr">maxon x Enchanted Tools: strong collaboration brings the project to life.</h2>From the start of the project, Enchanted Tools' teams worked hand in hand with precision mechatronic system company, maxon France. Maxon assisted Enchanted Tool's to select the perfect drive system, size the products correctly, and customize certain elements according to the current and future needs of their project. "Right from the start, we were thinking in terms of industrialization and volume", says Kévin Schwartz, sales engineer for maxon France. He continues: "There's a huge gap between current production volumes and projections for the coming years. Thanks to the maxon group's industrial volumes and our production facilities, we were able to be extremely reactive, since these products were already in our catalog, and we were able to make the necessary customizations.”<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTA0MDE3MTA5LU1heG9uIEZyYW5jZSAtIDI3IEp1aW4gMjAyMy00OTIuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib">The prototypes are tested against a variety of constraints.&nbsp;The maxon team then helped Enchanted Tools with product sizing and the choice of motorization ranges, taking into account the technical constraints and very tight deadlines required to present this new robot. EC-i 40 motors with planetary gearboxes were chosen for the Ball Bot, while ECX brushless motors with a diameter of 22mm are used for the orbital axes.To control industrialization, the maxon and Enchanted Tools project teams worked on different categories of prototypes (A, B and C). Each prototype is tested against functional, design, service life, and normative constraints, moving toward the final product and production, this approach enables the teams to anticipate the product's maturity.&nbsp;<blockquote>"The advantage of this approach is that it's robust when it comes to mass production. We go through all the design validation stages, to eliminate as many risks as possible, and then be able to calmly launch large volumes for industrialization", adds Kévin Schwartz.</blockquote>Thanks to the technological support provided by <a href="https://www.maxongroup.fr/maxon/view/content/index" target="_blank" rel="noopener noreferrer">maxon France</a>, the Enchanted Tools teams were able to move ahead quickly with industrialization. This partnership is one crucial step towards the realization of the Mirokaï project, which is set to influence the robotics landscape.&nbsp;

A year ago, French company Enchanted Tools unveiled its prototype humanoid robot, Mirokaï. The ambitious team plans to produce 100,000 robots by 2032. 



Building a humanoid robot in under a year

If you've ever been annoyed about a supermarket purchase that cost more at the checkout than the shelf price tag suggested, you've probably thought, irritated, that the retailer was trying to trick you. In reality, store price mismatches are often caused by errors that inevitably appear at large stores. Prices can change as often as several times per day, and with thousands of price tags that must be updated manually, the stores simply don’t have enough employees to update prices immediately.Adapta Robotics, a Bucharest-based robotics startup, offers an almost automated solution to this problem. They created ERIS (Effective Retail Intelligent Scanner), called MARCEL in Carrefour stores, a robot that automatically scans all the labels on a shelf, reads and compares the prices with the correct ones, and prints updated labels. At the same time, the robot also analyzes how fully the store shelves are stocked and, in the future, could even check the accuracy of the placement of the products themselves. ERIS was officially launched on November 3rd, 2022, and the company has already partnered with one of the leading European retail giants, Carrefour, to use the robot in their stores.ERIS is not the first successful product for the Bucharest-based company. Adapta Robotics has previously launched MATT, a robot that tests physical inputs for device screens in many industries, from smart washing machines to instrument panels on airplanes, on the international market.Founded by the members of a robotics team created by students from the Polytechnic University of Bucharest, with awards at dozens of professional competitions, Adapta Robotics got its start as a division of the software development company <a href="http://rinf.tech/" target="_blank">rinf.tech</a>, and later turned into an independent company. I spoke with the three founders of the company – Diana Baicu, Mihai Crăciunescu, and Cristian Dobre – about how they turned their passion for robotics into a successful business, how the robots they invented work and about the training and development process of using them.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284618626-Adapta_Founders-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">Co-founders of Adapta Robotics, from left to right: Cristian Dobre, Diana Baicu, and Mihai Crăciunescu. PHOTO: Andrei Cojan/Adapta Robotics<h3 id="isPasted">Mindcraft Stories: How did Adapta Robotics get started?</h3>Mihai Crăciunescu:&nbsp;Our team started many years ago at the Bucharest Polytechnic when we were building robots to participate in various competitions. After a while, when we won almost all the competitions we went to, national or international, we decided to take the step to create a company.Cristian Dobre:&nbsp;Robotics is a very expensive hobby, and for this reason, we had to work hard. After a few summer jobs, I ended up at <a href="http://rinf.tech/" target="_blank">rinf.tech</a>. They wanted to build a robot to test mobile devices, but they didn't have the know-how. I joined their team, and they helped us apply for a European funding project to develop such a robot, which we called MATT.<h3>M.S: So MATT was the first project you had as a robotics business?</h3>Diana Baicu:&nbsp;MATT was the first prototype we made when we were a division within <a href="http://rinf.tech/" target="_blank">rinf.tech</a>, this was around 2015. Since then, and especially in the last two years, we started to separate from them and create an identity of our own as a new robotics startup. In the meantime, we also started working on developing ERIS.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284638871-Adapta_Lab_Detail-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">Andrei Cojan/Adapta Robotics<h3 id="isPasted">M.S: Where did you get the idea of a robot to scan the labels on the shelves?</h3>M.C.:&nbsp;The shorter answer is that we talked with a retailer from Romania, who presented us with some problems they faced in the stores, and we saw it as a kind of technical challenge.Retailers can pay serious fines if the prices on the shelf are not in line with the actual ones. And the mislabeling part isn't bad faith on the part of a retailer, it's an error. To stay competitive, retailers issue daily price changes or temporary promotions on certain products. Logistically, at a store with 10,000 labels, making sure every label is updated every time is a problem. Human error will still occur somewhere.As engineers who have been developing robots for so many years, so obviously, our solution was a robot.D.B.:&nbsp;From discussions with various actors in the retail environment, we realized that most solutions in this field propose autonomous platforms. Which, of course, from a technical point of view sounds very good, but it is not necessarily the most efficient solution. This is because the problems identified by that robot must still be solved by humans.With ERIS, we decided from the beginning that it must be operated by a human. Thus, we make the pipeline more efficient, as the operator can solve the problems that arise on the spot.For example, when ERIS starts scanning after being placed in front of a shelf, there is a distance it must maintain from it. All of this information is specified by the robot, which tells the operator how close or far away to stand, as well as the speed at which it can scan the tags.We started with one prototype, of course, which we tested in a pilot project in stores. Now we've done a redesign based on everything I've learned in about a year and a half since ERIS has been running.<h3>M.S: How exactly does ERIS work?</h3>C.D.:&nbsp;ERIS is a tool. You can think of it as a mega scanner. Instead of looking at every item on a shelf and every barcode to check every price, as happens when you check manually, it scans the entire surface, and the operator just has to push it past the shelf.ERIS is equipped with sixteen cameras. Eight of these are distance-measuring cameras — that is, instead of recording color, the camera will return a black-and-white image; white means that the product is very close, and black means it is very far.After taking in the data, the robot analyzes each image captured with conventional cameras and uses a software detection model to locate the regions where the price tags are located.M.C.:&nbsp;The robot takes images from all the cameras simultaneously and builds a single large vertical image, almost two meters high, as long as the shelf. From this, it identifies price label locations.Once it has discovered the areas with price tag, they are cropped out of the big picture and sent to two neural networks: one that identifies the barcode and another that reads the price. The latter is developed with a machine learning model created by us. Once it has all of this information, it can query the current product price against the retailer's updated database.There is also a separate module that identifies shelf occupancy to alert the retailer if the shelf is empty or out of stock.The accuracy of the entire system is over 98%, in some cases even 100%, because the entire flow has been optimized to work as well as possible in the real store environment.D.B.:&nbsp;If the operator identifies a wrong label, he can print it on the spot and replace it because there are two printers on the robot. Alternatively, it also has the option to print several corrected labels at once after it has finished scanning. In this case, the robot has a location finder feature that can show the operator where the wrong label is.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284660931-ezgif-2-e22b91ac4c.png" style="width: 300px;" class="fr-fic fr-dib">ERIS1 in Store. Adapta Robotics<h3 id="isPasted">M.S: How did you train ERIS to detect labels and shelf occupancy in a store?</h3>M.C.:&nbsp;The biggest challenge we had was the datasets, that is, gathering enough relevant information that is labeled correctly so that you can train a neural network with the accuracy I mentioned earlier. In the beginning, we took many pictures of store shelves, which we had to manually annotate or use online annotation services.The volume of data required was enormous, and the quality of the results returned by online services was very poor. That is, out of 1,000 pictures, maybe 200 were properly annotated.We invested a lot of time and resources in a simulator, where we simulated the store with products, with real data from the store, through which we were able to train our networks with high accuracy.D.B.:&nbsp;We made a kind of game using Unreal Engine, a virtual store, to generate our data so that we could get a large amount of data with minimal annotation effort. At the same time, this also allowed us to more easily configure the models in case of later changes. If we change the retailer, for example, the solution works, by and large, but small adjustments still need to be made.C.D.:&nbsp;The colleague who worked on this area made a very cool algorithm that generates the shelves programmatically. Each time you started the simulator, their arrangement in the store remained the same, but the labels and shelf occupancy differed.A very big advantage is that in a game environment, you can also acquire data about the distance to objects and test that part. Such data can be obtained either if you go with the camera that measures the distance and scan the whole store or, much easier, if you load the algorithms into such a simulator. With Unreal Engine, it's very simple to calculate the distance from the camera to a product on the shelf.M.C.:&nbsp;Each retailer has labels with a different format and maybe even a different font. And we need to make sure that ERIS can recognize the price regardless of these variables. They sometimes exist even within the same store. For example, they are yellow or electronic tags, with a promotion or without a promotion - you can get up to ten different kinds. The solution must be robust to such changes.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284691878-ERIS_two_versions_MATT-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">ERIS (left) and two versions of the MATT robot. Andrei Cojan/Adapta Robotics<h3 id="isPasted">M.S: How efficient is ERIS compared to a human operator?</h3>D.B.:&nbsp;We are at the point where we have already seen that the robot is more efficient than a human operator. By our calculations, comparing how long it would take an experienced operator to scan a shelf of around 200-300 tags and how long it would take ERIS to do the same, it came out to an 85% improvement. Taking into account the label change part, the efficiency percentage is 45-50%. Our plan is to bring this percentage somewhere to 65-70%.<h3>M.S: What other plans to improve the ERIS robot do you have in the future?</h3>D.B.:&nbsp;We already have the second version, which we have already developed and are planning to introduce on the market. We optimized both the hardware side and costs, improved performance, and made it more reliable and easier to use. This will also increase the number of imaging cameras to ten.C.D:&nbsp;ERIS is already very efficient, and most data processing is done directly on the robot. To give you an idea, in one minute, around 60 GB of data are transmitted and processed on ERIS. We envision, in the future, that this data will be aggregated and transmitted to the retail and logistics systems so that it can be used there as well.D.B:&nbsp;That would mean integration with other store systems to know, for example, the historical inventory stock patterns. We have a lot of plans on that side, including an analytics platform based on all the information that ERIS gets to actually help the retailer understand everything that's going on in the store.M.C.:&nbsp;Another functionality we have on the radar is a planogram, through which you can check if a product is positioned where it should be on the shelf. For example, product manufacturers pay a premium to have their products on the top shelf, third shelf, etc., usually to be more visible, and the retailer has to ensure that those products are there. That's where you get into a detection area of the product itself, which is more complicated than the tag, but that's a feature we're discussing for the future.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284710441-MATT_testing_tablet_touchscreen-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">MATT testing the tablet touchscreen. Adapta Robotics<h3 id="isPasted">M.S: As for MATT, how did you come up with the idea for a robot that tests inputs for various devices?</h3>M.C.:&nbsp;MATT was designed as a robot to automate testing on phones, around 2015. Now you can automate pretty much everything that happens on a phone with emulation software, so MATT is not that relevant for smartphones anymore.It's just that other industries have emerged that require testing for touchscreens, such as automotive. There are other situations where you need to integrate a phone with a device and make sure that the Bluetooth connection works or that if you press a button on the phone, something happens on the other device, like in the case of appliances that you control through the phone.I saw that MATT is perfect for this, so I put it on input testing, and it became kind of a benchmark in this industry. Most of our customers for MATT are in the United States but also in the Netherlands, Germany, France, Spain, and even Lithuania.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284743662-Adapta_Team_Photo-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">Andrei Cojan/Adapta Robotics<h3 id="isPasted">M.S: What expansion plans do you have for the future?</h3>M.C:&nbsp;We are actively scaling the use of ERIS, primarily in Romania, because we are a Romanian startup. But our targets are abroad because that's where the impact will be much more relevant. In Romania, labor in stores is also cheaper. And as there is always the question of return on investment for the retailer, in countries like Sweden, Norway, Great Britain, Germany, or France, the investment will be much more attractive for the retailer.The vast majority of our competitors are well-known or are credibly well-funded companies. We have to focus on quality, that is, that robot does what we promise it does.Many of the initiatives in this field fail because they have a lot of nice words in the presentation, but when they get to launch a robot, it does not perform the basic functionalities or performs them very poorly. To get a "piece of the pie", we focus on quality: the AI and the robot must work at very good percentages so that we can deliver a reliable solution that actually improves store operations.<hr>This Q&amp;A interview session with Adapta Robotics happened at&nbsp;<a href="http://gotech.world/" target="_blank">gotech.world</a> 2022 Expo, where the company introduced the second iteration of ERIS, the retail robot.

ERIS, a robot specialized in scanning shelf labels created by Adapta Robotics, will soon be seen in supermarkets all across Romania

The Robot That Solves the Wrong Price Label Problem on Store Shelves

The recent advances in Large Language Models (LLM) have made a lot of noise recently – everybody is using ChatGPT. Robotics is no exception; there has been a lot of talk about how to integrate them in various kind of robotic use cases; in this article, we will talk about their applications to social robots like NAO and Pepper.&nbsp;<h2>Social Robots? &nbsp;</h2>Social robots are designed to be used in a places, interacting with people, and as such are a natural fit for conversation-oriented language models. But they have been used for years before that, and the field of human-robot interaction is well-developed.&nbsp;Compared to a GUI chat interface like ChatGPT, social robots have to deal with extra uncertainty linked to speech recognition (the input the langauge model received might not be what the user intended to say), and handling several users speaking, some of which might not even intend to speak to the robot.&nbsp;<h2>Improving what the robot understands&nbsp;</h2>One advantage LLMs have is in understanding open-ended instructions. There are several challenges in understanding voice commands:&nbsp;<ol><li>Speech recognition: turning the raw sounds into text; how effective this is is going to depend of how the user speaks (accents, slurred speech, volume, clarity of enunciation), the characteristics of the microphone, surrounding sound, etc.&nbsp;</li><li>Variety of expressions: In practice, people have a huge variety of ways of expressing even simple things (For example: "I'm looking for an ATM", "Would you show me the nearest ATM?", "Is there an ATM around here?", "Where's the nearest ATM?","Do you know where I can withdraw some money?", "Where can I withdraw cash around here?", "Is there an ATM in this building?" etc.); and handling all of those properly usually requires some bespoke customisation via a dialogue engine like dialogflow or qichat.&nbsp;</li><li>Context: In addition to the explicit information contains in a question like the ATM example above, some information comes from what was previously discussed ("How much does one cost?" "How can I get there?"), or from gestures such as pointing ("Is it that way?" "What's that?"); keeping track of that information usually requires more bespoke logic related to keeping track of users.&nbsp;</li></ol>These are all challenges that could be handled previously with some work; but LLMs introduced a great increase in quality: they can be very good at handling a wide variety of expressions out of the box. They are also very good at taking into account any context included in the conversation log (typically sent alongside the prompt); this usually includes previous conversation but (with a bit more work) can include any information that can be infered from the sensors (for example, by using an object recognition model to turn what the robot sees into a text description).&nbsp;One more surprising improvement when using LLMs is on speech recognitions – more specifically, even if there are mistakes in the speech recognition (that can be misheard words, but also homophones like their/there/they're), the LLM will often use context and common sense to handle the input appropriately anyway, giving the impression of a robot that hears better.&nbsp;<h2>Improving how the robot answers&nbsp;</h2>One obvious output is Speech, and plugged into a speech-to-text engine, a LLM can easily create a conversational robot.&nbsp;However, this brings a few new issues:&nbsp;<ul><li>One notorious ones is hallucinations, a problem that's far from solved for LLMs. The robot can understand a wider variety of inputs, his answers to those might be confident but highly inaccurate – if there's a word he doesn't understand, he might decide to invent a fanciful product description; if an unfamiliar name is given, he might make someone up. This can be mitigated, but at the risk of making the robot repetitive or boring – it's a fine balance, and it might depend on the use case.&nbsp;</li><li>A less obvious one is speech that's too verbose. If ChatGPT talks too much, the user can just skim the paragraphs; but a speaking robot takes time and can be much more annoying. This can be improved with better prompting, with pretraining etc. as well as UX tweaks (e.g. a way of interrupting the robot) but requires careful attention. More generally, the LLM's output might sometimes not be as good quality as hand-crafted one.&nbsp;</li></ul>However, a robot is more than a smart speaker – it might have arms, wheels, etc. and the LLM can be made to control those, either directly or from a library of presets (e.g. navigation waypoints) - in effect, the output of the LLM is not just speech but actions.&nbsp;<h2>How to deal with it the best&nbsp;</h2>How to deal with hallucinations and other problems may depend of the use case:&nbsp;<ul><li>For entertainment purposes, the hallucinations may not be a problem, and can even be part of the fun, as long as that is well-presented to the user.&nbsp;</li><li>For information purposes, a choice needs to be made between using LLM output and risking hallucinations, or using fixed, well-prepared outputs.&nbsp;</li><li>One should also keep in mind other non-speech actions and see how many of those can be controlled by the LLM.&nbsp;</li></ul>Apart from the technical performance on input and output, the largest strengths of LLMs is flexibility – adapting the robot to a new use case, or to a specific context for a new use case, can be as simple as editing a prompt, and produces a behavior of decent quality.&nbsp;This allows for much quicker iterations. &nbsp;<h2>Further directions&nbsp;</h2>In robotics (social or not) and elsewhere, engineers, researchers and tinkerers all over the world are experimenting with interesting ways to use LLMs. We've covered a few we have experience with here, but there are many more directions that could be applicable, such as:&nbsp;<ul><li>integrating vision models to plug it with what the robot sees, to make it even more aware of the context&nbsp;</li><li>making the robot proactively decide what to do (not just respond to what he heard, but decide to e.g. walk up to such-and-such a person and engage conversation)&nbsp;</li><li>giving the robot a long-time memory&nbsp;</li></ul>LLMs offer great possibilities for social robots if used well, and it will be interesting to observe how the field evolves in the coming years – in the coming months even.&nbsp;

A Pepper robot showing a ChatGPt-type dialogue

Lessons learnt from integrating ChatGPT in Pepper and NAO

Bringing the LLM revolution to social robots

In this episode, we discuss a novel new approach for increasing the field of view of autonomous systems by leveraging reflections of blindspots on reflective bodies.

Podcast: Blind Spot Begone! Cars and Robots Can Now See Around Corners

49 Pound Social-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Soft Robotics Podcast-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Soft Robotics Podcast-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Soft Robotics Podcast-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Soft Robotics Podcast-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Soft Robotics Podcast-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NEURA Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

NEURA Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NEURA Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NEURA Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

NEURA Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Rozum Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Rozum Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Rozum Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Rozum Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Rozum Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

READY Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

READY Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

READY Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

READY Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

READY Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Adapta Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Adapta Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Adapta Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Adapta Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Adapta Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Inovo Robotics Ltd-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Inovo Robotics Ltd-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Inovo Robotics Ltd-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Inovo Robotics Ltd-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Inovo Robotics Ltd-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

United Robotics Group-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

United Robotics Group-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

United Robotics Group-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

United Robotics Group-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

United Robotics Group-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NODE Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

NODE Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NODE Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

NODE Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

NODE Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Remix Robotics-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Remix Robotics-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Remix Robotics-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Remix Robotics-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Remix Robotics-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

social robotics

social robotics

Latest Posts

Using robotics to supercharge health care

Intra-hug gestures prove key to making HuggieBot 3.0 the most cuddly robot around

Podcast: Climate Change AI, Robots Getting You Dressed, Drones Evacuating Nursing Homes

Podcast: Hugging Robots, Scalable Solar Fuel, Non-Invasive Spinal Implants

Podcast: Tissue Printing During Surgery, Hair Brushing Robots, Smaller & Cheaper Batteries

Mixed Robotic Interface (MRI Gamma)

Stanford's JackRabbot 2: The polite pedestrian robot

Robot Uses Deep Learning and Big Data to Write and Play Its Own Music

Robots and Religion: Mediating the Divine.

Profiles