Machine-learning models can fail when they try to make predictions for individuals who were underrepresented in the datasets they were trained on.For instance, a model that predicts the best treatment option for someone with a chronic disease may be trained using a dataset that contains mostly male patients. That model might make incorrect predictions for female patients when deployed in a hospital.To improve outcomes, engineers can try balancing the training dataset by removing data points until all subgroups are represented equally. While dataset balancing is promising, it often requires removing large amount of data, hurting the model’s overall performance.MIT researchers developed a new technique that identifies and removes specific points in a training dataset that contribute most to a model’s failures on minority subgroups. By removing far fewer datapoints than other approaches, this technique maintains the overall accuracy of the model while improving its performance regarding underrepresented groups.In addition, the technique can identify hidden sources of bias in a training dataset that lacks labels. Unlabeled data are far more prevalent than labeled data for many applications.This method could also be combined with other approaches to improve the fairness of machine-learning models deployed in high-stakes situations. For example, it might someday help ensure underrepresented patients aren’t misdiagnosed due to a biased AI model.“Many other algorithms that try to address this issue assume each datapoint matters as much as every other datapoint. In this paper, we are showing that assumption is not true. There are specific points in our dataset that are contributing to this bias, and we can find those data points, remove them, and get better performance,” says Kimia Hamidieh, an electrical engineering and computer science (EECS) graduate student at MIT and co-lead author of a&nbsp;<a href="https://arxiv.org/pdf/2406.16846" target="_blank">paper on this technique</a>.She wrote the paper with co-lead authors Saachi Jain PhD ’24 and fellow EECS graduate student Kristian Georgiev; Andrew Ilyas MEng ’18, PhD ’23, a Stein Fellow at Stanford University; and senior authors Marzyeh Ghassemi, an associate professor in EECS and a member of the Institute of Medical Engineering Sciences and the Laboratory for Information and Decision Systems, and Aleksander Madry, the Cadence Design Systems Professor at MIT. The research will be presented at the Conference on Neural Information Processing Systems.<h2>Removing bad examples</h2>Often, machine-learning models are trained using huge datasets gathered from many sources across the internet. These datasets are far too large to be carefully curated by hand, so they may contain bad examples that hurt model performance.Scientists also know that some data points impact a model’s performance on certain downstream tasks more than others.The MIT researchers combined these two ideas into an approach that identifies and removes these problematic datapoints. They seek to solve a problem known as worst-group error, which occurs when a model underperforms on minority subgroups in a training dataset.The researchers’ new technique is driven by prior work in which they introduced a method, called&nbsp;<a href="https://arxiv.org/pdf/2303.14186">TRAK</a>, that identifies the most important training examples for a specific model output.For this new technique, they take incorrect predictions the model made about minority subgroups and use TRAK to identify which training examples contributed the most to that incorrect prediction.“By aggregating this information across bad test predictions in the right way, we are able to find the specific parts of the training that are driving worst-group accuracy down overall,” Ilyas explains.Then they remove those specific samples and retrain the model on the remaining data.Since having more data usually yields better overall performance, removing just the samples that drive worst-group failures maintains the model’s overall accuracy while boosting its performance on minority subgroups.<h2>A more accessible approach</h2>Across three machine-learning datasets, their method outperformed multiple techniques. In one instance, it boosted worst-group accuracy while removing about 20,000 fewer training samples than a conventional data balancing method. Their technique also achieved higher accuracy than methods that require making changes to the inner workings of a model.Because the MIT method involves changing a dataset instead, it would be easier for a practitioner to use and can be applied to many types of models.It can also be utilized when bias is unknown because subgroups in a training dataset are not labeled. By identifying datapoints that contribute most to a feature the model is learning, they can understand the variables it is using to make a prediction.“This is a tool anyone can use when they are training a machine-learning model. They can look at those datapoints and see whether they are aligned with the capability they are trying to teach the model,” says Hamidieh.Using the technique to detect unknown subgroup bias would require intuition about which groups to look for, so the researchers hope to validate it and explore it more fully through future human studies.They also want to improve the performance and reliability of their technique and ensure the method is accessible and easy-to-use for practitioners who could someday deploy it in real-world environments.“When you have tools that let you critically look at the data and figure out which datapoints are going to lead to bias or other undesirable behavior, it gives you a first step toward building models that are going to be more fair and more reliable,” Ilyas says.

MIT researchers developed an AI debiasing technique that improves the fairness of a machine-learning model by boosting its performance for subgroups that are underrepresented in its training data, while maintaining its overall accuracy. Credit: José-Luis Olivares, iStock

A new technique identifies and removes the training examples that contribute most to a machine-learning model’s failures.

Researchers reduce bias in AI models while preserving or improving accuracy

If someone advises you to “know your limits,” they’re likely suggesting you do things like exercise in moderation. To a robot, though, the motto represents learning constraints, or limitations of a specific task within the machine’s environment, to do chores safely and correctly.For instance, imagine asking a robot to clean your kitchen when it doesn’t understand the physics of its surroundings. How can the machine generate a practical multistep plan to ensure the room is spotless? Large language models (LLMs) can get them close, but if the model is only trained on text, it’s likely to miss out on key specifics about the robot’s physical constraints, like how far it can reach or whether there are nearby obstacles to avoid. Stick to LLMs alone, and you’re likely to end up cleaning pasta stains out of your floorboards.To guide robots in executing these open-ended tasks, researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) used vision models to see what’s near the machine and model its constraints. The team’s strategy involves an LLM sketching up a plan that’s checked in a simulator to ensure it’s safe and realistic. If that sequence of actions is infeasible, the language model will generate a new plan, until it arrives at one that the robot can execute.This trial-and-error method, which the researchers call “Planning for Robots via Code for Continuous Constraint Satisfaction” (PRoC3S), tests long-horizon plans to ensure they satisfy all constraints, and enables a robot to perform such diverse tasks as writing individual letters, drawing a star, and sorting and placing blocks in different positions. In the future, PRoC3S could help robots complete more intricate chores in dynamic environments like houses, where they may be prompted to do a general chore composed of many steps (like “make me breakfast”).“LLMs and classical robotics systems like task and motion planners can’t execute these kinds of tasks on their own, but together, their synergy makes open-ended problem-solving possible,” says PhD student Nishanth Kumar SM ’24, co-lead author of a new paper about PRoC3S. “We’re creating a simulation on-the-fly of what’s around the robot and trying out many possible action plans. Vision models help us create a very realistic digital world that enables the robot to reason about feasible actions for each step of a long-horizon plan.”The team’s work was presented this past month in a paper shown at the Conference on Robot Learning (CoRL) in Munich, Germany.<iframe width="640" height="360" src="https://www.youtube.com/embed/hBq0EeFSsPo?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Teaching a robot its limits for open-ended chores. MIT CSAILThe researchers’ method uses an LLM pre-trained on text from across the internet. Before asking PRoC3S to do a task, the team provided their language model with a sample task (like drawing a square) that’s related to the target one (drawing a star). The sample task includes a description of the activity, a long-horizon plan, and relevant details about the robot’s environment.But how did these plans fare in practice? In simulations, PRoC3S successfully drew stars and letters eight out of 10 times each. It also could stack digital blocks in pyramids and lines, and place items with accuracy, like fruits on a plate. Across each of these digital demos, the CSAIL method completed the requested task more consistently than comparable approaches like <a href="https://arxiv.org/pdf/2403.11552">“LLM3”</a> and&nbsp;<a href="https://arxiv.org/pdf/2209.07753">“Code as Policies”</a>. The CSAIL engineers next brought their approach to the real world. Their method developed and executed plans on a robotic arm, teaching it to put blocks in straight lines. PRoC3S also enabled the machine to place blue and red blocks into matching bowls and move all objects near the center of a table.Kumar and co-lead author Aidan Curtis SM ’23, who’s also a PhD student working in CSAIL, say these findings indicate how an LLM can develop safer plans that humans can trust to work in practice. The researchers envision a home robot that can be given a more general request (like “bring me some chips”) and reliably figure out the specific steps needed to execute it. PRoC3S could help a robot test out plans in an identical digital environment to find a working course of action — and more importantly, bring you a tasty snack.For future work, the researchers aim to improve results using a more advanced physics simulator and to expand to more elaborate longer-horizon tasks via more scalable data-search techniques. Moreover, they plan to apply PRoC3S to mobile robots such as a quadruped for tasks that include walking and scanning surroundings.“Using foundation models like ChatGPT to control robot actions can lead to unsafe or incorrect behaviors due to hallucinations,” says The AI Institute researcher Eric Rosen, who isn’t involved in the research. “PRoC3S tackles this issue by leveraging foundation models for high-level task guidance, while employing AI techniques that explicitly reason about the world to ensure verifiably safe and correct actions. This combination of planning-based and data-driven approaches may be key to developing robots capable of understanding and reliably performing a broader range of tasks than currently possible.”Kumar and Curtis’ co-authors are also CSAIL affiliates: MIT undergraduate researcher Jing Cao and MIT Department of Electrical Engineering and Computer Science professors Leslie Pack Kaelbling and Tomás Lozano-Pérez. Their work was supported, in part, by the National Science Foundation, the Air Force Office of Scientific Research, the Office of Naval Research, the Army Research Office, MIT Quest for Intelligence, and The AI Institute.

PhD students Aidan Curtis (left) and Nishanth Kumar. To help robots execute open-ended tasks safely, the researchers used vision models to see what’s near the machine and model its constraints. Their “PRoC3S” strategy has an LLM sketch up an action plan that’s checked in a simulator to ensure it will work in the real world. Credits: Mike Grimmett/MIT CSAIL

The “PRoC3S” method helps an LLM create a viable action plan by testing each step in a simulation. This strategy could eventually aid in-home robots to complete more ambiguous chore requests.

Teaching a robot its limits, to complete open-ended tasks safely

Professional piano players spend countless hours at the keys, perfecting their craft. For people with smaller hands, this dedication can take a physical toll. Repeated stretching to reach distant keys in a chord can strain muscles and joints and may lead to tendonitis, carpal tunnel syndrome, and other injuries.Researchers at Stanford are working to understand the forces involved in the hand movements of elite-level piano players, and how those change with different hand sizes and keyboard widths. On Dec. 5 at SIGGRAPH Asia 2024, Stanford Engineering researchers presented an AI-trained model that can recreate the hand movements required to play complicated pieces of music and the physical stresses those hands are under. This is the first step in an effort to reduce the risk of long-term injuries in piano players and test solutions – such as narrower keyboards – that could make piano playing more equitable for everyone.“We would never expect a world-class athlete to compete with equipment that does not fit their body. Yet we ask pianists, particularly women, to adapt to a one-size-fits-all design that was never built with them in mind,” said<a href="https://profiles.stanford.edu/elizabeth-schumann" data-extlink="">&nbsp;Elizabeth Schumann</a>, the Billie Bennet Achilles Director of Keyboard Studies at the&nbsp;<a href="https://humsci.stanford.edu/" data-extlink="">School of Humanities and Sciences</a> and co-author on the paper. “We can use research like this to reshape the future of performance and make it more sustainable.”&nbsp;<h2>Capturing elite performance</h2>The modern piano keyboard was designed in the 19th century with the average European man in mind. Today, an estimated 87% of adult women and 24% of adult men have hands that are smaller than ideal for the standard piano. Using traditional methods to study the impacts of smaller hands would require following a cohort of piano players for decades, and it could still be difficult to quantify injury risks or test solutions. Instead, the researchers have recorded the hand movements of professional piano players and are using artificial intelligence to predict how different sizes of hands would move to play new music.Schumann and her colleagues recruited 15 elite-level pianists to play a total of 10 hours of music while cameras filmed their hands from every angle. The researchers couldn’t put any sensors on the players without potentially interfering with their performance, so they used advanced computer vision techniques to combine the videos and reconstruct the players’ hand motions in three dimensions. They used additional processing to make sure that those movements were perfectly synced with the audio.“The quality of data that we were able to achieve is unprecedented,” said&nbsp;<a href="https://profiles.stanford.edu/c-karen-liu" data-extlink="">Karen Liu</a>, a professor of computer science at Stanford and lead author on the paper. “It’s not just that the reconstructed 3D motions are accurate and clean; the quality of the performance itself is truly remarkable. We were able to capture diverse movements from a group of talented musicians performing at a high level – I have not seen a dataset like that in the past.”Roucheng Wang, a graduate student in Liu’s lab, and Pei Xu, a postdoctoral researcher, used this dataset to train a model to accurately generate new piano-playing data. When they gave the model sheet music it had never seen, such as Beethoven’s “Für Elise,” the model could provide the correct 3D hand movements needed to play the piece, using a simulated piano to generate sound.“I was just so stunned at how accurately this model could simulate elite-level technique,” Schumann said. “It’s really incredible.”<iframe width="640" height="360" src="https://www.youtube.com/embed/sGQNCEl7pJQ?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>When researchers gave the AI-trained piano playing model sheet music it had never seen, such as Beethoven’s “Für Elise,” the model could provide the correct 3D hand movements needed to play the piece. | Ruocheng Wang<h2 id="isPasted">From robot joints to real muscles</h2>This model is just the first step for Liu, Schumann, and their colleagues. At the moment, the model can physically simulate hand movements, but it doesn’t simulate the muscles and tendons that create those movements or the strain that they are subjected to.“We are not at the biomechanics level yet, where the actuation of the hands is generated by muscles – currently our model is more like a robot that directly generates torque at joints,” Liu said. “Eventually, our model needs to be able to predict the tension level of muscles and the potential of injury.”The researchers are currently working to add these layers to their model so that they can use it to help piano players – especially those with small hands – avoid injuries. They are also adapting their work to model the movements of other dexterous musicians. They have applied some of the same techniques to an existing dataset of hand motions of highly skilled guitar players. Their results, which were also presented recently at SIGGRAPH Asia 2024, showed that they could successfully model the distinct left- and right-hand movements needed to play a variety of both popular and classical music.“With high-quality data, we could model the 3D movements needed for other types of music performance as well,” Liu said. “This isn’t a model that is replacing people – we’re working with musicians to help understand and solve problems. This project is about advancing people, and AI is just a tool for that.”

Stanford researchers collected the first large-scale 3D hand motion dataset containing 10 hours and 153 pieces of piano music performed by 15 elite-level pianists, along with synchronized audio and key pressing events. | The Movement Lab

Researchers at Stanford Engineering have developed an AI-trained model to accurately recreate the hand movements of elite-level pianists and the physical stresses they endure while playing.

AI could help reduce injury risk in pianists

According to <a href="https://www.statista.com/outlook/tmo/artificial-intelligence/ai-robotics/worldwide" target="_blank" id="isPasted">Statista</a>, the global AI robotics market will reach $17 billion in 2024 and will keep growing by 24% annually. Today, robots deliver packages, conduct geological surveys, and entertain visitors of parks and exhibitions – however, that’s not the limit of their abilities. In this article, we get into the technologies at the intersection of AI and robotics, as well as the research done in this field at ITMO, with Prof. Ivan Borisov, a researcher at ITMO’s International Laboratory of Biomechatronics and Energy-Efficient Robotics (BE2R Lab).<h1 dir="ltr" id="isPasted">Why mix AI and robotics</h1>Robotics is an interdisciplinary field that brings together mechanics, electronics, and control systems. In order to plan a robot’s route and manage its movements, experts need to have a good command of mathematics and algorithms. Without correctly functioning optimization algorithms, a robot won’t be able to plan its movement trajectory and steps, especially if there are obstacles in its way.A control algorithm is based on an accurate mathematical model of a robot, which describes its state, geometry, and dynamics. Using this model, the algorithm calculates the robot’s movements and interactions with the environment. In other words, the more precise the mathematical model is, the more efficient the actual robot will be.To build an accurate mathematical model, it is essential to consider key geometric parameters (such as mass, inertia, friction, drive power, torque, as well as the performance metrics of drives and sensors). For a robot isolated from its environment, creating an accurate model is straightforward. However, the task becomes fundamentally more complex when the robot is required to physically interact with objects in its environment, especially when this environment is unknown.<blockquote>“We can, for example, build an accurate model of a manipulator robot welding a car. It has a fixed base and predetermined environment – we know what will happen to it. That’s why we can make accurate models of the robot itself and its surroundings, calculate the welding machine’s trajectory, and make the robot simply follow a precalculated trajectory. However, if we need a mobile robot, one that will walk or move on wheels in a city, or a kitchen manipulator robot, the conventional paradigm would require a model of its surroundings. The case is, though, that we cannot create an accurate model for an undetermined, a priori unknown, and constantly changing environment,” says Prof.&nbsp;Ivan Borisov, a researcher at ITMO’s International Laboratory of Biomechatronics and Energy-Efficient Robotics (BE2R Lab).</blockquote><iframe width="640" height="360" src="https://www.youtube.com/embed/TbyHKQKYVHg?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>AI technologies can be used to address the challenges of designing robots as systems. For this purpose, researchers turn to gradient and global optimization algorithms, as well as neural network approaches, including training on large datasets and reinforcement learning. Thanks to these methods, it’s possible to automate the design process (by delegating the filtering of solutions to algorithms), solve the task of motion policy synthesis, and enable robots to perceive their environment.<h1 dir="ltr">How AI is facilitating robot design</h1>One robotics task that AI is already helping to solve is the design of entire robots or their individual parts, such as mechanisms or drives.Previously, the quality of a design depended solely on the skills of its creator: if the same task was given to several engineers, they would all solve it differently. As a result, there would be various designs, each with hardly guaranteed optimality.These days, this problem is being addressed through the use of numerical and generative design. The main goal is to automate the design process and help specialists find optimal robot designs more quickly by exploring all possible solutions. Numerical design can be based on both optimization algorithms and neural network approaches, such as reinforcement learning or supervised learning. It helps identify the optimal parameters to enhance a robot’s performance, while generative design helps create mechanical structures and optimize them.Reinforcement learning was proven effective in the industry. For example, it was used by Disney Research in collaboration with ETH Zurich to <a href="https://la.disneyresearch.com/publication/design-and-control-of-a-bipedal-robotic-character/" target="_blank">merge</a> the animation of a droid from Star Wars and a physical model of a bipedal robot.&nbsp;Researchers from the BE2R Lab have <a href="https://news.itmo.ru/en/science/cyberphysics/news/13989/" target="_blank">created</a> the open-source repository <a href="https://github.com/aimclub/rostok?tab=readme-ov-file" target="_blank">Rostok</a>; with it, it’s possible to use generative design to create single-drive adaptive grabbing tools. Taking into account the parameters given by an engineer and the robot’s design limitations, the technology balances between them, presenting a greater number of various solutions. This makes the design process quicker (compared to human performance) and mathematically justified. Each solution is tested in a simulator and graded on performance. The higher the grade, the more likely the algorithm is to recommend a particular solution to engineers.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1733879540331-1399600.jpg" style="width: 100%px;" class="fr-fic fr-dib">Grabbing robot design. Photo courtesy of ITMO scientistsThe researchers used a similar approach to improve their previous <a href="https://news.itmo.ru/en/news/13232/" target="_blank">design</a> – a mobile jumping robot. Earlier, the robot relied on bearings, making its structure more rigid and less mobile; during a fall, part of the impact energy damaged its solid links. Using global optimization genetic algorithms, the scientists <a href="https://news.itmo.ru/en/science/cyberphysics/news/13925/" target="_blank">found out</a> that it would be better to use soft links instead of bearings. By absorbing all the energy from a fall, such links protect the robot’s structure and allow for partial recouping of energy from the impact. Thanks to this change, quadrupeds, anthropomorphic robots, and collaborative manipulators can be more energy-efficient, withstand shock loads, and adapt to uneven surfaces.&nbsp;Both solutions were revealed at the International Conference on Intelligent Robots and Systems (IROS 2024), a major conference that <a href="https://portal.core.edu.au/conf-ranks/?search=IROS&by=all&source=all&sort=dsource&page=1" target="_blank">holds the A rank</a> (one of the highest) in the CORE conference ranking.. In 2024, the event was held in Abu Dhabi on October 14-18.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1733879566211-1399601.jpg" style="width: 100%px;" class="fr-fic fr-dib">Jumping robot prototype. Photo courtesy of ITMO scientists<h1 dir="ltr" id="isPasted">AI for robotic vision</h1>Computer vision is a key element in robotic design, as it helps robots build a 3D map of their surroundings and plan routes around obstacles. However, without AI solutions to these tasks are too expensive, time-consuming, and inefficient.In the 1980s, Prof. Hans Moravec from Carnegie Mellon University explained why vision became one of the hardest “capacities” to acquire for robots. It’s summarized in a paradox subsequently named after him: contrary to popular assumption, higher cognitive processes (such as playing chess or go) require relatively little computational resources and are thus available to robots; at the same time lower-level sensorimotor operations (vision, perception, and body coordination) require immense computational resources.&nbsp;With AI, it’s possible to overcome Moravec’s paradox. For that, engineers first assemble a dataset (or take an existing one) wherein images are clearly marked by features and categories: for example, some are cats, and some are humans. To map out its surroundings and plan routes, robots use LiDARs, cameras, and other sensory technologies, while trained networks help them “make sense” of the surrounding objects. If a robot encounters something unknown along the way, it might try to classify this object using similar images from its dataset.AI can also help robots navigate complex environments with mirrors or glass doors. Their surfaces reflect or permit light through, which means that robots can misinterpret obstacles or distances to them.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1733879591092-1399604.jpg" style="width: 100%px;" class="fr-fic fr-dib">Sergey Kolyubin with the team of ITMO's Faculty of Control Systems and Robotics. Photo by ITMO.NEWSHeaded by Sergey Kolyubin, researchers from ITMO’s BE2R Lab have&nbsp;<a href="https://ieeexplore.ieee.org/abstract/document/10342329" target="_blank" id="isPasted">developed</a> a 3D mapping system based on deep neural networks and the camera’s ego-motion (its movement relative to surrounding objects). Having tested the system in a space with glass partitions, the researchers concluded that it calculates distance to objects and creates 3D maps of spaces more accurately than other neural network-based solutions with RGB-D cameras that calculate distances to objects and their colors.

According to , the global AI robotics market will reach $17 billion in 2024 and will keep growing by 24% annually. Today, robots deliver packages, conduct geological surveys, and entertain visitors of parks and exhibitions – however, that’s not the limit of their abilities.

Higher, Faster, Stronger: How AI Makes Robots Smarter and More Efficient

<a href="https://scienceexchange.caltech.edu/topics/neuroscience/neuroscience-experts/brain-machine-interfaces-richard-andersen">Brain–machine interfaces</a> (BMIs) have enabled a handful of test participants who are unable to move or speak to communicate simply by thinking. An implanted device picks up the neural signals associated with a particular thought and converts them into control signals that are fed into a computer or a robotic limb. For example, a quadriplegic person is asked to think about moving a cursor on a computer screen. Once the BMI is trained to recognize the neurological activity as this intent, the person's thought is transmitted via the BMI to carry out the action of moving the cursor. BMIs currently in an experimental phase may also consist of robotic limbs that can execute manual tasks as instructed by a disabled person's thoughts alone.The hardware necessary for this incredible feat is a computer—either freestanding or within a robotic device—and an implant in the brain of the person who is using BMI technology to communicate their intent through their thoughts. At Caltech, researchers use implants that consist of arrays of 100 microelectrodes mounted on a 4x4 mm chip. The microelectrodes are typically 1.5 mm long and penetrate into the brain's cortex, where they can record the activity of individual neurons.Unfortunately, the performance of these microelectrode arrays is not consistent and degrades over time. To overcome this challenge, Caltech's&nbsp;<a href="https://www.ee.caltech.edu/people/azita">Azita Emami</a>, the Andrew and Peggy Cherng Professor of Electrical Engineering and Medical Engineering and director of the Center for Sensing to Intelligence (S2I), and her colleagues have used machine learning to effectively interpret the neuronal signals picked up by older implants."Not only do we observe day-to-day variations, but over time the performance of brain–computer interfaces degrade for a variety of reasons," Emami says. "There may be a small movement of the implant or its electrodes. The electrodes themselves may deteriorate or become encapsulated in brain tissue. Some people think that over time the neurons move away from the implant because they react to it as a foreign object in the brain. For whatever reason, the signals we receive become noisier."When a BMI is first set up, the microelectrode array produces a signal characterized by strong action potentials that appear like spikes in the recordings. Once this strong signal is no longer detected by the microelectrode array—that is, when the feedback from the array becomes noisier and neural spikes are no longer clearly detected—it is a far trickier task to link a pattern of neural activity from more distant neurons to a specific intent that can be successfully transmitted to a computer or other device. Researchers have tried to identify alternative signals, such as so-called threshold crossings or local field potentials recorded from distant neurons. One approach has been to use wavelets that measure small oscillations in neuronal activity. But the success of wavelets and other methods has been limited.Now Emami and her colleagues have found that by applying machine learning, BMIs can be trained to interpret data from neural activity even after the signal from an implant has become less clear.Benyamin Haghi, formerly a graduate student in the Emami lab explains: "Where before we relied on counting neural spikes, we have now created a neural network that automatically extracts information from the entire neural signal, from all the little dips and picks and changes in the signal, and converts this into the intent of the patient." Emami adds, "Over time the BMI has been trained on both a signal that is neural activity and a signal that looks like noise, and is therefore able to interpret the user's intent."Emami describes the experience of one participant, JJ, who lost mobility due to a vehicle accident. "When we first started working with him, his implant was three years old, and it had already degraded. We were thinking of removing the implant, but with our new algorithm, he's happy with continuing to use the system he already has. JJ can move a cursor very precisely on a grid, just as he did when the implant was new. He can play video games and control a computer environment that mimics driving."The team's algorithm is called FENet, for Feature Extraction Network. Remarkably, it can be trained on data from one patient and then used successfully in another. "This means that there is some fundamental type of information in the neural data that we are picking up," Emami says. Not only that, FENet can generalize across different brain regions and types of electrodes and be easily incorporated into existing BMIs.Richard Andersen, the James G. Boswell Professor of Neuroscience and leadership chair and director of the T&amp;C Chen Brain-Machine Interface Center, says "FENet has already extended our clinical study with JJ by two years. BMI research is a perfect field for interdisciplinary research, in this case melding the disciplines of engineering, computer science, and neuroscience."Today's BMIs require a manual connection from the electrode implant to a connector that then links with wires that lead to a microsystem that processes the raw data before sending it to a computer—the visual interface the patient manipulates. "This is quite a cumbersome system," Emami says. "Our goal now, after the creation of FENet to better interpret brain signals, is to miniaturize the system so that one day it can be a wearable or an implant that communicates wirelessly with the computer."<hr>This research was published in Nature Biomedical Engineering under the title "<a href="https://rdcu.be/d2D2L">Enhanced control of a brain-machine interface by tetraplegic participants via neural-network-mediated feature extraction</a>." Co-authors include Andersen; Emami; Haghi; Albert Yan Huang from the Emami lab; members of professional staff Tyson Aflalo and Spencer Kellis, postdoc Jorge A. Gamez de Leon, and graduate student Charles Guan from the Andersen lab; and Nader Pouratian from UCLA Neurosurgery. Funding was provided by the National Institutes of Health, Caltech's S2I, the T&amp;C Chen Brain-Machine Interface Center at Caltech, the Boswell Foundation, the Braun Foundation, and the Heritage Medical Research Institute.

Brain–machine interfaces (BMIs) have enabled a handful of test participants who are unable to move or speak to communicate simply by thinking.

Improving Brain-Machine Interfaces with Machine Learning

In 2018, Google DeepMind's AlphaZero program taught itself the games of chess, shogi, and Go using machine learning and a special algorithm to determine the best moves to win a game within a defined grid. Now, a team of Caltech researchers has developed an analogous algorithm for autonomous robots—a planning and decision-making control system that helps freely moving robots determine the best movements to make as they navigate the real world."Our algorithm actually strategizes and then explores all the possible and important motions and chooses the best one through dynamic simulation, like playing many simulated games involving moving robots," says&nbsp;<a href="https://www.eas.caltech.edu/people/sjchung">Soon-Jo Chung</a>, Caltech's Bren Professor of Control and Dynamical Systems and a senior research scientist at JPL, which Caltech manages for NASA. "The breakthrough innovation here is that we have derived a very efficient way of finding that optimal safe motion that typical optimization-based methods would never find."The team describes the technique, which they call Spectral Expansion Tree Search (SETS), in the December&nbsp;<a href="https://www.science.org/doi/10.1126/scirobotics.ado1010">cover article</a> of the journal&nbsp;Science Robotics.Many robots can move quite freely and in any direction. Consider, for example, a humanoid robot designed to assist an elderly person in a home. Such a robot should be able to move in many different ways and, essentially, in any direction within the space as it encounters obstacles or unexpected events while completing its tasks. That robot's set of movements, obstacles, and challenges will be very different from those of a self-driving car, for example.How, then, can a single algorithm guide different robotic systems to make the best decisions to move through their surroundings?"You don't want a designer to have to go in and handcraft these motions and say, 'This is the discrete set of moves the robot should be able to do,'" says John Lathrop, a graduate student in control and dynamical systems at Caltech and co-lead author of the new paper. "To overcome this, we came up with SETS."SETS uses control theory and linear algebra to find natural motions that use a robotic platform's capabilities to its fullest extent in a physical setting.The basic underlying concept is based on a Monte Carlo Tree Search, a decision-making algorithm also used by Google's AlphaZero. Here, Monte Carlo essentially means something random, and tree search refers to navigating a branching structure that represents the relationships of data in a system. In such a tree, a root branches off to so-called child nodes that are connected by edges. Using Monte Carlo Tree Search for a game like Go, possible moves are represented as new nodes, and the tree grows larger as more random samples of possible trajectories are attempted. The algorithm plays out the possible moves to see the final outcomes of the different nodes and then selects the one that offers the best outcome based on a point valuation.The problem, Lathrop explains, is that when using this branching tree structure for continuous dynamical systems such as robots operating in the physical world, the total number of trajectories in the tree grows exponentially. "For some problems, trying to simulate every single possibility and then figure out which one is best would take years, maybe hundreds of years," he says.To overcome this, SETS takes advantage of an exploration/exploitation trade-off. "We want to try simulating trajectories that we haven't investigated before—that's exploration," Lathrop says. "And we want to continue looking down paths that have previously yielded high reward—that's exploitation. By balancing the exploration and the exploitation, the algorithm is able to quickly converge on the optimal solution among all possible trajectories."For example, if a robot starts out calculating a couple of possible actions that it determines would cause it to smash into a wall, there is no need for it to investigate any of the other nodes on that branch of the tree."This exploration/exploitation tradeoff and search over the robot's natural motions enables our robots to think, move, and adapt to new information in real-time," says Benjamin Rivière (PhD '24), a postdoctoral scholar research associate in mechanical and civil engineering at Caltech and co-lead author of the paper.SETS can run an entire tree search in about a tenth of a second. During that time, it can simulate thousands to tens of thousands of possible trajectories, select the best one, and then act. The loop continues over and over, giving the robotic system the ability to make many decisions each second.A key feature of the SETS algorithm is that it can be applied to essentially any robotic platform. The features and capabilities do not have to be programmed individually. In the new paper, Chung and his colleagues demonstrate the algorithm's successful utility in three completely different experimental settings—something that is very rare in robotics papers.In the first, a quadrotor drone was able to observe four hovering white balls while avoiding four orange balls, all while navigating an airfield rife with randomly occurring, dangerous air currents, or thermals. The drone experiment was conducted at Caltech's&nbsp;<a href="https://cast.caltech.edu/">Center for Autonomous Systems and Technologies</a> (CAST). In the second, the algorithm augmented a human driver of a tracked ground vehicle to navigate a narrow and winding track without hitting the siderails. And in the final setup, SETS helped a pair of tethered spacecraft capture and redirect a third agent, which could represent another spacecraft, an asteroid or another object.A team of Caltech students and researchers are currently applying a version of the SETS algorithm to an Indy car that will participate in the Indy Autonomous Challenge at the Consumer Electronics Show (CES) in Las Vegas on January 9.The work was supported by the Defense Advanced Research Projects Agency's Learning Introspective Control (LINC) program, the Aerospace Corporation, and Supernal, and is in part based on work supported by the National Science Foundation Graduate Research Fellowship Program.

A team of Caltech researchers has developed an analogous algorithm for autonomous robots—a planning and decision-making control system that helps freely moving robots determine the best movements to make as they navigate the real world.

Helping Robots Make Good Decisions in Real Time

In my last blog on AI and Machine Learning (ML) I emphasized that AI and ML would transform the IoT, not in 10 or even five years from now, but on product roadmaps coming to market within the next couple of years. I’d like to ‘up’ the urgency on that prediction. AI and ML aren’t a couple of years away, they are here. Not a tool of the future, but a dynamic force driving innovation and change now. &nbsp;Any question about whether there is going to be a big role for AI and ‘Edge AI’—where AI and ML is performed locally rather than sending data up to the Cloud—in the IoT has been dispelled.&nbsp;<h2>The ML and TinyML difference for Edge AI</h2>When we talk about AI in the IoT, I want to make it clear that, for the foreseeable future, we are really talking about ML and the stripped-down TinyML variant.&nbsp;ML adopts a model-based approach that relies on learning, trial and error from captured data. There’s a learning component (‘training’) that happens where there is an abundance of compute power like in the Cloud, and an on-the-fly predictive component (‘inferencing’) that in many cases can be so lightweight that it can run on small, embedded devices.&nbsp;As data is gathered, one can continuously re-do the training part and update the predictive part to get more accurate. As such, any attached sensor will continuously get better at monitoring what it is supposed to be monitoring.&nbsp;AI in the IoT also needs to be performed as close to the sensors as possible, all the way to the edge, on increasingly capable devices in terms of compute power. But devices that will still, by and large, be battery powered. And so much more resource-constrained than if they were plugged into the mains power grid.&nbsp;&nbsp;<h2>Three types of AI learning for the IoT&nbsp;</h2>We are also primarily talking about three types of IoT AI: supervised learning, reinforcement learning, and unsupervised learning. And the three main uses cases are ‘three Vs’ as Arm coined the term: vibration (anomaly detection and movement classification), voice (using audio recognition as a sensor), and vision (using image recognition as a sensor).&nbsp;The only difference being how much the ML model can learn on its own: running from very little (supervised) all the way through to autonomous learning (unsupervised) when given the right model to start with. Right now, ML can be run locally at the edge on IoT platform devices like Nordic’s latest&nbsp;<a href="https://www.nordicsemi.com/Products/nRF54H20">nRF54 Series</a>. You can optimize ML on a constrained device by using ML accelerators, achieving performance that can rival much more powerful devices and far outperform traditional MCUs. Once you have trained the basic model, which will typically take a day on a desktop PC, the inferencing side of things can accelerate hugely using AI-optimized algorithms.&nbsp;Nordic’s nRF54 Series was built for running optimized ML on constrained edge devices in just this way. It is around 25 percent more powerful than market-leading microprocessors and consumes 20 percent less energy than the lowest energy microprocessors in AI and ML edge computing applications.&nbsp;This means Nordic has enabled the advantages of edge computing, AI, and ML to be accessible to the broadest range of customers and applications.&nbsp;<h2>The edge computing,&nbsp;AI&nbsp;and ML advantage&nbsp;</h2>There are five primary advantages to customers. The first is latency. If you need real-time or near real-time responsiveness you must compute and take decisions locally. This also enables your AI and ML to perform quicker decision making.&nbsp;&nbsp;The next benefit is bandwidth. Edge computing reduces the reliance of having to maintain continuous network or Cloud connectivity that incur huge data overheads and data costs.&nbsp;The third benefit is privacy. Transmitting sensitive or personal data to the Cloud raises privacy concerns. Local edge processing and storage means better control over data confidentiality. This will help minimize the potential for security breaches.&nbsp;The fourth benefit of operating at the edge is cost. Cloud-based AI is significantly more expensive because it costs money to transmit a lot of data to and from the Cloud, and then pay for all that Cloud computing capability on top. Processing at the edge, in comparison, is free of such recurring operating costs.&nbsp;The fifth, and perhaps most important benefit, is energy efficiency. The world simply does not have enough resources for everything to be run in the Cloud. Smarter edge devices minimize power consumption, making them more environmentally friendly and beneficial for combating climate change.&nbsp;&nbsp;<h2>Ultra-low power connectivity leadership</h2>No company in the world does ultra-low power connectivity like Nordic. And no SoC does energy-efficient edge AI computing like the Nordic nRF54 Series. The latest advances in AI and ML acceleration that&nbsp;<a href="https://www.nordicsemi.com/Nordic-news/2023/08/Nordic-to-acquire-AI-ML-technology-in-the-US">Nordic’s Atlazo acquisition</a>&nbsp;has brought to our internal R&amp;D capabilities will only further extend our leadership in edge AI and ML.&nbsp;In my view, this development signifies a pivotal transformation for Nordic, elevating its identity far above being just a wireless connectivity or Bluetooth company within the IoT landscape, but also offering a seamless integration platform for AI in the IoT.&nbsp;

AI in the IoT also needs to be performed as close to the sensors as possible, all the way to the edge, on increasingly capable devices in terms of compute power. 

AI and machine learning goes mainstream

While generative artificial intelligence models for images can streamline artistic processes by enabling creators to produce lifelike 2D images from text prompts, these models are not designed to generate 3D shapes. To bridge the gap, a recently developed technique called&nbsp;<a href="https://dreamfusion3d.github.io/">Score Distillation</a> leverages 2D image generation models to create 3D shapes, but its output often ends up blurry or cartoonish.MIT researchers explored the relationships and differences between the algorithms used to generate 2D images and 3D shapes, identifying the root cause of lower-quality 3D models. From there, they crafted a simple fix to Score Distillation, which enables the generation of sharp, high-quality 3D shapes that are closer in quality to the best model-generated 2D images.Some other methods try to fix this problem by retraining or fine-tuning the generative AI model, which can be expensive and time-consuming.By contrast, the MIT researchers’ technique achieves 3D shape quality on par with or better than these approaches without additional training or complex postprocessing. Moreover, by identifying the cause of the problem, the researchers have improved mathematical understanding of Score Distillation and related techniques, enabling future work to further improve performance.“Now we know where we should be heading, which allows us to find more efficient solutions that are faster and higher-quality,” says Artem Lukoianov, an electrical engineering and computer science (EECS) graduate student who is lead author of a paper on this technique. “In the long run, our work can help facilitate the process to be a co-pilot for designers, making it easier to create more realistic 3D shapes.”Lukoianov’s co-authors are Haitz Sáez de Ocáriz Borde, a graduate student at Oxford University; Kristjan Greenewald, a research scientist in the MIT-IBM Watson AI Lab; Vitor Campagnolo Guizilini, a scientist at the Toyota Research Institute; Timur Bagautdinov, a research scientist at Meta; and senior authors Vincent Sitzmann, an assistant professor of EECS at MIT who leads the Scene Representation Group in the Computer Science and Artificial Intelligence Laboratory (CSAIL) and Justin Solomon, an associate professor of EECS and leader of the CSAIL Geometric Data Processing Group. The research will be presented at the Conference on Neural Information Processing Systems.<h2>From 2D images to 3D shapes</h2>Diffusion models, such as DALL-E, are a type of generative AI model that can produce lifelike images from random noise. To train these models, researchers add noise to images and then teach the model to reverse the process and remove the noise. The models use this learned “denoising” process to create images based on a user’s text prompts.But diffusion models underperform at directly generating realistic 3D shapes because there are not enough 3D data to train them. To get around this problem, researchers developed a technique called <a href="https://dreamfusion3d.github.io/">Score Distillation Sampling</a> (SDS) in 2022 that uses a pretrained diffusion model to combine 2D images into a 3D representation.The technique involves starting with a random 3D representation, rendering a 2D view of a desired object from a random camera angle, adding noise to that image, denoising it with a diffusion model, then optimizing the random 3D representation so it matches the denoised image. These steps are repeated until the desired 3D object is generated.However, 3D shapes produced this way tend to look blurry or oversaturated.“This has been a bottleneck for a while. We know the underlying model is capable of doing better, but people didn’t know why this is happening with 3D shapes,” Lukoianov says.The MIT researchers explored the steps of SDS and identified a mismatch between a formula that forms a key part of the process and its counterpart in 2D diffusion models. The formula tells the model how to update the random representation by adding and removing noise, one step at a time, to make it look more like the desired image.Since part of this formula involves an equation that is too complex to be solved efficiently, SDS replaces it with randomly sampled noise at each step. The MIT researchers found that this noise leads to blurry or cartoonish 3D shapes.<h2>An approximate answer</h2>Instead of trying to solve this cumbersome formula precisely, the researchers tested approximation techniques until they identified the best one. Rather than randomly sampling the noise term, their approximation technique infers the missing term from the current 3D shape rendering.“By doing this, as the analysis in the paper predicts, it generates 3D shapes that look sharp and realistic,” he says.In addition, the researchers increased the resolution of the image rendering and adjusted some model parameters to further boost 3D shape quality.In the end, they were able to use an off-the-shelf, pretrained image diffusion model to create smooth, realistic-looking 3D shapes without the need for costly retraining. The 3D objects are similarly sharp to those produced using other methods that rely on ad hoc solutions.“Trying to blindly experiment with different parameters, sometimes it works and sometimes it doesn’t, but you don’t know why. We know this is the equation we need to solve. Now, this allows us to think of more efficient ways to solve it,” he says.Because their method relies on a pretrained diffusion model, it inherits the biases and shortcomings of that model, making it prone to hallucinations and other failures. Improving the underlying diffusion model would enhance their process.In addition to studying the formula to see how they could solve it more effectively, the researchers are interested in exploring how these insights could improve image editing techniques.This work is funded, in part, by the Toyota Research Institute, the U.S. National Science Foundation, the Singapore Defense Science and Technology Agency, the U.S. Intelligence Advanced Research Projects Activity, the Amazon Science Hub, IBM, the U.S. Army Research Office, the CSAIL Future of Data program, the Wistron Corporation, and the MIT-IBM Watson AI Laboratory.

Creating realistic 3D models for applications like virtual reality, filmmaking, and engineering design can be a cumbersome process requiring lots of manual trial and error.

A new way to create realistic 3D shapes using generative AI

The deep neural network models that power today’s most demanding machine-learning applications have grown so large and complex that they are pushing the limits of traditional electronic computing hardware.Photonic hardware, which can perform machine-learning computations with light, offers a faster and more energy-efficient alternative. However, there are some types of neural network computations that a photonic device can’t perform, requiring the use of off-chip electronics or other techniques that hamper speed and efficiency.Building on a decade of research, scientists from MIT and elsewhere have developed a new photonic chip that overcomes these roadblocks. They demonstrated a fully integrated photonic processor that can perform all the key computations of a deep neural network optically on the chip.The optical device was able to complete the key computations for a machine-learning classification task in less than half a nanosecond while achieving more than 92 percent accuracy — performance that is on par with traditional hardware.The chip, composed of interconnected modules that form an optical neural network, is fabricated using commercial foundry processes, which could enable the scaling of the technology and its integration into electronics.In the long run, the photonic processor could lead to faster and more energy-efficient deep learning for computationally demanding applications like lidar, scientific research in astronomy and particle physics, or high-speed telecommunications.“There are a lot of cases where how well the model performs isn’t the only thing that matters, but also how fast you can get an answer. Now that we have an end-to-end system that can run a neural network in optics, at a nanosecond time scale, we can start thinking at a higher level about applications and algorithms,” says Saumil Bandyopadhyay ’17, MEng ’18, PhD ’23, a visiting scientist in the Quantum Photonics and AI Group within the Research Laboratory of Electronics (RLE) and a postdoc at NTT Research, Inc., who is the lead author of a paper on the new chip.Bandyopadhyay is joined on the paper by Alexander Sludds ’18, MEng ’19, PhD ’23; Nicholas Harris PhD ’17; Darius Bunandar PhD ’19; Stefan Krastanov, a former RLE research scientist who is now an assistant professor at the University of Massachusetts at Amherst; Ryan Hamerly, a visiting scientist at RLE and senior scientist at NTT Research; Matthew Streshinsky, a former silicon photonics lead at Nokia who is now co-founder and CEO of Enosemi; Michael Hochberg, president of Periplous, LLC; and Dirk Englund, a professor in the Department of Electrical Engineering and Computer Science, principal investigator of the Quantum Photonics and Artificial Intelligence Group and of RLE, and senior author of the paper. The research&nbsp;<a href="https://www.nature.com/articles/s41566-024-01567-z" target="_blank">appears today in&nbsp;Nature Photonics</a>.<h2>Machine learning with light</h2>Deep neural networks are composed of many interconnected layers of nodes, or neurons, that operate on input data to produce an output. One key operation in a deep neural network involves the use of linear algebra to perform matrix multiplication, which transforms data as it is passed from layer to layer.But in addition to these linear operations, deep neural networks perform nonlinear operations that help the model learn more intricate patterns. Nonlinear operations, like activation functions, give deep neural networks the power to solve complex problems.In 2017, Englund’s group, along with researchers in the lab of Marin Soljačić, the Cecil and Ida Green Professor of Physics,&nbsp;<a href="https://news.mit.edu/2017/new-system-allows-optical-deep-learning-0612" target="_blank">demonstrated an optical neural network on a single photonic chip</a> that could perform matrix multiplication with light.But at the time, the device couldn’t perform nonlinear operations on the chip. Optical data had to be converted into electrical signals and sent to a digital processor to perform nonlinear operations.“Nonlinearity in optics is quite challenging because photons don’t interact with each other very easily. That makes it very power consuming to trigger optical nonlinearities, so it becomes challenging to build a system that can do it in a scalable way,” Bandyopadhyay explains.They overcame that challenge by designing devices called nonlinear optical function units (NOFUs), which combine electronics and optics to implement nonlinear operations on the chip.The researchers built an optical deep neural network on a photonic chip using three layers of devices that perform linear and nonlinear operations.<h2>A fully-integrated network</h2>At the outset, their system encodes the parameters of a deep neural network into light. Then, an array of programmable beamsplitters, which was&nbsp;demonstrated&nbsp;in the 2017 paper, performs matrix multiplication on those inputs.The data then pass to programmable NOFUs, which implement nonlinear functions by siphoning off a small amount of light to photodiodes that convert optical signals to electric current. This process, which eliminates the need for an external amplifier, consumes very little energy.“We stay in the optical domain the whole time, until the end when we want to read out the answer. This enables us to achieve ultra-low latency,” Bandyopadhyay says.Achieving such low latency enabled them to efficiently train a deep neural network on the chip, a process known as in situ&nbsp;training that typically consumes a huge amount of energy in digital hardware.“This is especially useful for systems where you are doing in-domain processing of optical signals, like navigation or telecommunications, but also in systems that you want&nbsp;to learn in real time,” he says.The photonic system achieved more than 96 percent accuracy during training tests and more than 92 percent accuracy during inference, which is comparable to traditional hardware. In addition, the chip performs key computations in less than half a nanosecond. &nbsp; &nbsp;&nbsp;“This work demonstrates that computing — at its essence, the mapping of inputs to outputs — can be compiled onto new architectures of linear and nonlinear physics that enable a fundamentally different scaling law of computation versus effort needed,” says Englund.The entire circuit was fabricated using the same infrastructure and foundry processes that produce CMOS computer chips. This could enable the chip to be manufactured at scale, using tried-and-true techniques that introduce very little error into the fabrication process.Scaling up their device and integrating it with real-world electronics like cameras or telecommunications systems will be a major focus of future work, Bandyopadhyay says. In addition, the researchers want to explore algorithms that can leverage the advantages of optics to train systems faster and with better energy efficiency.

Researchers demonstrated a fully integrated photonic processor that can perform all key computations of a deep neural network optically on the chip, which could enable faster and more energy-efficient deep learning for computationally demanding applications like lidar or high-speed telecommunications. Image: Sampson Wilcox, Research Laboratory of Electronics.

This new device uses light to perform the key operations of a deep neural network on a chip, opening the door to high-speed processors that can learn in real-time.

Photonic processor could enable ultrafast AI computations with extreme energy efficiency

ChatGPT exploded onto the public scene in late 2022 attracting more than 100-million users in just its first month. Since then, there have been rising examples of how AI may transform society in the coming years, from employment and communication to education.In higher education, AI assistants are being increasingly used by students. While these tools provide opportunities for improved teaching and education, they also pose significant challenges for assessment and learning outcomes yet, until now, there has been no comprehensive study of their potential impact on the assessment methods that educational institutions use.As outlined in their&nbsp;<a href="https://www.pnas.org/doi/full/10.1073/pnas.2414955121" target="_blank" rel="noopener">new paper published in the Proceedings of the National Academy of Sciences</a> (PNAS), researchers from EPFL’s&nbsp;<a href="https://www.epfl.ch/schools/ic/">School of Computer and Communication Sciences</a> have conducted a large-scale study across 50 EPFL courses to measure the current performance of Large Language Models on higher education course assessments. The selected courses were sampled from 9 Bachelor’s, Master’s, and online programs, covering a broad spectrum of STEM disciplines, including computer science, mathematics, biology, chemistry, physics, and material sciences."We were lucky that a large consortium of EPFL professors, teachers, and teaching assistants helped us collect the largest data set to date of course materials, assessments, and exams to get a diverse array of materials across our degree programs,” explained Assistant Professor Antoine Bosselut, Head of the&nbsp;<a href="https://nlp.epfl.ch/">Natural Language Processing Laboratory</a> (NLP) and member of the&nbsp;<a href="https://ai.epfl.ch/">EPFL AI Center</a>. “This data was curated into a format that we thought would most resemble the ways that students would actually give this information to models and then we generated responses from the models and saw how well they answered.”Focused on GPT-3.5 and GPT-4, the researchers used eight prompting strategies to produce responses and found that GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions.“We were surprised at the results, nobody expected that the AI assistants would achieve such a high percentage of correct answers across so many courses. Importantly, the 65% of questions answered correctly was achieved using the most basic, no knowledge prompt strategy, so anyone, without understanding anything technically could achieve this. With some subject knowledge, which is typical, it was possible to achieve an 85% success rate and that was really the shock,” said Anna Sotnikova, a scientist with the NPL and co-author on the paper.<blockquote>We were surprised at the results, nobody expected that the AI assistants would achieve such a high percentage of correct answers across so many courses.<footer>Anna Sotnikova, a scientist at the Natural Language Processing Laboratory</footer></blockquote><h2>AI's&nbsp;impact on&nbsp;student&nbsp;learning and&nbsp;skill&nbsp;development</h2>The researchers theoretically grounded the problems associated with these AI systems being used by students around vulnerability – on the one hand, assessment vulnerability or whether traditionally used assessments can be ‘gamed’ by these systems and on the other hand, educational vulnerability, that is, could these systems be used to circumvent the typical cognitive paths that students take to learn the academic skills they need.In this context, the researchers believe that the results of the study bring up clear questions around how to ensure that students are able to learn the core concepts needed in order to grasp more complex topics later on.“The fear is that if these models are as capable as what we indicate, students that use them might shortcut the process through which they would learn new concepts. This could build weaker foundations for certain skills earlier on, making it harder to learn more complex concepts later. Maybe this needs to be a debate about what we should be teaching in the first place in order to come up with the best synergies of the technologies we have and what students will do in decades to come,” said Bosselut.Another key point of the development of AI assistants is that they are not going to get worse, they will only get better. In this research, which was completed a year ago, one single model was used for all subjects and, for example, it had particular trouble with mathematics questions. Now there are specific models for math. The conclusion, the researchers say, is that if the study was rerun today, the numbers would be even higher.<h2>Emphasizing&nbsp;complex&nbsp;assessments and&nbsp;adapting education</h2>“In the short term we should push for harder assessments - not in the sense of question difficulty, but in the sense of the complexity of the assessment itself where multiple skills have to be taken from different concepts that are learned throughout the course during the semester and that are brought together in a holistic assessment,” Bosselut suggested. “The models aren’t yet really designed to plan and work in this kind of way and in the end, we actually think this project-based learning is better for students anyway.”“AI challenges higher education institutions in many ways, for example: which new skills are required for future graduates, which are becoming obsolete, how can we provide feedback at scale and how do we measure knowledge? These kinds of questions come up in almost every management meeting at EPFL and what matters most is that our teams initiate projects which provide evidence-based answers to as many of them as we can,” said Pierre Dillenbourg, Vice President for Academic Affairs at EPFL.<blockquote>AI challenges higher education institutions in many ways.<footer>Pierre Dillenbourg, Vice President for Academic Affairs at EPFL</footer></blockquote>Longer-term it’s clear that education systems will need to adapt and the researchers want to bring this ongoing project closer towards educators, aligning studies and then recommendations with what they will find useful.“This is only the beginning and I think a good analogy to LLMs right now are calculators when they were introduced, there were a similar set of concerns that children would no longer learn mathematics. Now, in the earlier phases of education calculators are usually not allowed but by high school and above, they are expected, taking care of lower level work while students are learning more advanced skills that rely upon them,” added Beatriz Borges, a PhD student in the NLP and co-author of the paper.“I think that we will see a similar, gradual adaptation and shift to an understanding of both what these systems can do for us and what we can't rely on them to do. Ultimately, we include practical suggestions to better support students, teachers, administrators, and everyone else during this transition, while also while also helping to reduce some of the risks and vulnerabilities outlined in the paper,” she concluded.The&nbsp;<a href="https://ai.epfl.ch/">EPFL AI Center</a>&nbsp;brings together 80 laboratories and professors and has 1000 affiliates, leading&nbsp;the way towards trustworthy, accessible, and inclusive AI.

EPFL research investigating the potential impact on education of AI assistants has found that systems like GPT-4 can answer up to 85% of university assessment questions correctly.

Could ChatGPT get an engineering degree?

Visualizing the potential impacts of a hurricane on people’s homes before it hits can help residents prepare and decide whether to evacuate.MIT scientists have developed a method that generates satellite imagery from the future to depict how a region would look after a potential flooding event. The method combines a generative artificial intelligence model with a physics-based flood model to create realistic, birds-eye-view images of a region, showing where flooding is likely to occur given the strength of an oncoming storm.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1732838043094-MIT-AI-Flood-01-press_0.jpg" style="width: 832px;" class="fr-fic fr-dib">Caption:A generative AI model visualizes how floods in Texas would look like in satellite imagery. The original photo is on the left, and the AI generated image is in on the right.&nbsp; Credit: Pre-flood images from Maxar Open Data Program via Gupta et al., CVPR Workshop Proceedings. Generated images from Lütjen et al., IEEE TGRS.As a test case, the team applied the method to Houston and generated satellite images depicting what certain locations around the city would look like after a storm comparable to Hurricane Harvey, which hit the region in 2017. The team compared these generated images with actual satellite images taken of the same regions after Harvey hit. They also compared AI-generated images that did not include a physics-based flood model.The team’s physics-reinforced method generated satellite images of future flooding that were more realistic and accurate. The AI-only method, in contrast, generated images of flooding in places where flooding is not physically possible.The team’s method is a proof-of-concept, meant to demonstrate a case in which generative AI models can generate realistic, trustworthy content when paired with a physics-based model.&nbsp;In order to apply the method to other regions to depict flooding from future storms, it will need to be trained on many more satellite images to learn how flooding would look in other regions.“The idea is: One day, we could use this before a hurricane, where it provides an additional visualization layer for the public,” says Björn Lütjens, a postdoc in MIT’s Department of Earth, Atmospheric and Planetary Sciences, who led the research while he was a doctoral student in MIT’s Department of Aeronautics and Astronautics (AeroAstro). “One of the biggest challenges is encouraging people to evacuate when they are at risk. Maybe this could be another visualization to help increase that readiness.”To illustrate the potential of the new method, which they have dubbed the “Earth Intelligence Engine,” the team has made it <a href="https://climate-viz.github.io/" target="_blank">available</a> as an online resource for others to try.The researchers <a href="https://ieeexplore.ieee.org/document/10758300" target="_blank">report their results today in the journal IEEE Transactions on Geoscience and Remote Sensing</a>.&nbsp;The study’s MIT co-authors include Brandon Leshchinskiy; Aruna Sankaranarayanan; and Dava Newman, professor of AeroAstro and director of the MIT Media Lab; along with collaborators from multiple institutions.<h2>Generative adversarial images</h2>The new study is an extension of the team’s efforts to apply generative AI tools to visualize future climate scenarios.“Providing a hyper-local perspective of climate seems to be the most effective way to communicate our scientific results,” says Newman, the study’s senior author. “People relate to their own zip code, their local environment where their family and friends live. Providing local climate simulations becomes intuitive, personal, and relatable.”For this study, the authors use a conditional generative adversarial network, or GAN, a type of machine learning method that can generate realistic images using two competing, or “adversarial,” neural networks. The first “generator” network is trained on pairs of real data, such as satellite images before and after a hurricane. The second “discriminator” network is then trained to distinguish between the real satellite imagery and the one synthesized by the first network.Each network automatically improves its performance based on feedback from the other network. The idea, then, is that such an adversarial push and pull should ultimately produce synthetic images that are indistinguishable from the real thing. Nevertheless, GANs can still produce “hallucinations,” or factually incorrect features in an otherwise realistic image that shouldn’t be there.“Hallucinations can mislead viewers,” says Lütjens, who began to wonder whether such hallucinations could be avoided, such that generative AI tools can be trusted to help inform people, particularly in risk-sensitive scenarios. “We were thinking: How can we use these generative AI models in a climate-impact setting, where having trusted data sources is so important?”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1732838099417-MIT-AI-Flood-02-press.jpg" style="width: 100%px;" class="fr-fic fr-dib">Here, the AI model visualizes how flood in North Carolina would look like in satellite imagery. These new visualizations of physics-based flood models might facilitate more intuitive communication of climate risks.&nbsp; Credit: Pre-flood images from Maxar Open Data Program via Gupta et al., CVPR Workshop Proceedings. Generated images from Lütjen et al., IEEE TGRS.<h2>Flood hallucinations</h2>In their new work, the researchers considered a risk-sensitive scenario in which generative AI is tasked with creating satellite images of future flooding that could be trustworthy enough to inform decisions of how to prepare and potentially evacuate people out of harm’s way.Typically, policymakers can get an idea of where flooding might occur based on visualizations in the form of color-coded maps. These maps are the final product of a pipeline of physical models that usually begins with a hurricane track model, which then feeds into a wind model that simulates the pattern and strength of winds over a local region.&nbsp;This is combined with a flood or storm surge model that forecasts how wind might push any nearby body of water onto land. A hydraulic model then maps out where flooding will occur based on the local flood infrastructure and generates a visual, color-coded&nbsp;map of flood elevations over a particular region.“The question is: Can visualizations of satellite imagery add another level to this, that is a bit more tangible and emotionally engaging than a color-coded map of reds, yellows, and blues, while still being trustworthy?” Lütjens says.The team first tested how generative AI alone would produce satellite images of future flooding. They trained a GAN on actual satellite images taken by&nbsp;satellites as they passed over Houston before and after Hurricane Harvey. When they tasked the generator to produce new flood images of the same regions, they found that the images resembled typical satellite imagery, but a closer look revealed hallucinations in some images, in the form of floods where flooding should not be possible (for instance, in locations at higher elevation).To reduce hallucinations and increase the trustworthiness of the AI-generated images, the team paired the GAN with a physics-based flood model that incorporates real, physical parameters and phenomena, such as an approaching hurricane’s trajectory, storm surge, and flood patterns.&nbsp;With this physics-reinforced method, the team generated satellite images around Houston that depict the same flood extent, pixel by pixel, as forecasted by the flood model.“We show a tangible way to combine machine learning with physics for a use case that’s risk-sensitive, which requires us to analyze the complexity of Earth’s systems and project future actions and possible scenarios to keep people out of harm’s way,” Newman says. “We can’t wait to get our generative AI tools into the hands of decision-makers at the local community level, which could make a significant difference and perhaps save lives.”<hr>The research was supported, in part, by the MIT Portugal Program, the DAF-MIT Artificial Intelligence Accelerator, NASA, and Google Cloud.

The method could help communities visualize and prepare for approaching storms.

New AI tool generates realistic satellite images of future flooding

Cutting back on animal protein in our diets can save on resources and greenhouse gas emissions. But convincing meat-loving consumers to switch up their menu is a challenge. Looking at this problem from a mechanical engineering angle, Stanford engineers are pioneering a new approach to food texture testing that could pave the way for faux filets that fool even committed carnivores.In a <a href="https://www.nature.com/articles/s41538-024-00330-6" data-extlink="">new paper</a> in Science of Food, the team demonstrated that a combination of mechanical testing and machine learning can describe food texture with striking similarity to human taste testers. Such a method could speed up the development of new and better plant-based meats. The team also found that some plant-based products are already nailing the texture of the meats they’re mimicking.“We were surprised to find that today’s plant-based products can reproduce the whole texture spectrum of animal meats,” said <a href="https://profiles.stanford.edu/ellen-kuhl" data-extlink="">Ellen Kuhl</a>, professor of mechanical engineering and senior author of the study. Meat substitutes have come a long way from when tofu was the only option, she added.Industrial animal agriculture contributes to climate change, pollution, habitat loss, and antibiotic resistance. That burden on the planet can be eased by swapping animal proteins for plant proteins in diets. One study estimated that plant-based meats, on average, have <a href="https://www.sciencedirect.com/science/article/pii/S0921344922006632" data-extlink="">half the environmental impact</a> as animal meat. But many meat eaters are reluctant to change; <a href="https://www.frontiersin.org/journals/sustainable-food-systems/articles/10.3389/fsufs.2019.00011/full" data-extlink="">only about a third of Americans</a> in one survey indicated they were “very likely” or “extremely likely” to buy plant-based alternatives.“People love meat,” said <a href="https://livingmatter.stanford.edu/people/skyler-st-pierre" data-extlink="">Skyler St. Pierre</a>, a PhD student in mechanical engineering and lead author of the paper. “If we want to convince the hardcore meat eaters that alternatives are worth trying, the closer we can mimic animal meat with plant-based products, the more likely people might be open to trying something new.”To successfully mimic animal meat, food scientists analyze the texture of plant-based meat products. Unfortunately, traditional food testing methods are not standardized and the results are rarely made available to science and to the public, said St. Pierre. This makes it harder for scientists to collaborate and create new recipes for alternatives.<h2>New food texture tests</h2>The research grew out of a class project by St. Pierre. Looking for affordable materials to use in mechanical tests, he turned to hot dogs and tofu. Over the summer of 2023, undergraduate researchers joined in to test the foods and learn how engineers depict material responses to stress, loading, and stretching.Realizing how this work could aid the development of plant-based meats, the Stanford team debuted a three-dimensional food test. They put eight products to the test: animal and plant-based hot dog, animal and plant-based sausage, animal and plant-based turkey, and extra firm and firm tofu. They mounted bits of meat into a machine that pulled, pushed, and sheared on the samples. “These three loading modes represent what you do when you chew,” said Kuhl, who is also the Catherine Holman Johnson Director of <a href="http://biox.stanford.edu/" data-extlink="">Stanford Bio-X</a> and the Walter B. Reinhold Professor in the School of Engineering.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1732580654113-20241108_edible_foods2_0.jpg.webp" style="width: 100%px;" class="fr-fic fr-dib">A mechanical test of plant-based meat can help quantify food texture. | Kurt HickmanThen, they used machine learning to process the data from these tests: They designed a new type of neural network that takes the raw data from the tests and produces equations that explain the properties of the meats.To see if these equations can explain the perception of texture, the team carried out a taste test survey. The tasters—who first completed surveys on their openness to new foods and their attachment to meat—ate samples of the eight products and rated them on five-point scale for 12 categories: soft, hard, brittle, chewy, gummy, viscous, springy, sticky, fibrous, fatty, moist, and meat-like.&nbsp;<h2>Impressive hot dogs and sausages</h2>In the mechanical testing, the plant-based hot dog and sausage behaved very similarly in the pulling, pushing, and shear tests to their animal counterparts, and showed similar stiffnesses. Meanwhile, the plant-based turkey was twice as stiff as animal turkey, and the tofu was much softer than the meat products. Strikingly, the human testers also ranked the stiffness of the hot dogs and sausages very similarly to the mechanical tests. “What’s really cool is that the ranking of the people was almost identical to the ranking of the machine,” said Kuhl. “That’s great because now we can use the machine to have a quantitative, very reproducible test.”The findings suggest that new, data-driven methods hold promise for speeding up the process of developing tasty plant-based products. “Instead of using a trial-and-error approach to improve the texture of plant-based meat, we could envision using generative artificial intelligence to scientifically generate recipes for plant-based meat products with precisely desired properties,” the authors wrote in the paper.But artificial intelligence recipe development, like other AIs, needs lots of data. That’s why the team is&nbsp;<a href="https://github.com/LivingMatterLab/CANN" data-extlink="">sharing their data</a> online, making it open for other researchers to view and add to. “Historically, some researchers, and especially companies, don’t share their data and that’s a really big barrier to innovation,” said St. Pierre. Without sharing information and working together, he added, “how are we going to come up with a steak mimic together?”The team is continuing to test foods and build a public database. This summer, St. Pierre oversaw undergraduates testing veggie and meat deli slices. The researchers also plan to test engineered fungi developed by&nbsp;<a href="https://profiles.stanford.edu/349659" data-extlink="">Vayu Hill-Maini</a>, who recently joined Stanford as an assistant professor of bioengineering. “If anybody has an artificial or a plant-based meat they want to test,” said Kuhl, “we’re so happy to test it to see how it stacks up.”Additional Stanford co-authors of the paper include mechanical engineering professor&nbsp;<a href="https://profiles.stanford.edu/marc-levenston" data-extlink="">Marc Levenston</a>, associate professor of mechanical engineering; postdoctoral researcher Kevin Linka; graduate students Ethan C. Darwin, Divya Adil, Reese A. Dunne, and Valerie A. Perez Medina; visiting researcher María Parra Vallecillo; and undergraduate researchers Magaly C. Aviles, Archer Date, Yanav Lall.Kuhl is also a professor, by courtesy, of bioengineering in the Schools of Engineering and&nbsp;<a href="https://med.stanford.edu/" data-extlink="">Medicine</a>, and a member of the&nbsp;<a href="http://cvi.stanford.edu/" data-extlink="">Cardiovascular Institute</a>, the&nbsp;<a href="https://humanperformance.stanford.edu/" data-extlink="">Wu Tsai Human Performance Alliance</a>, the&nbsp;<a href="https://icme.stanford.edu/" data-extlink="">Institute for Computational and Mathematical Engineering</a>, and the&nbsp;<a href="https://neuroinstitute.stanford.edu/" data-extlink="">Wu Tsai Neurosciences Institute</a>. Levenston is also a member of Bio-X, the Cardiovascular Institute, the Wu Tsai Human Performance Alliance and the Maternal &amp; Child Health Research Institute.

Is it possible to close the texture gap between plant-based meat and animal meat? Ellen Kuhl and her lab are trying. From left: Skyler St. Pierre, Marc Levenston, Ellen Kuhl, Reese Dunne, Ethan Darwin, Valerie Perez Medina, and Divya Adil pose with the meat and plant-based meat they analyzed. | Kurt Hickman

Stanford engineers are testing foods using an approach that combines a three-dimensional mechanical technique and AI to mimic our sensory experience.

Can AI help improve plant-based meats?

Artificial intelligence (AI) is rapidly transforming healthcare by leveraging machine learning techniques to analyze medical images, patient data, and genetic information with unprecedented speed and accuracy, significantly improving early disease detection and enhancing diagnostic precision.


AI-Powered Health Diagnostics

Optical sensors are the “eyes” of systems in industrial applications. However, just as with humans, in industrial applications a powerful peripheral system must ensure that image processing permanently functions reliably, without errors, and at the required performance level.As optical sensors become increasingly powerful, the number of applications in which they can be used is also increasing. Along with the associated quality management and documentation system, the monitoring and evaluation electronics in state-of-the art production and documentation systems produce a flood of data. This mass of data must be transmitted, collected, and processed reliably. The underlying technical structures of machine vision systems must remain fully functional even under demanding conditions, such as permanent mechanical stress due to shocks and vibrations. Data connectors in particular, which are important interfaces within the system, also have to meet these requirements so that there is no wear on the male and female contacts (cf. fretting corrosion). Otherwise, contact interruptions will lead to downtimes in production or process errors that require time-consuming and cost-intensive measures.Increasingly compact systems in smaller spaces, following the trend of miniaturization, pose additional challenges for device developers of machine vision systems. These include electromagnetic interference pulses that can influence the functionality of the individual components. The reduced structures within the system architecture also jeopardize stable system integrity. The increasingly powerful components, installed increasingly closer together, also bring with them a thermal problem that may not only have an impact on the operation of the sensors, but may also make device approval more difficult. Miniaturized interfaces and device-integrated connection technology that provide secure shielding and data transmission are active solutions to these potential problems. The optimized positioning of components on the printed circuit board in accordance with electromagnetic and thermal considerations during device development also ensures reliability. Simulations of heat generation, signal integrity, and impedance are helpful, especially when non-standard components are used.&nbsp;<h2>Standards for optimized image processing</h2>There are numerous transmission standards, including Camera Link, CoaXPress, GigE Vision, and USB 3, to suit the respective machine vision applications and associated interfaces. All factors must be coordinated and work reliably together to ensure optimized image processing.&nbsp;Phoenix Contact is synonymous with high quality in the field of industrial connection technology. The portfolio of the company, founded in 1923, includes tried and tested data connectors such as D-SUB, RJ45, USB, HDMI, coaxial, and fiber-optic connection technology.&nbsp;With its commitment to quality and innovation, Phoenix Contact is also an expert partner for new components, such as the Single Pair Ethernet (SPE), RJ45 Industrial (RJ45-IND), and the connection system for shielded and unshielded Finepitch series board-to-board connectors.RJ45 IndustrialThe RJ45 Industrial (RJ45-IND) product family was developed especially for industrial applications. This powerful product family has been further developed and optimized based on a large amount of experience gained with the standard RJ45 products in cooperation with customers from different industries. With this development, Phoenix Contact has created a product range featuring an optimized 360° shielding concept, for example.To counter electromagnetic interference, the housing shield springs and shield pins for the printed circuit board have been raised. This also improves the mechanical strength of the socket on the printed circuit board.The connection between RJ45-IND connectors or patch cables and RJ45-IND female printed circuit board (PCB) connectors has been decisively optimized. Shocks and vibrations are now counteracted by the so-called “perfect match” of the connector system. The development also included optimizing the contact depth of the female PCB connector from 8.3 m to 9 mm, thus enlarging the contact zone of the connector system. There have also been decisive improvements in the tolerances in the connection of the socket and plug, meaning that their contact connection is even better matched. On the connector side, the contact pressure was increased further through the inclusion of a compression spring.&nbsp;The temperature resistance of the female PCB connector has been improved through the use of a reinforced housing, now utilizing fiberglass technology. Temperatures from -40 up to +105 C are now possible without any problems.&nbsp;The RJ45-IND patch cables, which are suitable for drag chain applications and robots, are also particularly robust. A major advantage and unique selling point of the Phoenix Contact solution is the IDC fast connection, which enables the one-piece connector to be assembled quickly and easily without the use of a special tool.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1732577673789-Bild+3_RJ45+IND+rechts+%281%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Single Pair EthernetThe IEEE 802.3 task force has developed the transmission standards for Single Pair Ethernet (SPE), defining the standards 10Base-T1, 100Base-T1, 1000Base-T1, and MultiGigBASE-T1.&nbsp;The SPE standards enable data rates of 10 Mbps up to 1 Gbps at distances of 15 to 1,000 m. The MultiGigBASE-T1 standard goes so far as to enable data rates of up to 10 Gbps at distances of up to 15 m. The power transmission for the SPE standards 100BASE-T1 and 1000BASE-T1 are defined by the IEEE 802.3 bu standard (PoDL – Power over Data Lines).&nbsp;The load outputs of the end devices (power devices) are divided into 10 performance classes from 0.5 to 50 W.&nbsp;With regard to industrial image processing, SPE is an exciting field of application not just because of the potential to transmit data and power. In particular, the SPE portfolio from Phoenix Contact plays to its strengths here with its compact and 50 W consistent design. For example, the SPE-IP20 socket saves about 50 percent of space compared to a standardized RJ45 jack.&nbsp;Phoenix Contact also has a consistent SPE product portfolio in which all components have been matched and tested. This portfolio provides compact IP20 connectors (in accordance with IEC&nbsp;63171-2) and IP67-protected M8 connectors (in accordance with IEC&nbsp;63171-5). A brand new addition to the portfolio is the SPE-M12 hybrid connector from the Onepair series.The advantages of the SPE connection technology include the uniform mating face and the possibility to achieve a high degree of common part usage. This means that adapters are no longer necessary and no additional costs are incurred.Various SPE components are available for the developers of machine vision sensors. These include new semiconductors (SPE PHYs and PoDL controllers), SPE magnetics, connection technology in accordance with IEC 63171-2 and -5, and SPE-compatible cables in accordance with IEC 61156.&nbsp;Tip: With a membership in the System Alliance (<a href="http://www.singlepairethernet.com/">www.singlepairethernet.com/</a>), it is possible to gain detailed information about Single Pair Ethernet and become familiar with specific solutions for industrial applications.Board-to-board connectorsContact systems are to be preferred over simply designed male and female strips, in particular in demanding applications – such as for industrial image processing, for example. They are sophisticated yet at the same time robust.&nbsp;With its board-to-board connectors, the Finepitch product series from Phoenix Contact offers a professional contact system for space-saving PCB connections. Various numbers of positions as well as stack heights provide a great deal of flexibility for developers. The system features high-speed data transmission rates of up to 52&nbsp;Gbps. The innovative ScaleX double contacts (FP&nbsp;0,8) are designed as hermaphroditic, meaning that vibration-resistant connections can be created in very confined spaces.The shielded version available as an option also prevents electromagnetic interferences.As a valuable service offering, Phoenix Contact experts provide support during the entire design-in process as well as design-in support. This means that device developers of machine vision systems can quickly obtain preconfigured 2D/3D data for their CAD software.&nbsp;Phoenix Contact also provides individual thermal and housing simulations based on the finite element method (FEM). Free simulations for data and signal integrity complete the comprehensive service offering. With this support, customers can realize stable communication right from the start and throughout the entire development process.&nbsp;Phoenix Contact rounds off its services with a free sample service and reliable logistics.&nbsp;<h2>Summary&nbsp;</h2>The developers of modern machine vision systems must consider various important factors during the product development process to ensure that reliable and high-performance systems are ready for the market in the environment of high data transmission rates and miniaturization. The cost factor also plays a decisive role. Phoenix Contact is a reliable partner, providing both products and services. In addition to the proven data connectors and innovative product solutions – such as for SPE, for example – Phoenix Contact provides simulations that help to realize stable device communication and reliable systems with cost-efficiency. This creates a high-performance periphery for reliable industrial image processing.Written by:&nbsp;Dipl.-Ing. Verena Neuhaus, Product Management Data Connectors, Business Unit Field Device Connectors, Phoenix Contact GmbH &amp; Co. KG, Blomberg, GermanyThomas Schulze, M.Sc., Product Management Board-to-Board Connectors, Business Unit PCB Connectors, Phoenix Contact GmbH &amp; Co. KG, Blomberg, Germany

Optical sensors are the “eyes” of industrial systems, relying on robust peripherals for reliable image processing. As sensors grow more powerful, challenges like data management, mechanical stress, and thermal issues in compact designs demand innovative solutions.

Image data in industrial applications: Ensuring reliability and performance during transmission

Now, researchers have introduced a technique for compressing an LLM’s reams of data, which could increase privacy, save energy and lower costs.The&nbsp;<a href="https://arxiv.org/abs/2405.18886">new algorithm</a>, developed by engineers at Princeton and Stanford Engineering, works by trimming redundancies and reducing the precision of an LLM’s layers of information. This type of leaner LLM could be stored and accessed locally on a device like a phone or laptop and could provide performance nearly as accurate and nuanced as an uncompressed version.“Any time you can reduce the computational complexity, storage and bandwidth requirements of using AI models, you can enable AI on devices and systems that otherwise couldn’t handle such compute- and memory-intensive tasks,” said study coauthor&nbsp;<a href="https://engineering.princeton.edu/faculty/andrea-goldsmith">Andrea Goldsmith</a>, dean of Princeton’s School of Engineering and Applied Science and Arthur LeGrand Doty Professor of&nbsp;<a href="https://ece.princeton.edu/">Electrical and Computer Engineering</a>.“When you use ChatGPT, whatever request you give it goes to the back-end servers of OpenAI, which process all of that data, and that is very expensive,” said coauthor&nbsp;<a href="https://sites.google.com/view/rajarshi-saha">Rajarshi Saha</a>, a Stanford Engineering Ph.D. student. “So, you want to be able to do this LLM inference using consumer GPUs [graphics processing units], and the way to do that is by compressing these LLMs.” Saha’s graduate work is coadvised by Goldsmith and coauthor&nbsp;<a href="https://stanford.edu/~pilanci">Mert Pilanci</a>, an assistant professor at Stanford Engineering.The researchers will present&nbsp;<a href="https://arxiv.org/abs/2405.18886">their new algorithm</a> CALDERA, which stands for Calibration Aware Low precision DEcomposition with low Rank Adaptation, at the Conference on Neural Information Processing Systems (NeurIPS) in December. Saha and colleagues began this compression research not with LLMs themselves, but with the large collections of information that are used to train LLMs and other complex AI models, such as those used for image classification. This technique, a forerunner to the new LLM compression approach, was&nbsp;<a href="https://openreview.net/forum?id=rxsCTtkqA9">published</a> in 2023.Training data sets and AI models are both composed of matrices, or grids of numbers that are used to store data. In the case of LLMs, these are called weight matrices, which are numerical representations of word patterns learned from large swaths of text.“We proposed a generic algorithm for compressing large data sets or large matrices,” said Saha. “And then we realized that nowadays, it’s not just the data sets that are large, but the models being deployed are also getting large. So, we could also use our algorithm to compress these models.”While the team’s algorithm is not the first to compress LLMs, its novelty lies in an innovative combination of two properties, one called “low-precision,” the other “low-rank.” As digital computers store and process information as bits (zeros and ones), “low-precision” representation reduces the number of bits, speeding up storage and processing while improving energy efficiency. On the other hand, “low-rank” refers to reducing redundancies in the LLM weight matrices.“Using both of these properties together, we are able to get much more compression than either of these techniques can achieve individually,” said Saha.The team tested their technique using Llama 2 and Llama 3, open-source large language models released by Meta AI, and found that their method, which used low-rank and low-precision components in tandem with each other, can be used to improve other methods which use just low-precision. The improvement can be up to 5%, which is significant for metrics that measure uncertainty in predicting word sequences.They evaluated the performance of the compressed language models using several sets of benchmark tasks for LLMs. The tasks included determining the logical order of two statements, or answering questions involving physical reasoning, such as how to separate an egg white from a yolk or how to make a cup of tea.“I think it’s encouraging and a bit surprising that we were able to get such good performance in this compression scheme,” said Goldsmith, who moved to Princeton from Stanford Engineering in 2020. “By taking advantage of the weight matrix rather than just using a generic compression algorithm for the bits that are representing the weight matrix, we were able to do much better.”Using an LLM compressed in this way could be suitable for situations that don’t require the highest possible precision. Moreover, the ability to fine-tune compressed LLMs on edge devices like a smartphone or laptop enhances privacy by allowing organizations and individuals to adapt models to their specific needs without sharing sensitive data with third-party providers. This reduces the risk of data breaches or unauthorized access to confidential information during the training process. To enable this, the LLMs must initially be compressed enough to fit on consumer-grade GPUs.Saha also cautioned that running LLMs on a smartphone or laptop could hog the device’s memory for a period of time. “You won’t be happy if you are running an LLM and your phone drains out of charge in an hour,” said Saha. Low-precision computation can help reduce power consumption, he added. “But I wouldn’t say that there’s one single technique that solves all the problems. What we propose in this paper is one technique that is used in combination with techniques proposed in prior works. And I think this combination will enable us to use LLMs on mobile devices more efficiently and get more accurate results.”The paper, “<a href="https://arxiv.org/abs/2405.18886">Compressing Large Language Models using Low Rank and Low Precision Decomposition</a>,” will be presented at the Conference on Neural Information Processing Systems (NeurIPS) in December 2024. In addition to Goldsmith, Saha and Pilanci, coauthors include Stanford Engineering researchers Naomi Sagan and Varun Srivastava. This work was supported in part by the U.S. National Science Foundation, the U.S. Army Research Office, and the Office of Naval Research.

Large language models (LLMs) are increasingly automating tasks like translation, text classification and customer service. But tapping into an LLM’s power typically requires users to send their requests to a centralized server — a process that’s expensive, energy-intensive and often slow.

machine learning

Profiles