Digital fabrication engineers are now working toward expanding the display capabilities of other everyday objects. One avenue they’re exploring is reprogrammable surfaces — or items whose appearances we can digitally alter — to help users present important information, such as health statistics, as well as new designs on things like a wall, mug, or shoe. Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), the University of California at Berkeley, and Aarhus University have taken an intriguing step forward by fabricating “<a href="https://hcie.csail.mit.edu/research/portachrome/portachrome.html">PortaChrome</a>,” a portable light system and design tool that can change the color and textures of various objects. Equipped with ultraviolet (UV) and red, green, and blue (RGB) LEDs, the device can be attached to everyday objects like shirts and headphones. Once a user creates a design and sends it to a PortaChrome machine via Bluetooth, the surface can be programmed into multicolor displays of health data, entertainment, and fashion designs.<iframe width="640" height="360" src="https://www.youtube.com/embed/eQcwTSWOzR8?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>To make an item reprogrammable, the object must be coated with photochromic dye, an invisible ink that can be turned into different colors with light patterns. Once it’s coated, individuals can create and relay patterns to the item via the team’s graphic design software, or use the team’s API to interact with the device directly and embed data-driven designs. When attached to a surface, PortaChrome’s UV lights saturate the dye while the RGB LEDs desaturate it, activating the colors and ensuring each pixel is toned to match the intended design.Zhu and her colleagues’ integrated light system changes objects’ colors in less than four minutes on average, which is eight times faster than their prior work, “Photo-Chromeleon.” This speed boost comes from switching to a light source that makes contact with the object to transmit UV and RGB rays. Photo-Chromeleon used a projector to help activate the color-changing properties of photochromic dye, where the light on the object's surface is at a reduced intensity.“PortaChrome provides a more convenient way to reprogram your surroundings,” says Yunyi Zhu ’20, MEng ’21, an MIT PhD student in electrical engineering and computer science, affiliate of CSAIL, and lead author on a paper about the work. “Compared with our projector-based system from before, PortaChrome is a more portable light source that can be placed directly on top of the photochromic surface. This allows the color change to happen without user intervention and helps us avoid contaminating our environment with UV. As a result, users can wear their heart rate chart on their shirt after a workout, for instance.”<h2>Giving everyday objects a makeover</h2>In demos, PortaChrome displayed health data on different surfaces. A user hiked with PortaChrome sewed onto their backpack, putting it into direct contact with the back of their shirt, which was coated in photochromic dye. Altitude and heart rate sensors sent data to the lighting device, which was then converted into a chart through a reprogramming script developed by the researchers. This process created a health visualization on the back of the user’s shirt. In a similar showing, MIT researchers displayed a heart gradually coming together on the back of a tablet to show how a user was progressing toward a fitness goal.PortaChrome also showed a flair for customizing wearables. For example, the researchers redesigned some white headphones with sideways blue lines and horizontal yellow and purple stripes. The photochromic dye was coated on the headphones and the team then attached the PortaChrome device to the inside of the headphone case. Finally, the researchers successfully reprogrammed their patterns onto the object, which resembled watercolor art. Researchers also recolored a wrist splint to match different clothes using this process.Eventually, the work could be used to digitize consumers’ belongings. Imagine putting on a cloak that can change your entire shirt design, or using your car cover to give your vehicle a new look.<h2>PortaChrome’s main ingredients</h2>On the hardware end, PortaChrome is a combination of four main ingredients. Their portable device consists of a textile base as a sort of backbone, a textile layer with the UV lights soldered on and another with the RGB stuck on, and a silicone diffusion layer to top it off. Resembling a translucent honeycomb, the silicone layer covers the interlaced UV and RGB LEDs and directs them toward individual pixels to properly illuminate a design over a surface.This device can be flexibly wrapped around objects with different shapes. For tables and other flat surfaces, you could place PortaChrome on top, like a placemat. For a curved item like a thermos, you could wrap the light source around like a coffee cup sleeve to ensure it reprograms the entire surface.The portable, flexible light system is crafted with maker space-available tools (like laser cutters, for example), and the same method can be replicated with flexible PCB materials and other mass manufacturing systems.While it can also quickly convert our surroundings into dynamic displays, Zhu and her colleagues believe it could benefit from further speed boosts. They'd like to use smaller LEDs, with the likely result being a surface that could be reprogrammed in seconds with a higher-resolution design, thanks to increased light intensity.“The surfaces of our everyday things are encoded with colors and visual textures, delivering crucial information and shaping how we interact with them,” says Georgia Tech postdoc Tingyu Cheng, who was not involved with the research. “PortaChrome is taking a leap forward by providing reprogrammable surfaces with the integration of flexible light sources (UV and RGB LEDs) and photochromic pigments into everyday objects, pixelating the environment with dynamic color and patterns. The capabilities demonstrated by PortaChrome could revolutionize the way we interact with our surroundings, particularly in domains like personalized fashion and adaptive user interfaces. This technology enables real-time customization that seamlessly integrates into daily life, offering a glimpse into the future of ‘ubiquitous displays.’”Zhu is joined by nine CSAIL affiliates on the paper: MIT PhD student and MIT Media Lab affiliate Cedric Honnet; former visiting undergraduate researchers Yixiao Kang, Angelina J. Zheng, and Grace Tang; MIT undergraduate student Luca Musk; University of Michigan Assistant Professor Junyi Zhu SM ’19, PhD ’24; recent postdoc and Aarhus University assistant professor Michael Wessely; and senior author Stefanie Mueller, the TIBCO Career Development Associate Professor in the MIT departments of Electrical Engineering and Computer Science and Mechanical Engineering and leader of the HCI Engineering Group at CSAIL.

When Nikola Tesla predicted we’d have handheld phones that could display videos, photographs, and more, his musings seemed like a distant dream. Nearly 100 years later, smartphones are like an extra appendage for many of us.

A portable light system that can digitize everyday objects

Thinking outside the box, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and Aarhus University researchers have now engineered “MouthIO,” a dental brace that can be fabricated with sensors and feedback components to capture in-mouth interactions and data. This interactive wearable could eventually assist dentists and other doctors with collecting health data and help motor-impaired individuals interact with a phone, computer, or fitness tracker using their mouths.Resembling an electronic retainer, MouthIO is a see-through brace that fits the specifications of your upper or lower set of teeth from a scan. The researchers created a plugin for the modeling software Blender to help users tailor the device to fit a dental scan, where you can then 3D print your design in dental resin. This computer-aided design tool allows users to digitally customize a panel (called PCB housing) on the side to integrate electronic components like batteries, sensors (including detectors for temperature and acceleration, as well as tongue-touch sensors), and actuators (like vibration motors and LEDs for feedback). You can also place small electronics outside of the PCB housing on individual teeth.<iframe width="640" height="360" src="https://www.youtube.com/embed/PsDgvZ_6vT4?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h2 dir="ltr" id="isPasted">The active mouth</h2>“The mouth is a really interesting place for an interactive wearable and can open up many opportunities, but has remained largely unexplored due to its complexity,” says senior author Michael Wessely, a former CSAIL postdoc and senior author on a&nbsp;<a href="https://dl.acm.org/doi/10.1145/3654777.3676443">paper about MouthIO</a> who is now an assistant professor at Aarhus University. “This compact, humid environment has elaborate geometries, making it hard to build a wearable interface to place inside. With MouthIO, though, we’ve developed a new kind of device that’s comfortable, safe, and almost invisible to others. Dentists and other doctors are eager about MouthIO for its potential to provide new health insights, tracking things like teeth grinding and potentially bacteria in your saliva.” The excitement for MouthIO’s potential in health monitoring stems from initial experiments. The team found that their device could track bruxism (the habit of grinding teeth) by embedding an accelerometer within the brace to track jaw movements. When attached to the lower set of teeth, MouthIO detected when users grind and bite, with the data charted to show how often users did each. Wessely and his colleagues’ customizable brace could one day help users with motor impairments, too. The team connected small touchpads to MouthIO, helping detect when a user’s tongue taps their teeth. These interactions could be sent via Bluetooth to scroll across a webpage, for example, allowing the tongue to act as a “third hand” to open up a new avenue for hands-free interaction."MouthIO is a great example how miniature electronics now allow us to integrate sensing into a broad range of everyday interactions,” says study co-author Stefanie Mueller, the TIBCO Career Development Associate Professor in the MIT departments of Electrical Engineering and Computer Science and Mechanical Engineering and leader of the HCI Engineering Group at CSAIL. “I'm especially excited about the potential to help improve accessibility and track potential health issues among users."<h2 dir="ltr">Molding and making MouthIO</h2>To get a 3D model of your teeth, you can first create a physical impression and fill it with plaster. You can then scan your mold with a mobile app like Polycam and upload that to Blender. Using the researchers’ plugin within this program, you can clean up your dental scan to outline a precise brace design. Finally, you 3D print your digital creation in clear dental resin, where the electronic components can then be soldered on. Users can create a standard brace that covers their teeth, or opt for an “open-bite” design within their Blender plugin. The latter fits more like open-finger gloves, exposing the tips of your teeth, which helps users avoid lisping and talk naturally.This “do it yourself” method costs roughly $15 to produce and takes two hours to be 3D-printed. MouthIO can also be fabricated with a more expensive, professional-level teeth scanner similar to what dentists and orthodontists use, which is faster and less labor-intensive.Compared to its closed counterpart, which fully covers your teeth, the researchers view the open-bite design as a more comfortable option. The team preferred to use it for beverage monitoring experiments, where they fabricated a brace capable of alerting users when a drink was too hot. This iteration of MouthIO had a temperature sensor and a monitor embedded within the PCB housing that vibrated when a drink exceeded 65 degrees Celsius (or 149 degrees Fahrenheit). This could help individuals with mouth numbness better understand what they’re consuming. In a user study, participants also preferred the open-bite version of MouthIO. “We found that our device could be suitable for everyday use in the future,” says study lead author and Aarhus University PhD student Yijing Jiang. “Since the tongue can touch the front teeth in our open-bite design, users don’t have a lisp. This made users feel more comfortable wearing the device during extended periods with breaks, similar to how people use retainers.” The team’s initial findings indicate that MouthIO is a cost-effective, accessible, and customizable interface, and the team is working on a more long-term study to evaluate its viability further. They’re looking to improve its design, including experimenting with more flexible materials, and placing it in other parts of the mouth, like the cheek and the palate. Among these ideas, the researchers have already prototyped two new designs for MouthIO: a single-sided brace for even higher comfort when wearing MouthIO while also being fully invisible to others, and another fully capable of wireless charging and communication. Jiang, Mueller, and Wessely’s co-authors include PhD student Julia Kleinau, master’s student Till Max Eckroth, and associate professor Eve Hoggan, all of Aarhus University. Their work was supported by a Novo Nordisk Foundation grant and was presented at ACM’s Symposium on User Interface Software and Technology.

When you think about hands-free devices, you might picture Alexa and other voice-activated in-home assistants, Bluetooth earpieces, or asking Siri to make a phone call in your car. You might not imagine using your mouth to communicate with other devices like a computer or a phone remotely. 

Interactive mouthpiece opens new opportunities for health data, assistive technology, and hands-free interactions

The prestigious <a href="https://www.merckgroup.com/en/research/grants-and-awards/heinrich-emanuel-merck-award.html" target="_blank">Heinrich Emanuel Merck Award</a> is a biannial recognition of individuals who have made outstanding contributions to the advancement of artificial intelligence, quantum computing, neuro-inspired computing, and their applications to enable scientific discovery. The recipient of the award will be given a €15,000 prize and will be invited to present their research at an award ceremony.&nbsp;The Heinrich Emanuel Merck Award is given in memory of the company’s founder, Heinrich Emanuel Merck, and has been a recognition of excellence and innovation in research for over two decades. Originally established to recognize research in the analytical sciences and presented as such until 2022, the award has undergone a significant transformation. As of 2024, the focus of the award has shifted to computational science.&nbsp;Describing the significance of this award, Dr. Ulrich Betz, Senior Vice President Innovation at Merck said:<blockquote>“Computational sciences are increasingly making contributions to enable new scientific discoveries in all major fields. With the new Heinrich Emanuel Merck Award we want to honor scientists that enhance and apply these modern digital technologies to enhance scientific discovery.”</blockquote><h2 dir="ltr">Who is eligible to apply?</h2>Previous award laureates include top researchers whose work has provided the foundation for groundbreaking discoveries. Are you or do you know someone who is developing innovative solutions in domains at the forefront of innovation in computing like artificial intelligence, quantum computing, or neuro-inspired computing? Visit the application portal and submit your application!<a href="https://app.bitly.com/Bi6chcjWpnI/links?query=https://wevlv.co/HEMAportal" target="_blank" rel="noopener noreferrer" id="isPasted" class="button-main-action">TO THE APPLICATION PORTAL</a><table style="width: 100%;"><tbody><tr><td style="width: 100%;"><h3>Key Dates</h3>When to apply:&nbsp;Applications for the award are open. <a href="https://app.bitly.com/Bi6chcjWpnI/links?query=https://wevlv.co/HEMAportal" target="_blank" rel="noopener noreferrer">Visit the application portal</a> to read more and apply.&nbsp;Application closes: October 31stAward ceremony: Beginning of 2025</td></tr></tbody></table><h2 dir="ltr" id="isPasted">Key technology areas in computational sciences</h2>The shift in focus of the award to computational sciences aims to highlight groundbreaking research in fields such as artificial intelligence (AI), quantum computing, and neuro-inspired computing to transform industries, drive innovation, and solve global challenges. By celebrating advances in these areas, the award recognizes work that will enable scientific discovery.<h4 dir="ltr">Artificial Intelligence (AI):</h4>AI is in a phase of rapid advancement, particularly in machine learning, natural language processing, and computer vision. Potential areas of interest for innovative research include <a href="https://www.wevolver.com/article/shaping-the-future-of-manufacturing-with-llms-and-generative-ai" target="_blank">Smart manufacturing</a>, <a href="https://www.wevolver.com/article/internet-of-things-for-smart-cities" target="_blank">IoT for Smart Cities</a>, and <a href="https://www.wevolver.com/article/smart-grid-navigating-the-future-of-energy" target="_blank">leveraging AI for energy efficiency</a>.<h4 dir="ltr">Quantum Computing:</h4>Quantum computing has made significant strides, moving from theoretical models to experimental demonstrations of quantum supremacy. Emerging fields of interest for cutting-edge research include <a href="https://www.wevolver.com/article/modular-scalable-hardware-architecture-for-a-quantum-computer" target="_blank">scalable hardware architecture</a>, <a href="https://www.wevolver.com/article/scientists-tune-the-entanglement-structure-in-an-array-of-qubits" target="_blank">entanglement structure of quantum bits</a> and <a href="https://www.wevolver.com/article/breakthroughs-in-quantum-computing" target="_blank">processor developments</a>.&nbsp;Neuro-Inspired Computing:Neuro-inspired computing seeks to replicate the efficiency and structure of the human brain in artificial systems. Current research within this area include <a href="https://www.wevolver.com/article/learning-how-to-learn-neuromorphic-ai-inference-at-the-edge" target="_blank">neuromorphic AI at the edge</a>, <a href="https://www.wevolver.com/article/analyzing-human-behavior-with-edge-ai" target="_blank">human behavior analysis</a> and <a href="https://www.wevolver.com/article/advancing-earth-management-through-state-of-the-art-satellite-infrastructure" target="_blank">autonomous robots for remote maintenance</a>.<h2 dir="ltr">How to apply:</h2><ol><li dir="ltr"><a href="https://app.bitly.com/Bi6chcjWpnI/links?query=https://wevlv.co/HEMAportal" target="_blank" rel="noopener noreferrer">Visit the application portal</a> and follow the instructions to create an account and submit your application.</li><li dir="ltr">Application documents: Documents needed to apply include a recent CV and a clear description of your cutting-edge research and results (no older than 3 years).</li><li dir="ltr">Deadline: Last day for applications is October 31st.</li></ol>Visit the application portal to read more and submit your application:<a href="https://app.bitly.com/Bi6chcjWpnI/links?query=https://wevlv.co/HEMAportal" target="_blank" rel="noopener noreferrer" id="isPasted" class="button-main-action">TO THE APPLICATION PORTAL</a>

Recognizing Groundbreaking Research in AI, Quantum, and Neuro-Inspired Computing

Honoring Innovation in Computational Sciences: Apply for The Heinrich Emanuel Merck Award

In this episode, we chat with Karsten Roscher - a renowned AI expert at the Fraunhofer Institute for Cognitive Systems IKS - about the importance of safety in AI products, the current challenges facing experts, and what the future holds.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/2a32914a-cb70-4e6b-83c9-39693967722c?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><hr><h3 id="isPasted">Episode Notes</h3>(2:00) - Karsten’s journey from undergraduate research to Department Head at Fraunhofer Institute for Cognitive Systems IKS(7:40) - Why Karsten &amp; his team are fixated on AI safety&nbsp;(10:40) - How the team impacts policy decisions(17:00) - The ways the average person can detect the outcomes of the team’s work(18:56) - How to detect what an AI doesn’t know &amp; why it’s so hard(28:40) - Robuscope: Fraunhofer’s free AI testing tool(34:46) - Why being benchmark driven can be bad for AI development(41:10) - How comfortable Karsten feels about using Autonomous vehicles &amp; AI in general(49:55) - How to balance safety and progress with AI progress(56:04) - Karsten &amp; SAE International’s machine learning &amp; safety course(58:30) - Karsten’s unintentional road to becoming a teacher &nbsp;This episode is part of a series brought to you by&nbsp;<a href="https://www.sae.org/" target="_blank">SAE International</a> to explore the leading edge of mobility with the support of experts from industry &amp; academia. Other episodes in this series include&nbsp;<a href="https://pod.link/wevolver/episode/a9e88f02f379c03548ffdbedbb563429" target="_blank">Episode 190</a>, where we talked about Karsten’s free AI evaluator tool, Robuscope!You can check out the information about the course Karsten mentioned for safety &amp; machine learning by clicking&nbsp;<a href="https://www.wevolver.com/article/mastering-machine-learning-for-critical-applications" target="_blank">HERE</a>.&nbsp;Enjoyed listening to Karsten? Want to hear more from him? Checkout his links below:<ul><li><a href="https://www.linkedin.com/in/karsten-roscher-17a2907/" target="_blank">Karsten’s LinkedIn</a></li><li><a href="https://safe-intelligence.fraunhofer.de/en/" target="_blank">Safe Intelligence Blog</a></li></ul>Become a founding reader of our newsletter:&nbsp;<a href="http://read.thenextbyte.com/" target="_blank">http://read.thenextbyte.com/</a>&nbsp;As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.

In this episode, we chat with Karsten Roscher - a renowned AI expert at the Fraunhofer Institute for Cognitive Systems IKS - about the importance of safety in AI products, the current challenges facing experts, and what the future holds. 

Podcast: Karsten Roscher & The Need For Safe AI (Interview)

Large language models (LLMs) such as GPT-4 have become indispensable for tasks like text generation, coding, operating chatbots, translation and others. At their heart, LLMs work by predicting the next word in a sentence based on the previous words – a simple but powerful idea that drives much of their functionality. But what happens when we ask these models to predict backward — to go “backwards in time” and determine the previous word from the subsequent ones?The question led&nbsp;<a href="https://www.epfl.ch/labs/csft/" target="_blank">Professor Clément Hongler</a> at EPFL and&nbsp;<a href="https://jeremiewenger.com/" target="_blank" rel="noopener">Jérémie Wenger</a> of Goldsmiths (London) to explore whether LLMs could construct a story backward, starting from the end. Working with Vassilis Papadopoulos, a machine learning researcher at EPFL, they discovered something surprising: LLMs are consistently less accurate when predicting backward than forward.<h2>A fundamental asymmetry</h2>The researchers tested LLMs of different architectures and sizes, including Generative Pre-trained Transformers (GPT), Gated Recurrent Units (GRU), and Long Short-Term Memory (LSTM) neural networks. Every one of them showed the “Arrow of Time” bias, revealing a fundamental asymmetry in how LLMs process text.Hongler explains: “The discovery shows that while LLMs are quite good both at predicting the next word and the previous word in a text, they are always slightly worse backwards rather than forward: their performance at predicting the previous word is always a few percent worse than at predicting the next word. This phenomenon is universal across languages, and can be observed with any large language model.”The work is also connected to the work of Claude Shannon, the father of Information Theory, in his seminal&nbsp;<a href="https://ieeexplore.ieee.org/document/6773263" target="_blank" rel="noopener">1951 paper</a>. Shannon explored whether predicting the next letter in a sequence was as easy as predicting the previous one. He discovered that although both tasks should theoretically be equally difficult, humans found backward prediction more challenging – though the performance difference was minimal.<h2>Intelligent agents</h2>“In theory, there should be no difference between the forward and backward directions, but LLMs appear to be somehow sensitive to the time direction in which they process text,” says Hongler. “Interestingly, this is related to a deep property of the structure of language that could only be discovered with the emergence of Large Language Models in the last five years.”The researchers link this property to the presence of intelligent agents processing information, meaning that it could be used as a tool to detect intelligence or life, and help design more powerful LLMs. Finally, it could point out new directions to the long-standing quest to understand the passage of time as an emergent phenomenon in physics.The work was presented at the prestigious&nbsp;<a href="https://icml.cc/" target="_blank" rel="noopener">International Conference on Machine Learning</a> (2024) and&nbsp;<a href="https://arxiv.org/html/2401.17505v4" target="_blank" rel="noopener">is currently available on arXiv</a>.<h2>From theater to math</h2>The study itself has a fascinating backstory, which Hongler relates: “In 2020, with Jérémie [Wenger], we were collaborating with&nbsp;<a href="https://www.manufacture.ch/en" target="_blank" rel="noopener">The Manufacture theater school</a> to make a chatbot that would play alongside actors to do improv; in improv, you often want to continue the story, while knowing what the end should look like. “In order to make stories that would finish in a specific manner, we got the idea to train the chatbot to speak ‘backwards’, allowing it to generate a story given its end – e.g., if the end is ‘they lived happily ever after’, the model could tell you how it happened. So, we trained models to do that, and noticed they were a little worse backwards than forwards. “With Vassilis [Papadopoulos], we later realized that this was a profound feature of language, and that it was a completely general new phenomenon, which has deep links with the passage of time, intelligence, and the notion of causality. Quite cool for some theater project!” Hongler’s excitement with this work stems in good part from the unexpected surprises that came along the way: “Only time could tell that something that started as a theater project would end up giving us new tools to understand so many things about the world.”<hr>ReferencesVassilis Papadopoulos, Jérémie Wenger, Clément Hongler. Arrows of Time for Large Language Models. arXiv:&nbsp;<a href="https://arxiv.org/html/2401.17505v4" target="_blank" rel="noopener">2401.17505v4</a>

Researchers have found that AI large language models, like GPT-4, are better at predicting what comes next than what came before in a sentence. This “Arrow of Time” effect could reshape our understanding of the structure of natural language, and the way these models understand it.

Large Language Models feel the direction of time

In a constantly changing world – saturated with a multitude of impressions – it can sometimes be difficult to maintain an overview. After all, all impressions not only have to be perceived, but also interpreted, which ultimately opens up a wide range of options for action. This is where LUMINOUS (Language Augmentation for Humanverse) system comes into play. The DFKI-technology collects countless impressions, interprets them and can suggest an appropriate action using MLLM.<blockquote id="isPasted">"Through our developed technology, virtual worlds become increasingly intelligent. The intuitive interaction (via text) with the system and automatic generation of complex behaviors and processes through 'generative AI' or so-called 'Multi-Modal Large Language Models' enable us not only to experience them, but also to test them. To achieve this, at project LUMINOUS we are working in parallel on several approaches such as automatic code generation, the rapid input of new data and other solutions."<footer>Didier Stricker, Lead "Augmented Vision" at DFKI</footer></blockquote><header><h2>System observes, interprets – and refines itself</h2></header>In the new LUMINOUS project, DFKI is working on next-generation augmented reality (XR) systems. In the future, generative and multimodal language models (MLLM) will join the existing technical extensions of our visually perceived reality, such as the form of texts, animations or the insertion of virtual objects, and redefine interaction with augmented reality (XR) technology.Muhammad Zeshan Afzal, researcher in the department 'Augmented Vision' at DFKI explains, how this might look like in practise:„Let’s take the following scenario: a fire breaks out in a room. In this case, our system first determines where the person – who is equipped with our technology – is currently located. It then collects relevant data from their immediate surroundings, such as the presence of a fire extinguisher or an emergency exit and passes this on to the generative and multimodal language model. This then determines a suitable recommendation for action, such as initiating the extinguishing process using a fire extinguisher, closing windows, or getting to safety.”<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI2NTMwNzIzODgwLTEyNDIxNC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib"><header id="isPasted"><h2>Learning from descriptions creates flexibility</h2></header>Until now, R&amp;D endeavours have been largely limited to the spatial tracking of users and their environment. The result: very specific, limited, and non-generalisable representations, as well as predefined graphic visualizations and animations. „Language Augmentation for Humanverse“ aims to change this in the future.To achieve this, the researchers at DFKI are developing a platform with language support that adapts to individual, non-predefined user needs and previously unknown augmented reality environments. The adaptable concept originates from zero-shot learning (ZSL), an AI system that is trained to recognize and categorize objects and scenarios – without having seen exemplary reference material in advance.The assistance helps LUMINOUS to learn and constantly expand its knowledge – beyond the pure training data. By making names and text descriptions available to the LLM, LUMINOUS will use its database of image descriptions to build up a flexible image and text vocabulary that makes it possible to recognize even unknown objects or scenes.<header><h2>Interactive support in real time</h2><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1726530985514-vinicius-amnx-amano-XOnbUXqILzA-unsplash.jpg" style="width: 100%px;" class="fr-fic fr-dib">Photo by Vinicius "amnx" Amano on UnsplashThe areas of application for augmented reality systems such as LUMINOUS go beyond situational assistance. „We are currently investigating possible applications for everyday care of sick people, implementation of training programmes, performance monitoring and motivation”, says Zeshan Afzal.As a kind of translator, the LLM from LUMINOUS should be able to describe everyday activities on command and mediate them to users via a voice interface or avatar. The visual assistance and recommended actions provided in this way will then support everyday activities in real time.<h2>LUMINOUS in application</h2>The results of the project are being tested in three pilot projects focussing on neurorehabilitation (support for stroke patients with speech disorders), immersive safety training in the workplace and the testing of 3D architectural designs.In the case of neurorehabilitation of stroke patients with severe communication deficits (aphasia), realistic virtual characters (avatars) support the initiation of conversation through image-directed models. These are based on natural language and enable generalization to other activities of daily life. Objects in the scene (including people) are recognized in real time using eye-tracking and object recognition algorithms. Patients can then ask the avatar or MLLM to articulate either the name of the object or the first phoneme. To use the language models in the patient’s respective environment, patients undergo personalised and intensive XR-supported training.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1726531264095-amr-taha-JpmhbbPhuZA-unsplash.jpg" style="width: 100%px;" class="fr-fic fr-dib">Photo by Amr Taha™ on UnsplashThe LUMINOUS-Tech captures the movements and style of the human trainer with a minimal number of sensors to enable the modelling and instantiation of three-dimensional avatars. The aim is to use only kinematic information derived exclusively from the input of the headset, the position of the head and the hands during training.Future users of these new XR-systems will be able to interact seamlessly with their environment by using language models while having access to constantly updated global and domain-specific knowledge sources. In this way, new XR-technologies can be used for distance learning and education, entertainment, or healthcare services, for example.<hr><header id="isPasted"><h2>Partners:</h2></header><ul><li>German Research Centre for Artificial Intelligence GmbH (DFKI)</li><li>Ludus Tech SL</li><li>Mindesk Societa a Responsabilita Limita</li><li>Fraunhofer Society for the Advancement of Applied Research</li><li>Universidad del Pais Vasco/Euskal Herriko Universitatea</li><li>Fundación Centro de Tecnologias de Interacción visual y Comunicaciones Vicomtech</li><li>University College Dublin</li><li>National University of Ireland</li><li>Hypercliq IKE</li><li>Ricoh International B.V. – Brach Office Germany</li><li>MindMaze SA</li><li>Centre Hospitalier Universitaire Vaudois</li><li>University College London</li></ul>The project is subsidised by the European Union. Project LUMINOUS (Language Augmentation for Humanverse) does in no way shape or form relate to the AI Language Model Luminous, developed by Aleph Alpha.</header>

The DFKI-technology collects countless impressions, interprets them and can suggest an appropriate action using MLLM.

Project Luminous: The next level of Augmented Reality

On a research cruise around Hawaii in 2018, Yuening Zhang SM ’19, PhD ’24 saw how difficult it was to keep a tight ship. The careful coordination required to map underwater terrain could sometimes led to a stressful environment for team members, who might have different understandings of which tasks must be completed in spontaneously changing conditions. During these trips, Zhang considered how a robotic companion could have helped her and her crewmates achieve their goals more efficiently. Six years later, as a research assistant in the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), Zhang developed what could be considered a missing piece: an AI assistant that communicates with team members to align roles and accomplish a common goal. In a paper presented at the International Conference on Robotics and Automation (ICRA) and&nbsp;<a href="https://ieeexplore.ieee.org/document/10609865" target="_blank">published on IEEE Xplore on Aug. 8</a>, she and her colleagues present a system that can oversee a team of both human and AI agents, intervening when needed to potentially increase teamwork effectiveness in domains like search-and-rescue missions, medical procedures, and strategy video games. The CSAIL-led group has developed a theory of mind model for AI agents, which represents how humans think and understand each other’s possible plan of action when they cooperate in a task. By observing the actions of its fellow agents, this new team coordinator can infer their plans and their understanding of each other from a prior set of beliefs. When their plans are incompatible, the AI helper intervenes by aligning their beliefs about each other, instructing their actions, as well as asking questions when needed. For example, when a team of rescue workers is out in the field to triage victims, they must make decisions based on their beliefs about each other’s roles and progress. This type of epistemic planning could be improved by CSAIL’s software, which can send messages about what each agent intends to do or has done to ensure task completion and avoid duplicate efforts. In this instance, the AI helper may intervene to communicate that an agent has already proceeded to a certain room, or that none of the agents are covering a certain area with potential victims. “Our work takes into account the sentiment that ‘I believe that you believe what someone else believes,’” says Zhang, who is now a research scientist at Mobi Systems. “Imagine you’re working on a team and you ask yourself, ‘What exactly is that person doing? What am I going to do? Does he know what I am about to do?’ We model how different team members understand the overarching plan and communicate what they need to accomplish to help complete their team’s overall goal.” AI to the rescue Even with a sophisticated plan, both human and robotic agents will encounter confusion and even make mistakes if their roles are unclear. This plight looms especially large in search-and-rescue missions, where the objective may be to locate someone in danger despite limited time and a vast area to scan. Thankfully, communication technology augmented with the new robotic assistant could potentially notify the search parties about what each group is doing and where they’re looking. In turn, the agents could navigate their terrain more efficiently.This type of task organization could aid in other high-stakes scenarios like surgeries. In these cases, the nurse first needs to bring the patient to the operation room, then the anesthesiologist puts the patient to sleep before the surgeons begin the operation. Throughout the operation, the team must continuously monitor the patient’s condition while dynamically responding to the actions of each colleague. To ensure that each activity within the procedure remains well-organized, the AI team coordinator could oversee and intervene if confusion about any of these tasks arises.Effective teamwork is also integral to video games like “Valorant,” where players collaboratively coordinate who needs to attack and defend against another team online. In these scenarios, an AI assistant could pop up on the screen to alert individual users about where they’ve misinterpreted which tasks they need to complete.Before she led the development of this model, Zhang designed EPike, a computational model that can act as a team member. In a 3D simulation program, this algorithm controlled a robotic agent that needed to match a container to the drink chosen by the human. As rational and sophisticated as they may be, cases arise where these AI-simulated bots are limited by their misconceptions about their human partners or the task. The new AI coordinator can correct the agents’ beliefs when needed to resolve potential problems, and it consistently intervened in this instance. The system sent messages to the robot about the human’s true intentions to ensure it matched the container correctly.“In our work on human-robot collaboration, we’ve been both humbled and inspired over the years by how fluid human partners can be,” says Brian C. Williams, MIT professor of aeronautics and astronautics, CSAIL member, and senior author on the study. “Just look at a young couple with kids, who work together to get their kids breakfast and off to school. If one parent sees their partner serving breakfast and still in their bathrobe, the parent knows to shower quickly and shuffle the kids off to school, without the need to say a word. Good partners are well in tune with the beliefs and goals of each other, and our work on epistemic planning strives to capture this style of reasoning.”The researchers' method incorporates probabilistic reasoning with recursive mental modeling of the agents, allowing the AI assistant to make risk-bounded decisions. In addition, they focused on modeling agents’ understanding of plans and actions, which could complement previous work on modeling beliefs about the current world or environment. The AI assistant currently infers agents’ beliefs based on a given prior of possible beliefs, but the MIT group envisions applying machine learning techniques to generate new hypotheses on the fly. To apply this counterpart to real-life tasks, they also aim to consider richer plan representations in their work and reduce computation costs further.Dynamic Object Language Labs President Paul Robertson, Johns Hopkins University Assistant Professor Tianmin Shu, and former CSAIL affiliate Sungkweon Hong PhD ’23 join Zhang and Williams on the paper. Their work was supported, in part, by the U.S. Defense Advanced Research Projects Agency (DARPA) Artificial Social Intelligence for Successful Teams (ASIST) program.

A system developed by MIT CSAIL researchers can oversee a team of both human and AI agents, communicating with team members to align roles and accomplish a common goal. Image: Alex Shipps/MIT CSAIL

An AI team coordinator aligns agents’ beliefs about how to achieve a task, intervening when necessary to potentially help with tasks in search and rescue, hospitals, and video games.

AI assistant monitors teamwork to promote effective collaboration

Automatic speech recognition and translation systems such as the Lecture Translator of the Karlsruhe Institute of Technology (KIT) can convert the spoken word in lectures into a text in several languages in real time. Such systems improve access to information for students with disabilities and foreign students. They also speed up work and learning in general through intelligent post-processing and archiving of spoken texts. KIT researchers have added new functions to the Lecture Translator. Automatic recognition of the spoken word in several languages at the same time, text segmentation and title generation in real time, summaries and links to technical terms as well as queries on what has been heard now make it easier to understand and efficiently process lectures."With Lecture Translator's automatic simultaneous translation, we have brought spoken lectures closer to an international audience. However, this usually represents only 15 percent of the audience. With the new AI tools, we want to break down not only language barriers, but also comprehension barriers," says Alexander Waibel, Professor of Computer Science at KIT. "Automatically transcribed texts of spoken language are often difficult to read, because they appear too quickly as a long text without paragraphs and subheadings - just as the lecture was delivered orally". Processing the lecture is also time-consuming, because you have to search the lecture for gaps in understanding," says Waibel.<h2>Better overview of documents</h2>The further development of the Lecture Translator provides a remedy. The researchers have developed several new automatic functions such as "Smart Chaptering", "Summarization", "Q&amp;A" and "Auto-Links". A new type of artificial intelligence (AI) that automatically recognizes language converts the spoken text into a transcript in multiple languages and automatically identifies paragraphs, chapter headings and key points. It also creates an acoustic rendition in which the user can select one of 18 languages. The program also automatically displays links as cross-references to relevant sources in lecture notes or Wikipedia, which students can use to better process the lecture. "With our new AI models, conversations and lectures can be better structured and even videos can be divided into easily navigable chapters," says Waibel. This enables better understanding not only during the lecture, but also after the lecture.<h2>Lecture Translator translates into 18 languages</h2>The research team has integrated the work into the Lecture Translator, which is used at KIT to automatically transcribe lectures in real time. Chapter division, title generation, paragraph layout, summaries with links - which can also be used online and offline - now extend the Lecture Translator service and make it easier to work with the material. Translation is available in 18 languages. The technology has specific applications for content creators, students, teachers and podcasters who can structure their audio and video content for the first time. "Users can navigate through videos and lectures more efficiently, find relevant sections more quickly, and capture important core content more compactly and efficiently - they have a much better overall view and faster access to details," says Waibel.The research was carried out as part of the project "How is AI Changing Science?" and was funded for four years by the Volkswagen Foundation. In addition to KIT, the University of Bonn and the University of Vienna were also involved in the project.<a href="https://www.kcist.kit.edu/" target="_blank">Details on the KIT Center Information - Systems - Technologies</a>

With new functions such as automatic chapter division and summaries, the Lecture Translator offers improved speech recognition. (Photo: Markus Breig, KIT)

IT Researchers Further Develop Intelligent Speech Recognition and Translation System: Automatic Text Segmentation, Summarization, and Intelligent Query Possible

Automatic Speech Recognition and Translation: Faster Working and Learning

A new cross-disciplinary study by WashU researchers has uncovered an unexpected psychological phenomenon at the intersection of human behavior and artificial intelligence: When told they were training AI to play a bargaining game, participants actively adjusted their own behavior to appear more fair and just, an impulse with potentially important implications for real-world AI developers.“The participants seemed to have a motivation to train AI for fairness, which is encouraging, but other people might have different agendas,” said Lauren Treiman, a PhD student in the Division of Computational &amp; Data Sciences (DCDS) and lead author of the study. “Developers should know that people will intentionally change their behavior when they know it will be used to train AI.”The study, published in PNAS, was supported by a seed grant from the&nbsp;<a href="https://triads.wustl.edu/" target="_blank">Transdisciplinary Institute in Applied Data Sciences (TRIADS)</a>, a signature initiative of the Arts &amp; Sciences Strategic Plan. The co-authors are&nbsp;<a href="https://psych.wustl.edu/people/wouter-kool" target="_blank">Wouter Kool</a>, assistant professor of psychological and brain sciences in Arts &amp; Sciences, and&nbsp;<a href="https://engineering.wustl.edu/faculty/Chien-Ju-Ho.html" target="_blank">Chien-Ju Ho</a>, assistant professor of computer science and engineering in the McKelvey School of Engineering. Kool and Ho are Treiman’s graduate advisors.The study included five experiments, each with roughly 200-300 participants. Subjects were asked to play the “Ultimatum Game,” a challenge that requires them to negotiate small cash payouts (just $1 to $6) with other human players or a computer. In some cases, they were told their decisions would be used to teach an AI bot how to play the game.The players who thought they were training AI were consistently more likely to seek a fair share of the payout, even if such fairness cost them a few bucks. Interestingly, that behavior change persisted even after they were told their decisions were no longer being used to train AI, suggesting the experience of shaping technology had a lasting impact on decision-making. &nbsp;“As cognitive scientists, we’re interested in habit formation,” Kool said. “This is a cool example because the behavior continued even when it was not called for anymore.”Still, the impulse behind the behavior isn’t entirely clear. Researchers didn’t ask about specific motivations and strategies, and Kool explained that participants may not have felt a strong obligation to make AI more ethical. It’s possible, he said, that the experiment simply brought out their natural tendencies to reject offers that seemed unfair. “They may not really be thinking about the future consequences,” he said. “They could just be taking the easy way out.”&nbsp;“The study underscores the important human element in the training of AI,” said Ho, a computer scientist who studies the relationships between human behaviors and machine learning algorithms. “A lot of AI training is based on human decisions,” he said. “If human biases during AI training aren’t taken into account, the resulting AI will also be biased. In the last few years, we’ve seen a lot of issues arising from this sort of mismatch between AI training and deployment.”Some facial recognition software, for example, is less accurate at identifying people of color, Ho said. “That’s partly because the data used to train AI is biased and unrepresentative,” he said.Treiman is now conducting follow-up experiments to get a better sense of the motivations and strategies of people training AI. “It’s very important to consider the psychological aspects of computer science,” she said.&nbsp;<hr>Treiman LS, Ho CJ, Kool W. The consequences of AI training on human decision-making. Proceedings of the National Academy of Sciences (PNAS) Aug. 6, 2024.DOI:&nbsp;<a href="https://doi.org/10.1073/pnas.2408731121" target="_blank">https://doi.org/10.1073/pnas.2408731121</a>&nbsp;

WashU researchers have found humans adjust their own behavior to be more fair when training AI.

McKelvey’s Chien-Ju Ho working with Art & Sciences’ Wouter Kool, DCDS PhD student Lauren Treiman to understand how human behavior changes in training AI

Humans change their own behavior when training AI

A team of computer scientists, engineers, mathematicians and cognitive scientists, led by the University of Cambridge, developed an open-source evaluation platform called CheckMate, which allows human users to interact with and evaluate the performance of large language models (LLMs).The researchers tested CheckMate in an experiment where human participants used three LLMs – InstructGPT, ChatGPT and GPT-4 – as assistants for solving undergraduate-level mathematics problems.The team studied how well LLMs can assist participants in solving problems. Despite a generally positive correlation between a chatbot’s correctness and perceived helpfulness, the researchers also found instances where the LLMs were incorrect, but still useful for the participants. However, certain incorrect LLM outputs were thought to be correct by participants. This was most notable in LLMs optimised for chat.The researchers suggest models that communicate uncertainty, respond well to user corrections, and can provide a concise rationale for their recommendations, make better assistants. Human users of LLMs should verify their outputs carefully, given their current shortcomings.The&nbsp;<a href="https://www.pnas.org/doi/10.1073/pnas.2318124121" target="_blank">results</a>, reported in the&nbsp;Proceedings of the National Academy of Sciences (PNAS), could be useful in both informing AI literacy training, and help developers improve LLMs for a wider range of uses.While LLMs are becoming increasingly powerful, they can also make mistakes and provide incorrect information, which could have negative consequences as these systems become more integrated into our everyday lives.“LLMs have become wildly popular, and evaluating their performance in a quantitative way is important, but we also need to evaluate how well these systems work with and can support people,” said co-first author Albert Jiang, from Cambridge’s Department of Computer Science and Technology. “We don’t yet have comprehensive ways of evaluating an LLM’s performance when interacting with humans.”The standard way to evaluate LLMs relies on static pairs of inputs and outputs, which disregards the interactive nature of chatbots, and how that changes their usefulness in different scenarios. The researchers developed CheckMate to help answer these questions, designed for but not limited to applications in mathematics.“When talking to mathematicians about LLMs, many of them fall into one of two main camps: either they think that LLMs can produce complex mathematical proofs on their own, or that LLMs are incapable of simple arithmetic,” said co-first author&nbsp;<a href="https://www.eng.cam.ac.uk/profiles/kmc61" target="_blank">Katie Collins</a>, PhD student&nbsp;from the Department of Engineering. “Of course, the truth is probably somewhere in between, but we wanted to find a way of evaluating which tasks LLMs are suitable for and which they aren’t.”The researchers recruited 25 mathematicians, from undergraduate students to senior professors, to interact with three different LLMs (InstructGPT, ChatGPT, and GPT-4) and evaluate their performance using CheckMate. Participants worked through undergraduate-level mathematical theorems with the assistance of an LLM and were asked to rate each individual LLM response for correctness and helpfulness. Participants did not know which LLM they were interacting with.The researchers recorded the sorts of questions asked by participants, how participants reacted when they were presented with a fully or partially incorrect answer, whether and how they attempted to correct the LLM, or if they asked for clarification. Participants had varying levels of experience with writing effective prompts for LLMs, and this often affected the quality of responses that the LLMs provided.An example of an effective prompt is “what is the definition of X” (X being a concept in the problem) as chatbots can be very good at retrieving concepts they know of and explaining it to the user.“One of the things we found is the surprising fallibility of these models,” said Katie Collins. “Sometimes, these LLMs will be really good at higher-level mathematics, and then they’ll fail at something far simpler. It shows that it’s vital to think carefully about how to use LLMs effectively and appropriately.”However, like the LLMs, the human participants also made mistakes. The researchers asked participants to rate how confident they were in their own ability to solve the problem they were using the LLM for. In cases where the participant was less confident in their own abilities, they were more likely to rate incorrect generations by LLM as correct.“This kind of gets to a big challenge of evaluating LLMs, because they’re getting so good at generating nice, seemingly correct natural language, that it’s easy to be fooled by their responses,” said Albert Jiang. “It also shows that while human evaluation is useful and important, it’s nuanced, and sometimes it’s wrong. Anyone using an LLM, for any application, should always pay attention to the output and verify it themselves.”Based on the results from CheckMate, the researchers say that newer generations of LLMs are increasingly able to collaborate helpfully and correctly with human users on undergraduate-level maths problems, as long as the user can assess the correctness of LLM-generated responses. Even if the answers may be memorised and can be found somewhere on the internet, LLMs have the advantage of being flexible in their inputs and outputs over traditional search engines (though should not replace search engines in their current form).While CheckMate was tested on mathematical problems, the researchers say their platform could be adapted to a wide range of fields. In the future, this type of feedback could be incorporated into the LLMs themselves, although none of the CheckMate feedback from the current study has been fed back into the models.“These kinds of tools can help the research community to have a better understanding of the strengths and weaknesses of these models,” said Katie Collins. “We wouldn’t use them as tools to solve complex mathematical problems on their own, but they can be useful assistants, if the users know how to take advantage of them.”The research was supported in part by the Marshall Commission, the Cambridge Trust, Peterhouse, Cambridge, The Alan Turing Institute, the European Research Council, and the Engineering and Physical Sciences Research Council (EPSRC), part of UK Research and Innovation (UKRI).<hr>Reference: Katherine M. Collins, Albert Q. Jiang, et al. ‘<a href="https://www.pnas.org/doi/10.1073/pnas.2318124121" target="_blank">Evaluating Language Models for Mathematics through Interactions</a>.’ Proceedings of the National Academy of Sciences (2024). DOI: 10.1073/pnas.2318124121

Chatbot concept. Credit: wutzkoh – stock.adobe.com

Researchers have developed a platform for the interactive evaluation of AI-powered chatbots such as ChatGPT. 

New open-source platform allows users to evaluate performance of AI-powered chatbots

In this episode, we discuss how 2 Carnegie Mellon University graduate students developed an AI system capable of giving you feedback on your interview performance in real time!

Podcast: Meet the AI That Helps You Ace Job Interviews

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/d97ba401-cff2-4367-adb7-043c01d4a88a?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>When Haoguang Cai and Khem Agarwal decided they wanted to work on a project to present at the annual IEEE Rising Stars conference, they knew they wanted to build something that would be of interest and use to the Institute for Electrical and Electronics Engineers audience of students and young technical professionals.The <a href="https://www.sv.cmu.edu/index.html" rel="noopener" target="_blank">Silicon ValleyOpens in new window</a> graduate students, who care about making AI that benefits people, came up with an idea that achieved both goals. They knew that IEEE Rising Star attendees are focused on career opportunities so they set to work devising a system that could help job seekers improve their performance and become more confident in job interviews.Cai and Agarwal also knew that many of their fellow students practiced interviewing with friends or colleagues. But when mock interview options were exhausted or not available, they still wanted more practice and helpful feedback.The system that the pair developed and later demonstrated at the Las Vegas conference analyzes a user’s facial and speech expression to detect what emotions they are conveying and tracks their gaze or eye motion to detect where they are focusing their attention. Feedback in both real time and after the interview session gives users insight as to how they are likely being perceived during an interview.“When I showed the live demo on stage, everyone wanted to try it. They really saw its potential and lined up after the workshop to try it and learn how we did it,” said Cai.<blockquote>When I showed the live demo on stage, everyone wanted to try it. They really saw its potential and lined up after the workshop to try it and learn how we did it.<cite>Haoguang Cai, MSSM Student, SV - III</cite></blockquote>The project originated in their Software Engineering Management class. Their instructor, <a href="https://engineering.cmu.edu/directory/bios/fang-catherine.html" target="_blank">Catherine Fang</a>, was scheduled to speak at the January 2024 conference. The distinguished service professor is an entrepreneur who has co-founded two companies, holds multiple U.S. patents, has worked for LinkedIn, Yahoo, and Sun Microsystems, and serves as an associate editor for the IEEE reliability society.“I’m so proud of all of our students, but this team really stood out. They have a very solid technical and product management background. Their presentation was a big success,” said Fang.<blockquote>I’m so proud of all of our students, but this team really stood out. They have a very solid technical and product management background. Their presentation was a big success.<cite>Catherine Fang, Distinguished Service Professor, SV - III</cite></blockquote>She added that emotion detection technology is an emerging trend with many useful consumer applications that she has seen demonstrated at other big conferences.Cai and Agarwal, who will both earn their Master of Science in Software Management (MSSM) from the Integrated Innovation Institute in December 2024, knew how to program the machine learning model to identify and measure facial expressions and eye tracking but they needed a data set with a large number of labelled images that represented a diverse set of people including those of varying color, age, and gender.They contacted Hume AI, a research lab and technology company that uses artificial intelligence to measure emotional expression, which was able to provide access to their extensive data set through use of their Application Programming Interface (API). APIs are commonly used in software development to enable integration between different systems or services for applications such as accessing data from a remote server.Jeremy Hadfield, who is a technical product manager at Hume, has supported the pair through the development of their emotion detection system. He was recently a guest at a campus event.Cai, who hopes to complete his transition from a machine learning engineer to product manager in the AI field after graduation, will intern with Hume AI this summer. Agarwal is planning for a career that will enable him to use AI to create virtual reality learning tools.They both agree that the challenge and extra work to build and demonstrate their emotion detection system was a worthwhile endeavor and that it only furthered their desire to use AI to build products that will help people.

Face expression analysis that users can view in real time while answering interview questions indicate the emotion they are conveying.

Silicon Valley students use AI to help job seekers improve performance and build confidence.

Emotion detection system puts a smile on their face

In this episode, we discuss research coming out of the Karlsruhe Institute of Technology to detect the emotions of tennis players better than humans can!

Podcast: Catching Feelings with 69 percent Accuracy: AI Reads Emotions in Sports

Structured text is a standardized method of organizing and presenting information in a hierarchical format, ensuring precision and consistency in engineering and programming environments.

Structured Text Syntax in Programming, Automation and Engineering Documentation

Brain-computer interfaces, or BCIs, hold immense potential for individuals with a wide range of neurological conditions, but the road to implementation is long and nuanced for both the invasive and noninvasive versions of the technology. <a href="https://engineering.cmu.edu/directory/bios/he-bin.html" target="_blank">Bin He</a> of Carnegie Mellon University is highly driven to improve noninvasive BCIs, and his lab uses an innovative electroencephalogram (EEG) wearable to push the boundaries of what’s possible. For the first time on record, the group successfully integrated a novel focused ultrasound stimulation to realize bidirectional BCI that both encodes and decodes brain waves using machine learning in a study with 25 human subjects. This work opens up a new avenue to significantly enhance not only the signal quality, but also, overall nonivasive BCI performance by stimulating targeted neural circuits.Noninvasive BCI is lauded for its merits of being cheap, safe, and virtually applicable to everyone, but because signals are recorded over the scalp versus inside the brain, low signal quality presents some limitations. The He group is exploring ways to improve the effectiveness of noninvasive BCIs and, over time, has used deep learning approaches to decode what an individual was thinking and then facilitate control of a cursor or robotic arm.<iframe width="640" height="360" src="https://www.youtube.com/embed/EoKHw6bdoZs?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;</iframe>In their latest research, published in <a href="https://www.nature.com/articles/s41467-024-48576-8.epdf?sharing_token=l63IL0ujKJwg9YW5RiuvmdRgN0jAjWel9jnR3ZoTv0N_uPWXwoTWkVO_kYNoQfCvZ9eb-1HxHFUK4NM7oxILM0CfREmenUY3abynvgquHLRn_7RPWL1rZyFXqJCnBSQm10CXtHA62yRq2BlSAcVQCigM1TtLWtL1uYkOTXn0ObY%3D" rel="noopener" target="_blank">Nature CommunicationsOpens in new window</a>,&nbsp;the He group demonstrated that through precision noninvasive neuromodulation using focused ultrasound, the performance of a BCI could be improved for communication.“This paper reports a breakthrough in noninvasive BCIs by integrating a novel focused ultrasound stimulation to realize bidirectional BCI functionality,” explained Bin He, professor of <a href="https://www.cmu.edu/bme/" rel="noopener" target="_blank">biomedical engineeringOpens in new window</a> at Carnegie Mellon University. “Using a communication prosthetic, 25 human subjects spelled out phrases like ‘Carnegie Mellon’ using a BCI speller. Our findings showed that the addition of focused ultrasound neuromodulation significantly boosted the performance of EEG-based BCI. It also elevated theta neural oscillation that enhanced attention and led to enhanced BCI performance.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1718173646497-ezgif-2-339b99d07b.gif" style="width: 100%px;" class="fr-fic fr-dib">This video shows the BCI speller progress used by study subjects to spell “Carnegie Mellon.”For context, a BCI speller is a 6x6 visual motion aide containing the entire alpabet that is commonly used by nonspeakers to communicate. In He’s study, subjects donned an EEG cap and just by looking at the letters, were able to generate EEG signals to spell the desired words. When a focused ultrasound beam was applied externally to the V5 area (part of the visual cortex) of the brain, the performance of the noninvasive BCI greatly improved among subjects. The neuromodulation-integrated BCI actively altered the engagement of neural circuits to maximize the BCI performance, compared to previous uses, which consisted of pure processing and decoding recorded signals.“The BRAIN Initiative has supported more than 60 ultrasound projects since its inception. This unique application of noninvasive recording and modulation technologies expands the toolkit, with a potentially scalable impact on assisting people living with communication disabilities,” said Dr. Grace Hwang, program director at the Brain Research Through Advancing Innovative Neurotechnologies® initiative (The BRAIN Initiative®) at the National Institutes of Health (NIH).&nbsp;<blockquote>The BRAIN Initiative has supported more than 60 ultrasound projects since its inception. This unique application of noninvasive recording and modulation technologies expands the toolkit, with a potentially scalable impact on assisting people living with communication disabilities.<cite>Grace Hwang, program director, National Institutes of Health</cite></blockquote>Following this discovery, the He lab is further investigating the merits and applications of focused ultrasound neuromodulation to the brain, beyond the visual system, to enhance noninvasive BCIs. They also aim to develop more compact-focused ultrasound neuromodulation device for better integration with EEG-based BCIs, and to integrate AI to continue to enhance the overall system performance.“This is my lifelong interest, and I will never give up,” emphasized He. “Working to improve noninvasive technology is difficult, but I strongly believe that if we can find a way to make it work, everyone will benefit. I will keep working, and someday, noninvasive lifesaving technology will be available for every household.This work was supported by NIH’s The BRAIN Initiative and the Helping to End Addiction Long-term® Initiative (NIH HEAL Initiative®), the National Institute of Neurological Disorders and Stroke, the National Institute of Biomedical Imaging and Bioengineering, the National Center for Complementary and Integrative Health, and by the National Science Foundation.Other collaborators on the Nature Communications paper include the first author Joshua Kosnoff, BME Ph.D. student, Kai Yu, BME research scientist; and Chang Liu, former BME master’s student.

To strengthen the case for noninvasive BCIs, Carnegie Mellon researchers have demonstrated that through precision neuromodulation using focused ultrasound, the performance of a BCI could be improved for communication.

Human-Computer Interaction

Profiles