Where verbal communication is hindered, such as in locations with lots of background noise or where an individual has an existing speech impairment, SSI systems are a cutting-edge solution, enabling verbal communication without vocalisation. As such, it is a type of electronic lip-reading using human-computer interaction.In new research, led by the University of Cambridge, an overlying structured graphene layer is applied to an integrated textile strain sensor for robust speech recognition performance, even in noisy environments.Worn around the neck, the smart choker captures micromovements in the throat, which are then picked up by the strain sensor as an electrical signal and fed into brain-inspired computer software models for processing and speech recognition. It can pick up even silently mouthed words and broadcast it, which could help someone who is unable to speak following laryngeal surgery, for example.The smart choker’s unique structure features ordered thorough cracks on graphene-coated textiles. The structured graphene layer significantly enhances the sensitivity of the strain sensor. It can dynamically respond to throat micromovements, enabling the capture of information-rich speech signals. These signals are then processed through a computationally efficient neural network, with a record accuracy of 95.25% in speech decoding.&nbsp;The <a href="https://doi.org/10.1038/s41528-024-00315-1" target="_blank">results</a>, reported in the journal npj Flexible Electronics, offer a promising, non-invasive solution for practical, wearable SSI systems, paving the way for seamless, natural silent communication in diverse settings.The proposed SSI system is robust and able to decode a wide range of words, while swiftly adapting to new users and vocabularies. The technology was recently showcased as a live demonstration at the <a href="https://2024.ieee-biosensors.org/" target="_blank">IEEE Biosensors 2024</a> conference, attracting the attention of 180 plus participants to the event.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1728346384523-Screenshot+2024-08-15+at+15.51.32.png" style="width: 100%px;" class="fr-fic fr-dib">An overview of the wearable SSI, featuring an ultrasensitive strain sensor and a neural network for efficient speech recognition.The research was led by <a href="https://www.eng.cam.ac.uk/profiles/lgo23" target="_blank">Dr Luigi G. Occhipinti</a>, Director of Research in Smart Electronics, Biosystems and AI, and Head of the&nbsp;<a href="http://www.occhipintigroup.com/" target="_blank">Occhipinti Group</a> in the&nbsp;<a href="https://ee.eng.cam.ac.uk/" target="_blank">Electrical Engineering Division</a> at the Department of Engineering and&nbsp;the&nbsp;<a href="https://www.graphene.cam.ac.uk/" target="_blank">Cambridge Graphene Centre</a>.Dr Occhipinti said: “Our user-friendly smart choker demonstrates a remarkable ability to efficiently perform in real-world scenarios with users of different genders, geographical, and ethnic backgrounds, across new and potentially ambiguous words of varying lengths and familiarity, and at varying reading speeds.“Our SSI system operates with high precision and computational efficiency in distinguishing the speech of different users against different types of noise, caused by sensor imperfections, the external environment, and/or from the users’ own body movements while wearing the device.“Furthermore, the fabrication method of our ultrasensitive textile strain sensor technology is biocompatible, simple, low cost, and scalable. It is also adaptable to prolonged use and can withstand more than 10,000 stretching-releasing cycles while maintaining stable and reliable electrical functionality.“Put simply, the synergy of our sensor design and neural network optimisation sets a new standard in wearable silent speech communication technologies, offering a comfortable smart choker with groundbreaking potential.”<hr>Reference: Chenyu Tang, Muzi Xu, Wentian Yi, Luigi Giuseppe Occhipinti, et al. ‘<a href="http://doi.org/10.1038/s41528-024-00315-1" target="_blank">Ultrasensitive textile strain sensors redefine wearable silent speech interfaces with high machine learning efficiency’</a>. npj Flexible Electronics (2024). DOI: 10.1038/s41528-024-00315-1

Micromovements in the throat are captured by an ultrasensitive strain sensor integrated into a smart choker, comprising a textile substrate with an overlying structured graphene layer. Credit: Mr Zibo Zhang from University College London

A wearable ‘smart’ choker for speech recognition has the potential to redefine the field of silent speech interface (SSI), say researchers – thanks to embedded ultrasensitive textile strain sensor technology.

Graphene-based wearable strain sensor can detect and broadcast silently mouthed words

Automatic speech recognition and translation systems such as the Lecture Translator of the Karlsruhe Institute of Technology (KIT) can convert the spoken word in lectures into a text in several languages in real time. Such systems improve access to information for students with disabilities and foreign students. They also speed up work and learning in general through intelligent post-processing and archiving of spoken texts. KIT researchers have added new functions to the Lecture Translator. Automatic recognition of the spoken word in several languages at the same time, text segmentation and title generation in real time, summaries and links to technical terms as well as queries on what has been heard now make it easier to understand and efficiently process lectures."With Lecture Translator's automatic simultaneous translation, we have brought spoken lectures closer to an international audience. However, this usually represents only 15 percent of the audience. With the new AI tools, we want to break down not only language barriers, but also comprehension barriers," says Alexander Waibel, Professor of Computer Science at KIT. "Automatically transcribed texts of spoken language are often difficult to read, because they appear too quickly as a long text without paragraphs and subheadings - just as the lecture was delivered orally". Processing the lecture is also time-consuming, because you have to search the lecture for gaps in understanding," says Waibel.<h2>Better overview of documents</h2>The further development of the Lecture Translator provides a remedy. The researchers have developed several new automatic functions such as "Smart Chaptering", "Summarization", "Q&amp;A" and "Auto-Links". A new type of artificial intelligence (AI) that automatically recognizes language converts the spoken text into a transcript in multiple languages and automatically identifies paragraphs, chapter headings and key points. It also creates an acoustic rendition in which the user can select one of 18 languages. The program also automatically displays links as cross-references to relevant sources in lecture notes or Wikipedia, which students can use to better process the lecture. "With our new AI models, conversations and lectures can be better structured and even videos can be divided into easily navigable chapters," says Waibel. This enables better understanding not only during the lecture, but also after the lecture.<h2>Lecture Translator translates into 18 languages</h2>The research team has integrated the work into the Lecture Translator, which is used at KIT to automatically transcribe lectures in real time. Chapter division, title generation, paragraph layout, summaries with links - which can also be used online and offline - now extend the Lecture Translator service and make it easier to work with the material. Translation is available in 18 languages. The technology has specific applications for content creators, students, teachers and podcasters who can structure their audio and video content for the first time. "Users can navigate through videos and lectures more efficiently, find relevant sections more quickly, and capture important core content more compactly and efficiently - they have a much better overall view and faster access to details," says Waibel.The research was carried out as part of the project "How is AI Changing Science?" and was funded for four years by the Volkswagen Foundation. In addition to KIT, the University of Bonn and the University of Vienna were also involved in the project.<a href="https://www.kcist.kit.edu/" target="_blank">Details on the KIT Center Information - Systems - Technologies</a>

With new functions such as automatic chapter division and summaries, the Lecture Translator offers improved speech recognition. (Photo: Markus Breig, KIT)

IT Researchers Further Develop Intelligent Speech Recognition and Translation System: Automatic Text Segmentation, Summarization, and Intelligent Query Possible

Automatic Speech Recognition and Translation: Faster Working and Learning

Ever noticed online ads following you that are eerily close to something you&rsquo;ve recently talked about with your friends and family? Microphones are embedded into nearly everything today, from our phones, watches, and televisions to voice assistants, and they are always listening to you. Computers are constantly using neural networks and AI to process your speech, in order to gain information about you. If you wanted to prevent this from happening, how could you go about it?Back in the day, as portrayed in the hit TV show &ldquo;The Americans,&rdquo; you would play music with the volume way up or turn on the water in the bathroom. But what if you didn&rsquo;t want to constantly scream over the music to communicate?&nbsp;<a href="https://www.engineering.columbia.edu/" target="_blank">Columbia Engineering</a> researchers have developed a&nbsp;<a href="https://arxiv.org/pdf/2112.07076.pdf" target="_blank">new system that generates whisper-quiet sounds that you can play in any room, in any situation, to block smart devices from spying on you</a>. And it&rsquo;s easy to implement on hardware like computers and smartphones, giving people agency over protecting the privacy of their voice.&ldquo;A key technical challenge to achieving this was to make it all work fast enough,&rdquo; said&nbsp;<a href="http://www.cs.columbia.edu/~vondrick/" target="_blank">Carl Vondrick</a>, assistant professor of&nbsp;<a href="https://www.engineering.columbia.edu/departments/computer-science" target="_blank">computer science</a>. &ldquo;Our algorithm, which manages to block a rogue microphone from correctly hearing your words 80% of the time, is the fastest and the most accurate on our testbed. It works even when we don&rsquo;t know anything about the rogue microphone, such as the location of it, or even the computer software running on it. It basically camouflages a person&rsquo;s voice over-the-air, hiding it from these listening systems, and without inconveniencing the conversation between people in the room.&rdquo;<h2>Staying Ahead of Conversations</h2>While the team&rsquo;s results in corrupting automatic speech recognition systems have been theoretically known to be possible in AI for a while, achieving them fast enough to use in practical applications has remained a major bottleneck. The problem has been that a sound that breaks a person&rsquo;s speech now&mdash;at this specific moment&mdash;isn&rsquo;t a sound that will break speech a second later. As people talk, their voices constantly change as they say different words and speak very fast. These alterations make it almost impossible for a machine to keep up with the fast pace of a person&rsquo;s speech.&ldquo;Our algorithm is able to keep up by predicting the characteristics of what a person will say next, giving it enough time to generate the right whisper to make,&rdquo; said Mia Chiquier, lead author of the study and a PhD student in Vondrick&rsquo;s lab. &ldquo;So far our method works for the majority of the English language vocabulary, and we plan to apply the algorithm on more languages, as well as eventually make the whisper sound completely imperceptible.&rdquo;<h2>Launching &ldquo;Predictive Attacks&rdquo;</h2>The researchers needed to design an algorithm that could break neural networks in real time, that could be generated continuously as speech is spoken, and applicable to the majority of vocabulary in a language. While earlier work had successfully tackled at least one of these three requirements, none have achieved all three. Chiquier&rsquo;s new algorithm uses what she calls &ldquo;predictive attacks&rdquo;&mdash;a signal that can disrupt any word that automatic speech recognition models are trained to transcribe. In addition, when attack sounds are played over-the-air, they need to be loud enough to disrupt any rogue &ldquo;listening-in&rdquo; microphone that could be far away. The attack sound needs to carry the same distance as the voice.The researchers&rsquo; approach achieves real-time performance by forecasting an attack on the future of the signal, or word, conditioned on two seconds of input speech. The team optimized the attack so it has a volume similar to normal background noise, allowing people in a room to converse naturally and without being successfully monitored by an automatic speech recognition system. The group successfully demonstrated that their method works inside real-world rooms with natural ambient noise and complex scene geometries.<h2>Ethical AI</h2>&ldquo;For many of us in the research community, ethical concerns of AI technology are an essential issue, but it seems to belong to a separate thought process. It is like we are so happy that we finally made a driving car but forgot to design a steering wheel and a brake,&rdquo; says <a href="https://www.grasp.upenn.edu/people/jianbo-shi/" target="_blank">Jianbo Shi</a>, professor of computer and information science at the University of Pennsylvania and a leading researcher in machine learning. &ldquo;As a community, we need to `consciously&#39; think about the human and societal impact of the AI technology we develop from the earliest research design phase. Mia Chiquier and Carl Vondrick&rsquo;s study poses the question: &lsquo;how to use AI to protect us against unintended AI usages?&rsquo; Their work makes many of us think in the following direction: ask not what ethical AI can do for us, but what we can do for ethical AI? Once we believe in this direction, ethical AI research is just as fun and creative.&rdquo;<hr>Chiquier will present her paper on April 25, 2022, at the&nbsp;<a href="https://iclr.cc/" id="isPasted" target="_blank">International Conference for Learning Representations</a>.&nbsp;

Columbia computer scientists have built an algorithm that blocks a rogue microphone from correctly hearing your words—in English so far—80% of the time

Concerned Your Smartphone Is Spying on You?

<html><head></head><body>Wevolver and<a href="https://www.wevolver.com/profile/syntiant" target="_blank">&nbsp;Syntiant</a> are creating a series that explore the work of innovators and the future of pervasive AI. Syntiant is developing ultra-low-power AI processors. Because they believe in the importance of innovation, Syntiant is engaging in these fireside chats with engineers and designers who are on the cutting edge of their field.This video is part of a series of fireside chats in which Wevolver and Syntiant partner to engage with global innovators. In <a href="https://www.wevolver.com/tag/fireside-chat" target="_blank">previous discussions</a> we spoke with The Things Industries CEO Wienke Giezeman, Star Wars Animatronic Designer Gustav Hoegen, and Fashiontech Designer Anouk Wipprecht.&nbsp;<h2 dir="ltr">Using humans and AI to translate and transcribe&nbsp;</h2>When Rev was in its infancy, it was mainly comprised of human freelancers sitting behind their screens, listening with intent to provide true to word accuracy. But in 2016, with the onset of the deep learning revolution, speech-to-text took off at high speeds. &nbsp;“The coupling of hiring intelligent people and working with a dataset that we had created over many years of doing human transcription translated into a very powerful automated speech recognition ecosystem. Now it's humans and machines working together.”&nbsp;Rev’s&nbsp;speech recognition engine is trained on a corpus of human-transcribed text. When a client needs human level quality, a human will translate, but with access to more robust datasets, they’ll actually start with a draft from the speech recognition model.&nbsp;The dataset is fed to the speech recognition engine for training, and the more data it gets from humans, the better the engine becomes. In turn, the “Revvers”, or human freelancers, can save an exorbitant amount of time by starting with a more accurate draft produced by the machine.“You can imagine, if we're doing transcripts for earnings calls, and all those need to be 100 percent accurate, they’re going to want human quality. On the other hand, if you’re doing something that necessitates a quick turnaround, where you might want something within minutes of having available audio, or even live – imagine a Zoom app that could show you a live transcript of our conversation - that will just be AI.”&nbsp;<h2 dir="ltr">Speech-to-text for law enforcement </h2>The ubiquity of the spoken word translated to text has started to feel like a simple extension of how we speak. Digital assistants are bubbling up on our shelves and mantels, telling us the weather or switching on a light. Legal officials transcribe interviews for depositions. And more recently, law enforcement has used speech to text for body camera equipment, a vertical Kokotov is exploring.Rev is working with Axon, a manufacturer of body cam equipment, the largest that leverage software tools for police departments to capture the audio from body cameras, and quickly leverage it for writing up reports for evidence collection. They record, it gets uploaded to Axon, it goes through transcription with&nbsp;Rev,&nbsp;and they don't have to worry about forgetting crucial details when filing reports.“Many of us have seen more body camera footage in the last year and a half than we've ever had before. It’s extremely important that we get the content of those conversations right because they affect a lot of people's lives, and they need to be scalable because officers are constantly in the field.”&nbsp;<h2 dir="ltr">Low resource natural language processing</h2>Speech recognition technologies notoriously favor the English language, and oftentimes fail to recognize different languages, accents, or speech patterns. Stepping even further into multilingual conversations - swinging back and forth between Portuguese and Arabic for example - no speech recognition engine would really handle that very well. However, new techniques, such as using multilingual transformers, are much more amenable to understanding these low resource languages. &nbsp;“I believe this will be one of the biggest challenges and themes over the next five to ten years, in taking speech recognition global, doing better with more languages, and improving these cross language situations. Our goal is to bring what we think is the best speech recognition to as many people as possible. In the service of this vision is understanding every human voice.”</body></html>

#7 of our 'Voice of Innovation' fireside chat series: Robotics and AI reporter Rachel Gordon speaks to Kokotov on his vision to understand every human voice. 

Fireside chat with Rev.com's Daniel Kokotov, VP of Engineering within Speech Technologies

The advancements in computer vision, pattern recognition and signal processing have made visual speech recognition possible for a variety of applications, such as human-computer interaction, speaker recognition, talking heads, sign language recognition and video surveillance. Visual speech recognition deals with several technological breakthroughs, like image processing, artificial intelligence, object detection, and statistical modeling only uses visual signals of the lip movement that produce speech without relying on the audio [1]. In the first generation of VSR, the research could not meet the high accuracy demands of the industry due to factors concerning the size of the audio-visual dataset with limited vocabulary and hand-crafted visual features that are not optimal for VSR model training. &nbsp;The introduction of deep learning and neural networks has made visual speech recognition models more robust than ever before while achieving higher accuracy in unstructured real-life situations. Most recent innovations allow the models to perform much better due to the availability of larger datasets and training models with a huge amount of data. A group of researchers affiliated with the Department of Computing at Imperial College London and Meta AI proposed a novel approach to add prediction-based auxiliary tasks to the VSR model and outperform state-of-the-art methods trained on the publicly available data [2].<h2 dir="ltr">Looking to Listen</h2>In the research paper, “Visual Speech Recognition for Multiple Languages in the Wild,” the team carried out work with the aim to reduce the word error rate (WER) through appropriate data augmentation and hyperparameter optimization of the existing architecture while working on the publicly available datasets that are much smaller in size. Through this, the team points out the fact that the optimization of the models can be brought in through carefully designing the models and not only increasing the size of the training data. Also, the proposed novel method for visual speech recognition not only evaluates English as a sole language but also on Mandarin, Spanish, Italian, French and Portuguese and achieves state-of-the-art performance on all the languages.&nbsp;“We optimize hyperparameters and improve the language model with the aim of squeezing extra performance out of the model,” the team explains. Then, “we introduce time-masking which is a temporal augmentation method and is commonly used in ASR models. Finally, we use a VSR model with auxiliary tasks where the model jointly performs visual speech recognition and prediction of audio and visual representations extracted from pre-trained VSR and ASR models.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1647384806878-Visual+Speech+Recognition+Results+on+the+LSR2+Dataset.png" style="width: 800px;" class="fr-fic fr-dib" alt="Visual Speech Recognition Results on the LSR2 Dataset">Visual Speech Recognition Results on the LSR2 Dataset [Image Source: Research Paper]<div style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 374.688px; height: 39px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'> </div>Hyperparameter optimization aims to improve the performance of the model by adjusting the values of the parameters that are employed to control the training process and the model architecture. There are several hyperparameters taken into consideration including initial learning rate, learning rate decay parameter, number of layers, size of layers, dropout rate, and the loss function that was performed on the LRS2 dataset achieving reduction in WER on the validation set. The main hyperparameter that had a significant impact on the performance and which might not have been exhaustively optimized was the batch size. The team observed that reduction WER was achieved more when the batch size increased from 8 to 16. Due to GPU memory constraints, the team recognized that certain hyperparameters could not be increased further, but they would bring down the word error rate.&nbsp;<h2 dir="ltr">State-of-the-art performance</h2>The team decided to use all the open-source datasets available for public use, namely LRS2 (an audio-visual English dataset from BBC programs), LRS3 (an audio-visual English dataset taken from TED and TEDx talks), CMLR (a Mandarin dataset collected from Chinese news program), CMU-MOSEAS (contains multiple languages from YouTube videos), Multilingual TEDx (multilingual corpus taken from TEDx talks), and AVSpeech (dataset with over 4700 hours of video in multiple languages). To measure performance, the word error rate is a common performance metric in speech recognition, with an alternative of a character error rate that measures how close the predicted and target character sequences are.&nbsp;For the multilingual TEDxSpanish dataset, the approach results in a 5.6% absolute reduction in the WER and a further reduction of 4.2% can be achieved by using additional training data. While for the Italian corpus, the approach results in an absolute drop of 5.6% in the WER, the same as that for Spanish, but a further reduction of up to 8% for additional training data. “Results on the Multilingual TEDxFrench dataset are shown as a result of a 9.4 % absolute reduction in the WER and a further reduction of 7.6 % can be achieved by using additional training data.” [More details on the additional training data are available in the research article]<iframe width="640" height="360" src="https://www.youtube.com/embed/FIau-6JA9Po?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true"></iframe>The proposed novel method for visual speech recognition performs visual speech recognition and prediction of audio and visual representation on a publicly available dataset to outperform state-of-the-art methods. The research work was published on Cornell University’s research sharing platform, arXiv under <a href="https://arxiv.org/abs/2202.13084" target="_blank">open access terms</a>. The researchers have provided pre-trained networks and testing code on the authors’ <a href="https://github.com/mpc001/Visual_Speech_Recognition_for_Multiple_Languages" target="_blank">GitHub repository</a> for community access.<h2 dir="ltr">References</h2>[1] Ahmad B. A. Hassanat: Visual Speech Recognition. DOI arXiv: <a href="https://arxiv.org/abs/1409.1411" target="_blank">1409.1411 [cs.CV]</a>.[2] Pingchuan Ma, Stavros Petridis, Maja Pantic: Visual Speech Recognition for Multiple Languages in the Wild. DOI arXiv: <a href="https://arxiv.org/abs/2202.13084" target="_blank">2202.13084 [cs.CV]</a>.&nbsp;

Visual Speech Recognition on a Multilingual Dataset that outperforms the state-of-the-art methods [Image Source: Research Paper]

A novel approach to visual speech recognition that carefully designs the VSR model rather than increasing the size of the training data.


Research suggests improvements in visual speech recognition on a multilingual dataset

Researchers at Rutgers University-New Brunswick have&nbsp;<a href="https://dl.acm.org/doi/abs/10.1145/3447993.3483272" target="_blank">published “Face-Mic,” the first work</a> examining how voice command features on virtual reality headsets could lead to major privacy leakages, known as “eavesdropping attacks.”The research shows that hackers could use popular virtual reality (AR/VR) headsets with built in motion sensors to record subtle, speech-associated facial dynamics to steal sensitive information communicated via voice-command, including credit card data and passwords.Common AR/VR systems on the market include the popular brands Oculus Quest 2, HTC Vive Pro, and PlayStation VR.Led by&nbsp;<a href="http://www.winlab.rutgers.edu/~yychen/" target="_blank">Yingying “Jennifer” Chen, associate director of WINLAB and graduate director of Electrical and Computer Engineering at Rutgers University-New Brunswick</a>, the study will be presented at the annual International Conference on Mobile Computing and Networking in March. Other research collaborators include Nitesh Saxena of Texas A&amp;M University and Jian Liu at University of Tennessee at Knoxville.To demonstrate the existence of security vulnerabilities, Chen and her fellow WINLAB researchers developed an eavesdropping attack targeting AR/VR headsets, known as “Face-Mic.”“Face-Mic is the first work that infers private and sensitive information by leveraging the facial dynamics associated with live human speech while using face-mounted AR/VR devices,” said Chen. “Our research demonstrates that Face-Mic can derive the headset wearer’s sensitive information with four mainstream AR/VR headsets, including the most popular ones: Oculus Quest and HTC Vive Pro.”&nbsp;The researchers studied three types of vibrations captured by AR/VR headsets’ motion sensors, including speech-associated facial movements, bone-borne vibrations and airborne vibrations. Chen noted that bone-borne vibrations in particular are richly encoded with detailed gender, identity and speech information.“By analyzing the facial dynamics captured with the motion sensors, we found that both cardboard headsets and high-end headsets suffer security vulnerabilities, revealing a user’s sensitive speech and speaker information without permission,” Chen said.Although vendors usually have policies regarding utilizing the voice access function in headset microphones, Chen’s research found that built-in motion sensors, such as an accelerometer and gyroscope within a VR headset, do not require any permission to access. This security vulnerability can be exploited by malicious actors intent on committing eavesdropping attacks.Eavesdropping attackers can also derive simple speech content, including digits and words, to infer sensitive information, such as credit card numbers, Social Security numbers, phone numbers, PIN numbers, transactions, birth dates and passwords. Exposing such information could lead to identity theft, credit card fraud and confidential and health care information leakage.Chen said once a user has been identified by a hacker, an eavesdropping attack can lead to further exposure of user’s sensitive information and lifestyle, such as AR/VR travel histories, game/video preferences and shopping preferences. Such tracking compromises users’ privacy and can be lucrative for advertising companies.Oculus Quest, for example, supports voice dictation for entering web addresses, controlling the headset and exploring commercial products. Rutgers’ Face-Mic research shows that hackers may leverage these zero-permission sensors to capture sensitive information, leading to severe privacy leakages. &nbsp; &nbsp;Chen said she hopes these findings will raise awareness in the general public about AR/VR security vulnerabilities and encourage manufacturers to develop safer models.“Given our findings, manufacturers of VR headsets should consider additional security measures, such as adding ductile materials in the foam replacement cover and the headband, which may attenuate the speech-associated facial vibrations that would be captured by the built-in accelerometer/gyroscope,” she said.Chen and her WINLAB colleagues are now examining how facial vibration information can authenticate users and improve security, and how AR/VR headsets can capture a user’s breathing and heart rate to measure well-being and mood states unobtrusively. To learn more about ongoing studies at WINLAB,&nbsp;<a href="http://www.winlab.rutgers.edu/" target="_blank">click here</a>.

Rutgers researchers have published the first work examining how hackers could use popular virtual reality headsets to steal sensitive information communicated via voice-command, including credit card data and passwords.

Researchers Discover Security Vulnerabilities in Virtual Reality Headsets

EPFL’s Laboratory for Experimental Museology (eM+), led by Sarah Kenderdine, has received support from Engagement Migros to produce muse, a pioneering audience evaluation app, in collaboration with over 20 Swiss museums. The tool provides visitors with an engaging way to communicate their experiences, while helping museums to consider this feedback when developing exhibits and other attractions. Encouraging audiences to participate in evaluations in large enough numbers to provide useful data is notoriously difficult, and for good reason: respondents may find the process time-consuming or boring. But what if you were invited to communicate your experience at a museum exhibit by making drawings, leaving voice recordings, interacting with animated graphics, or taking photos?&nbsp;This is the aim of muse: a design-led audience evaluation app that helps museums get to know their visitors better. In addition to geographic and demographic information, the tablet-based tool uses some 30 interactive elements to collect qualitative, subjective data about museum visitors’ experiences.<h2>Capturing experiences</h2>Sarah Kenderdine leads the eM+ lab in EPFL’s College of Humanities. She explains that in Switzerland, as in many parts of the world, museums usually collect only basic demographic data from their visitors, such as number of entries and origin. The goal of muse is therefore to give museums access to data analytics tools that can help them focus on many visitors as individuals, and to use the quality of visitor engagement as a method to define, measure, and program for improved museum experiences. “It is the social, political, emotional, educational, and creative elements, in addition to how deeply people are participating in a museum’s programs and exhibitions, that should be the basis for measuring value,” Kenderdine says. Unlike a traditional audience evaluation or survey, visitors use muse to record their experiences as they pass through an exhibit, rather than afterward. “Muse is always meant to be deployed inside the space of experience, and not afterward, so people can reflect in real time on what their reactions to it really are. Simultaneously, museum staff can see aggregated results on a real-time dashboard,” says Kenderdine, who has been developing muse since 2012. She notes a recent example of the success of the muse model, when a previous version of the app was deployed for the Winter at Tantora arts festival in AlUla, Saudi Arabia. It engaged 6,000 respondents across five surveys with 120 data points, resulting in 1,200 pages of reports. “This data provided a phenomenal insight into customer loyalty, future planning, and creative responses to the cultural offering at this new world heritage destination.” With the support of Engagement Migros, and the participation of more than 20 museums across Switzerland, Kenderdine’s goal now is to bring muse to the next level, including more sophisticated analysis driving better public outcomes for museums and their audiences. The project will be officially launched on September 17th, 2020 at EPFL’s ArtLab.<h2>Looking ahead</h2>At first, the muse team will partner with eight ‘core’ museums and institutions for a three-year period, including: the Château de Morges (Morges), the Haus der Elektronischen Künste (Basel), the International Red Cross Museum (Geneva), the Musée d'Éthnographie (Geneva), the Museum für Gestaltung (Zürich), the Rietberg Museum (Zürich), and the Olympic Museum (Lausanne). Each institution will be supported by an administration portal for creating questions and surveys, an app for collecting data, and a dashboard for visualizing the data in real time. A further 16 museums will be invited to join the project through an open call starting in late 2020. Throughout the four-year funding period, the muse team will develop convincing case studies for the Swiss museum landscape. It will also explore options for commercializing the app and its licensing models, and for expanding to include all cultural offerings across Europe and beyond – all in close collaboration with its partners. “With its user-centric approach, muse supports museums in their efforts to continue being active places in society,” says Johanna Muther, Project Manager at Engagement Migros. Kenderdine adds that going forward, muse will also help museums to increasingly embrace their audiences with renewed attention as they adapt to the challenges of the coronavirus pandemic. “Visitors are active participants, and not just consumers, in helping to support museums as polyvocal places. Muse gives an amplifier to those voices,” she says.<hr>About muse Muse is the voice of the visitor. In addition to quantitative data, such as geographic and demographic information, the tool can also be used to collect qualitative findings on the direct impact of an exhibition visit. The focus of the app is on creating playful interactions, and on being easy to use – for museums and visitors alike. Museums can use the findings to take greater account of the perspective of visitors when curating and developing their exhibits, thus helping them to remain firmly established as active places in society, both now and in the future.About Engagement Migros The Engagement Migros development fund enables pioneering projects in the midst of social change, which break new ground and test solutions aimed at the future. This approach is oriented towards making a difference and combines financial support with coaching services in its Pioneerlab. Engagement Migros is made possible by the companies of the Migros Group thanks to annual funding of approximately CHF 10 million; it has been complementing the Migros Culture Percentage since 2012.

The tool provides visitors with an engaging way to communicate their experiences, while helping museums to consider this feedback when developing exhibits and other attractions.

Data-driven app brings 'voice of the visitor' to museum experiences

<h2 class="lw lx el aq ap cc ly lz ma mb mc md me mf mg mh mi" data-selectable-paragraph="" id="31c7">A very brief history of talking to computers</h2>Possibly the first voice activated machine was a 2$ toy called Radio Rex produced in 1920s. It was a spring activated dog that popped out of a little kennel whenever it detected a sound in the 500 Hz range. It wasn&rsquo;t Siri, Cortana or Alexa, but quite impressive for it&rsquo;s time. Later in the 80s, IBM introduced a dictation taking computer which didn&rsquo;t really catch on. It wasn&rsquo;t until recently with the introduction of Apple&rsquo;s Siri that the fiction of Star trek or HAL started approaching reality. A little while before Siri, in the 90s there existed voice dialers which allowed people to dial unto ten different phone numbers by voice. By 2000s, Interactive Voice Response (IVR) systems became fairly commonplace. Apple unveiled Siri in 2011 on the iPhone 4S, but it was the introduction of Amazon&rsquo;s Alexa in 2014 that brought voice interfaces to the forefront. Every major tech company followed suite later.<hr class="mx db hl my mz hy na nb nc nd ne"><section class="jt ju jv jw jx"><div class="bt jy af du dq dr"><blockquote class="nf"><div class="ng nh ht aq kb b ni nj nk nl nm nn mw" data-selectable-paragraph="" id="2bd1">Hey! But I already use my fingers to swipe on Tinder and scroll on Instagram. Why should I care about voice?</div></blockquote><h2 class="lw lx el aq ap cc ly nq ma nr mc ns me nt mg nu mi" data-selectable-paragraph=""> </h2><h2 class="lw lx el aq ap cc ly nq ma nr mc ns me nt mg nu mi" data-selectable-paragraph="" id="31a4">Voice has some clear advantages</h2><ul><li class="mj mk el aq ml b mm mn mo mp mq mr ms mt mu mv mw nv nw nx" data-selectable-paragraph="" id="e914">In many cases, it is faster to say something than perform complex actions to complete a task. It is easier to say &lsquo;set an alarm for 7 AM&rsquo; than finding your alarm app, click on the set alarm button, select 7, then select AM and finally click on the save button.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="0134">In cases where users can&rsquo;t use their hands like driving or cooking, speaking is much safer and practical.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="97fd">There is little or no learning required to interact with machines by voice. Talking is intuitive. Humans already know how to talk. Communication by speech is ubiquitous.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="75f2">Voice can convey much more information than text alone. Speech consists of tone of voice, intonation, rate of speech. Embedded in voice is information about emotions. A computer can respond much better to speech than text alone.</li></ul><h2 class="lw lx el aq ap cc ly lz ma mb mc md me mf mg mh mi" data-selectable-paragraph="" id="d936">That said, it&rsquo;s not always appropriate to use voice as the only medium of communication with computers</h2><ul><li class="mj mk el aq ml b mm mn mo mp mq mr ms mt mu mv mw nv nw nx" data-selectable-paragraph="" id="c1d9">It is downright&nbsp;irritating to operate computers with voice in public spaces. Imagine everyone in an office environment interacting with their computers through voice. It would be chaotic.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="11c0">Sometimes it is preferable to text. Right now, it is&nbsp;difficult to undo what you&rsquo;ve said. It&rsquo;s easier to delete text when communicating something that requires thought and introspection.</li><li class="mj mk el aq ml b mm ny mo nz mq oa ms ob mu oc mw nv nw nx" data-selectable-paragraph="" id="b25a">Privacy is a major concern. You wouldn&rsquo;t want to speak your health issues to a computer in a crowded environment. It would be a violation of privacy if the computer reads out your messages without confirming.</li></ul>The most important thing is to start thinking of voice interface as just another mode of interacting with machines. Talking to your computer in many cases will be cumbersome or downright inappropriate. Saying &lsquo;scroll up&rsquo; every time you want to scroll your facebook feed doesn&rsquo;t sound very intuitive but &lsquo;share my trip photos with my girlfriend&rsquo; is.</div></section>

Possibly the first voice activated machine was a 2$ toy called Radio Rex produced in 1920s. It was a spring activated dog that popped out of a little kennel whenever it detected a sound in the 500 Hz range.

speech-recognition