2023 Edge AI Technology Report. Chapter VII: Sensing Modalities

Edge AI, empowered by the recent advancements in Artificial Intelligence, is driving significant shifts in today's technology landscape. By enabling computation near the data source, Edge AI enhances responsiveness, boosts security and privacy, promotes scalability, enables distributed computing, and improves cost efficiency.

Wevolver has partnered with industry experts, researchers, and tech providers to create a detailed report on the current state of Edge AI. This document covers its technical aspects, applications, challenges, and future trends. It merges practical and technical insights from industry professionals, helping readers understand and navigate the evolving Edge AI landscape.

Introduction

Sensing modalities enable capturing of environmental data for analysis using traditional and modern models and AI algorithms. By facilitating efficient data acquisition and processing, these cutting-edge technologies empower informed decision-making and significantly enhance productivity. The exploration of sensing modalities is of utmost importance, as it highlights their critical role in the comprehensive process of data collection, analysis, and, ultimately, intelligent decision-making.

Sensing modalities encompass various sensors and data capture devices, such as cameras, microphones, temperature sensors, and motion sensors. Each modality specializes in capturing specific types of data, contributing to a holistic understanding of the device's operating environment. The significance of these modalities in Edge AI applications lies in their capacity to facilitate real-time data acquisition, ultimately enhancing device responsiveness and intelligence. By leveraging them, Edge AI systems can acquire valuable insights from the surrounding environment, enabling more informed decision-making and empowering devices to interact intelligently with their surroundings.

In terms of deployment, using sensing modalities at the edge enables devices to attain heightened situational awareness and make informed decisions, even in rapidly evolving environments. This attribute holds significant importance in applications that require quick response times, such as autonomous vehicles, industrial automation, and healthcare. Moreover, sensing modalities empower devices to capture data that would otherwise be difficult or impossible to collect. Environmental sensors, for example, provide insights into air quality and temperature fluctuations, while motion sensors track real-time object movement. By leveraging this diverse range of data, decision-making processes are enhanced, system performance is optimized, and energy efficiency is promoted. Here is a brief overview of the most common sensing modalities and their application value.

Vision-Based Sensing Modalities

Vision-based sensing modalities, employing sensors and cameras, play a vital role in Edge AI applications by capturing visual data and enabling sophisticated image and video processing techniques. These modalities encompass a range of technologies, including cameras and image sensors, which capture visual data from the environment. 

By utilizing optical principles, these sensors convert incoming light into electrical signals, which are subsequently processed to generate images or videos. Cameras, the most common vision-based sensors, employ lenses to focus light onto an image sensor, such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor.

A CMOS image sensor. Image credit: Association of Advanced Automation.

In the context of Edge AI, image and video processing techniques are crucial components of vision-based sensing. Image processing algorithms analyze individual images, while video processing techniques operate on sequences of images. These techniques encompass image enhancement, filtering, feature extraction, and compression. They are applied to individual frames or sequences of frames to detect patterns or anomalies in the data. Notably, deep learning techniques, such as convolutional neural networks, have demonstrated remarkable performance in image and video processing tasks.

With diverse applications in Edge AI, vision-based sensing modalities serve various purposes. Surveillance, for instance, relies on cameras to monitor specific areas, utilizing object detection algorithms to identify and track objects of interest, such as people or vehicles. Facial recognition, another vital application, employs cameras for face identification, serving security and access control purposes. Additionally, object tracking utilizes cameras to monitor object movement in specific areas, finding applications in industrial process monitoring and vehicle tracking in traffic. Vision-based sensing modalities also involve object recognition and detection algorithms for identifying objects in visual data. These algorithms utilize features extracted from the data to match objects with known models or templates. Object recognition algorithms are applied in facial recognition applications, while object detection algorithms find use in surveillance and object-tracking tasks. 

Audio-Based Sensing Modalities

Audio-based sensing modalities are essential for acquiring and analyzing audio data from the surrounding environment. The primary devices used for audio data acquisition are audio sensors and microphones, which come in various types, such as dynamic, condenser, and ribbon microphones, and can be customized to fit specific use cases. These components capture sound waves in the environment and convert them into electrical signals for further processing. 

Audio sensors can vary in design and characteristics, including omnidirectional or directional sensitivity, frequency response, and signal-to-noise ratio. On the other hand, microphones encompass a wide range of types, such as condenser microphones, dynamic microphones, and MEMS (Microelectromechanical Systems) microphones, each offering distinct advantages for different applications.

Another significant application of audio-based sensing is acoustic monitoring. By strategically placing microphones in diverse environments, acoustic monitoring systems can detect and analyze sounds for multiple purposes. In industrial settings, acoustic monitoring plays a crucial role in identifying machine malfunctions or anomalous sounds, enabling predictive maintenance and improving operational efficiency. Similarly, in healthcare, acoustic monitoring systems assist in patient monitoring by detecting events such as snoring, coughing, or abnormal respiratory sounds.

An audio-based sensing system.

In addition, sound event detection represents another notable application of audio-based sensing in Edge AI. Sound event detection systems leverage machine-learning algorithms to analyze audio signals and recognize specific sounds or events. By training on labeled data, these systems can identify various acoustic events, including glass breaking, gunshots, sirens, explosions, and more. This technology is essential for public safety and security applications, enabling real-time detection and response to potentially dangerous situations. 

For instance, in smart cities, sound event detection can be integrated with surveillance systems to automatically identify and alert authorities of incidents like explosions, facilitating faster emergency response and enhancing overall security. Similarly, in industrial settings, these systems can detect abnormal sounds that indicate equipment failure, helping prevent accidents and improve operational safety. By harnessing the power of audio-based sensing and AI, sound event detection edge AI systems could efficiently contribute to developing intelligent and proactive environments, fostering more secure societies.

Environmental Sensing Modalities

Environmental sensing modalities play a vital role in Edge AI systems by enabling devices to acquire and analyze data related to environmental conditions. Collecting and analyzing environmental data involves acquiring and processing sensor data, where environmental sensors generate continuous or periodic measurements that are logged and timestamped. 

Data can be collected using wired or wireless communication protocols and transmitted for analysis to a central processing unit or a cloud server. Techniques such as statistical methods, machine learning algorithms, and anomaly detection are commonly employed to extract insights, identify patterns, and detect anomalies in environmental data.

Environmental sensing modalities encompass diverse sensors designed to measure parameters such as temperature, humidity, and gas concentrations. These sensors play a pivotal role in Edge AI, offering a wide array of applications across various domains. In addition to contributing to climate control and energy management systems, environmental sensors serve as crucial safety measures, triggering alerts and facilitating proactive responses.

Data Collection Simplified: How Sparkfun Enables Data Logging It’s no secret that Edge computing requires accurate data collection. While SparkFun is not a software company, our aim is to develop a robust hardware ecosystem of Qwiic-compatible embedded sensors that enable Edge developers and partners to focus on innovating in this space. We want to see smarter sensors that enable users to collect and process data quickly for rapid development at the Edge.  As Kirk Benell, SparkFun CTO comments, “For over a decade, SparkFun has designed and built products aimed at simplifying the process of embedded device data logging, with the latest offerings delivering low-power, turnkey sensor connectivity to a growing set of SparkFun Qwiic boards."  A perfect example of how SparkFun simplifies the process of collecting data is showcased in our newest major product launch - the SparkFun DataLogger IoT - 9DoF. The DataLogger IoT is a data logger that comes preprogrammed to automatically log IMU, GPS, and various pressure, humidity, and distance sensors. It was specifically designed for users who just need to capture a lot of data to a CSV or JSON file and get back to their larger project. Save the data to a microSD card or send it wirelessly to your preferred Internet of Things (IoT) service! SparkFun will continue developing more capable devices, and we are excited to contribute to the Edge AI revolution! The SparkFun DataLogger IoT. Image credit: Sparkfun

In industrial monitoring and workplace safety, environmental sensors monitor equipment, detect toxic or flammable gases, and ensure a safe work environment. In precision agriculture, they provide vital information for optimizing crop growth and disease prevention, enabling efficient irrigation management, early pest detection, and resource optimization. 

One prominent example is the Arduino Edge Control by Arduino, which is used for controlling and monitoring remote environments while deploying AI on the edge. This board is suitable for a diverse range of applications requiring smart control in remote locations, such as agricultural automation, precision farming, and environmental and geophysical sensing. The Edge Control can enhance farming productivity by collecting data and making accurate data-driven decisions. It is especially useful in the control of highly sensitive farming processes such as hydroponics, aquaponics, and mushroom cultivation, which require precise temperature and humidity control. The components of the Edge Control board are designed with a higher temperature range to deal with harsh environments. It can be connected to a variety of sensors to collect real-time data on weather conditions, soil quality, CO2 levels, pest infections, and crop growth.

The Arduino Edge Control in an automated greenhouse application.

Integrating environmental sensing into Edge AI systems makes real-time monitoring, predictive maintenance, and proactive safety measures possible, contributing to enhanced operational efficiency and worker well-being.

Other Sensing Modalities

In addition to vision-based, audio-based, and environmental sensing modalities, several other components play a meaningful role in data collection and analysis within Edge AI applications. These diverse modalities expand the capabilities of Edge AI systems by capturing valuable information from the physical world. Touch sensors, for instance, enable the detection and interpretation of tactile interactions, allowing devices to perceive touch inputs and provide haptic feedback. Capacitive touch sensors, resistive touch sensors, and surface acoustic wave sensors are commonly used in touchscreens and touch-sensitive surfaces, allowing for interactive and intuitive user interfaces. In Edge AI, touch sensors find applications in smart devices, human-machine interfaces, and interactive displays.

On the other hand, pressure sensors play a vital role in measuring the force applied to a surface, providing valuable insights into various systems and applications. In automotive settings, pressure sensors are employed for monitoring tire pressure, ensuring optimal vehicle performance and enhanced safety by detecting deviations from recommended pressure levels. In medical devices, pressure sensors measure fluid pressure accurately, enabling precise monitoring of blood pressure, respiratory functions, and intracranial pressure in healthcare settings. Additionally, in Heating, Ventilation, and Air Conditioning (HVAC) systems, pressure sensors are crucial for detecting changes in building pressure, allowing for efficient control and adjustment of ventilation systems to maintain optimal indoor air quality and energy efficiency. Integrating pressure sensors into Edge AI applications empowers real-time monitoring, data-driven decision-making, and predictive maintenance, leading to enhanced operational efficiency and improved user experience. 

Finally, proximity sensors are highly versatile devices capable of detecting the presence or absence of objects in close proximity. They leverage various technologies, such as infrared, ultrasonic, and capacitive sensing, to achieve accurate detection. In the context of Edge AI applications, proximity sensors offer a multitude of possibilities. For instance, they can be employed in security systems to detect the presence of individuals or objects, enhancing overall surveillance effectiveness. In parking systems, proximity sensors play a key role in monitoring the proximity of vehicles, enabling efficient and automated parking assistance. Moreover, in touchless interfaces, proximity sensors are instrumental in detecting the approach of a hand or any other object, facilitating intuitive and hygienic user interactions.

Warehouse robotics use spatial AI that includes multiple sensing modalities to navigate between obstacles. Image credit: Gideon.ai.

The diverse range of sensing modalities discussed in this chapter offers unique characteristics that enable their suitability for specific applications in Edge AI. Combining sensing modalities with other types of sensors further enhances the capabilities of Edge AI applications, empowering them to provide intuitive and responsive interactions with the physical world. This integration opens up new possibilities for innovation and advancement in various fields, ranging from smart homes and healthcare to industrial automation and transportation.

The 2023 Edge AI Technology Report

The guide to understanding the state of the art in hardware & software in Edge AI.

Click through to read each of the report's chapters. 

Report Introduction

Chapter I: Overview of Industries and Application Use Cases

Chapter II: Advantages of Edge AI

Chapter III: Edge AI Platforms

Chapter IV: Hardware and Software Selection

Chapter V: Tiny ML

Chapter VI: Edge AI Algorithms

Chapter VII: Sensing Modalities

Chapter VIII: Case Studies

Chapter IX: Challenges of Edge AI

Chapter X: The Future of Edge AI and Conclusion

Report sponsors: