2023 Edge AI Technology Report. Chapter III: Edge AI Platforms

Edge AI, empowered by the recent advancements in Artificial Intelligence, is driving significant shifts in today's technology landscape. By enabling computation near the data source, Edge AI enhances responsiveness, boosts security and privacy, promotes scalability, enables distributed computing, and improves cost efficiency.

Wevolver has partnered with industry experts, researchers, and tech providers to create a detailed report on the current state of Edge AI. This document covers its technical aspects, applications, challenges, and future trends. It merges practical and technical insights from industry professionals, helping readers understand and navigate the evolving Edge AI landscape.

Understanding the scope of Edge AI platforms

Selecting and using the right Edge AI platform requires careful consideration of various factors. It is important to evaluate the project requirements, including latency, data processing capabilities, power consumption, size, weight, and heat dissipation. For example, if the project demands real-time processing and low latency, a platform with high processing capabilities and low latency is preferred. On the other hand, if power consumption is a concern, it is recommended to opt for an energy-efficient platform.

In addition to performance considerations, the level of expertise required to work with the platform should be taken into account. Some platforms are designed to be user-friendly, making them suitable for developers with varying levels of expertise. Conversely, other platforms may require specialized knowledge and skills. 

The cost of the platform is also an important consideration. While some platforms are free, others may require licensing fees or additional costs for specific features. Economic feasibility should be evaluated based on the project's budget and resource constraints. 

Furthermore, the level of support and ecosystem compatibility the platform provides should be assessed. A platform with a robust support system and a thriving ecosystem can offer valuable resources, documentation, and community support, enabling developers to leverage the platform more effectively.

There is a wide array of Edge AI platforms available today, catering to the diverse needs of developers and enabling the deployment of machine learning models on edge devices. These platforms offer unique features and advantages instrumental in building cutting-edge AI applications. Among the prominent platforms are PyTorch Mobile, OpenVINO, NVIDIA Jetson, BrainChip Akida™, Caffe2, and MXNet. Each platform brings its own strengths and capabilities to the table, providing developers with a range of options. By carefully evaluating the project requirements and considering factors such as performance, expertise level, support, ecosystem, and cost, developers can make an informed decision about the most suitable platform for their specific Edge AI projects. In the subsequent sections, we will delve into each of these platforms, offering a brief overview and highlighting the key attributes that make them valuable tools for Edge AI development.

TensorFlow Lite

TensorFlow Lite is a robust and versatile platform designed for deploying machine learning models on Edge devices. It offers low latency, cross-platform compatibility, user-friendly features, and advanced capabilities, making it the preferred choice for developers. It enables quick and efficient inference on a wide range of platforms, such as mobile devices and microcontrollers. Through benchmarking tests, it has demonstrated impressive performance, achieving a median inference latency of 1.3 milliseconds on the Google Pixel 4 smartphone and 2.5 milliseconds on the Raspberry Pi 4 microcontroller board. 

To further enhance its capabilities, TensorFlow Lite also supports hardware accelerators like the Google Coral Edge TPU, which effectively reduces latency. This platform empowers developers with diverse data processing capabilities, including image and audio recognition, natural language processing, and time-series analysis. It offers a collection of pre-trained models for these tasks and provides tools for training and optimizing custom models. TensorFlow Lite is adaptable to various input and output formats, such as image, audio, and sensor data, allowing it to cater to different use cases.

An important aspect of TensorFlow Lite is its accessibility. It is an open-source platform available for free, providing developers with a cost-effective solution. However, it is worth noting that using hardware accelerators like the Google Coral Edge TPU may incur additional expenses due to their higher cost. 

When deploying deep learning models on mobile and edge devices, power consumption becomes a critical consideration. TensorFlow Lite addresses this concern by being lightweight and optimized specifically for these devices. It has a small memory footprint and can seamlessly integrate into mobile apps and microcontroller projects. In addition, it pays attention to heat dissipation, a crucial factor for devices with size and power constraints. By prioritizing efficiency and lightweight design, TensorFlow Lite minimizes heat generation, ensuring optimal performance. 

The platform offers a comprehensive range of development tools and APIs, enabling developers to train and deploy their models across various platforms. From a technical perspective, TensorFlow Lite is designed to be flexible and extensible. Developers can customize and fine-tune their machine-learning models to suit their requirements. Furthermore, TensorFlow Lite supports multiple programming languages, including Python, C++, and Java, enhancing its versatility and ease of use.

The Akida edge neural network processor from BrainChip is also supported under TensorFlow using MetaTF, which adds functions to optimize the network for execution on the high-performance Akida hardware. As Peter van der Made, CTO and founder at BrainChip, explains, “One outstanding feature of Akida is that it executes the entire network internally without dependence on the CPU’s resources, thereby significantly increasing computing power at the edge.” Its event-based processing method is fast and power-efficient, running most networks within the microwatt or milliwatt range. The on-chip learning method allows for adding additional functions after the device is trained.

PyTorch Mobile

PyTorch is a widely adopted open-source machine learning framework that has recently extended its capabilities to include mobile devices through PyTorch Mobile. Its ease of use, flexibility, and support for state-of-the-art models make it a compelling choice for developers. PyTorch Mobile is optimized for speed and efficiency, offering low latency that enables its deployment on resource-constrained, low-power devices. 

According to PyTorch, models built using this platform can achieve latencies of less than ten milliseconds on select mobile devices, which is sufficiently fast for many real-time applications. However, the actual latency of a PyTorch Mobile model depends on various factors, including the hardware platform, model complexity, and input data size.

PyTorch Mobile empowers developers with a broad range of data processing capabilities, including image recognition and natural language processing. The costs associated with using PyTorch Mobile vary depending on the hardware platform, model size, complexity, and specific use case requirements.

Power consumption is crucial when implementing machine learning on edge devices, mainly due to the limited battery life in many such devices. PyTorch Mobile addresses this concern with a small memory footprint, enabling efficient execution on low-power devices with restricted resources. 

Additionally, PyTorch Mobile supports various hardware platforms, such as CPUs, GPUs, and DSPs, enabling developers to optimize their machine-learning models for specific hardware architectures and minimize power consumption.

Designed for lightweight and efficient operation, PyTorch Mobile has a compact binary size of approximately 4 MB, making it well-suited for mobile devices. It can also run on diverse hardware platforms, including microcontrollers with limited storage and processing capabilities, thereby catering to small form factors and weight constraints. The amount of heat dissipation experienced will depend on the specific hardware platform employed for deployment, as well as the size and complexity of the model.

Finally, PyTorch Mobile provides optimized models and tools for efficient and low-power inference, helping reduce the heat generated by the platform. It supports a variety of frameworks and programming languages, such as Python, offering adaptability to different use cases.

OpenVINO

OpenVINO (Open Visual Inference and Neural Network Optimization) is an agile platform that offers numerous benefits for deploying deep learning models. Its exceptional performance, support for multiple hardware platforms, and flexibility make it a preferred choice. Developed by Intel, OpenVINO enables efficient deployment across diverse platforms, including CPUs, GPUs, and FPGAs. Based on benchmarking tests conducted by Intel, OpenVINO achieves inference latencies as low as five milliseconds on specific hardware configurations, making it suitable for real-time applications like object detection and image recognition.

The platform provides optimized libraries and tools for popular deep learning frameworks such as TensorFlow, PyTorch, and Caffe, empowering developers to train and deploy models on various platforms. The costs of using OpenVINO depend on factors like model size, complexity, and the hardware platform employed.

In terms of power consumption, OpenVINO delivers optimized inference performance, resulting in lower power consumption than traditional CPU-based solutions. It also supports hardware accelerators like FPGAs, known for their low power consumption. OpenVINO has a relatively small footprint, and its size varies based on the selected hardware and software components for deployment. The platform's support for hardware accelerators reduces heat dissipation compared to traditional CPU-based solutions.

OpenVINO offers seamless integration with multiple programming languages, including C++, Python, and Java. It also integrates with other Intel tools and technologies, such as the Intel Distribution of OpenVINO Toolkit and the Intel System Studio.

Finally, OpenVINO achieves high performance through a combination of model optimization techniques like model quantization and compression. These techniques reduce the size and complexity of deep learning models, resulting in enhanced efficiency and faster execution on edge devices.

NVIDIA Jetson

NVIDIA Jetson offers a multitude of advantages, including high processing power, low latency, and support for various neural networks and development tools. One of the primary strengths of NVIDIA Jetson for edge AI lies in its exceptional processing power. The Jetson devices are equipped with NVIDIA GPUs specifically designed for high-performance computing in deep learning applications. These GPUs are optimized for parallel computing, enabling them to handle substantial data volumes within short timeframes efficiently. This makes Jetson devices particularly well-suited for real-time data processing requirements in applications like video analytics and autonomous robotics.

Another notable benefit of NVIDIA Jetson is its low latency. These devices feature high-speed interfaces that facilitate swift communication between the processor and other components. As a result, data can be processed rapidly, minimizing the time needed for decision-making in edge AI applications. Furthermore, Jetson devices find utility across a broad spectrum of edge AI use cases, encompassing smart cameras, drones, and industrial automation.

It is worth noting that NVIDIA Jetson devices exhibit relatively higher power consumption due to their significantly enhanced data processing capabilities. Consequently, additional cooling measures are often necessary to manage the generated heat effectively.

NVIDIA Jetson provides a diverse selection of neural networks and development tools. This ecosystem empowers developers with an extensive array of options to facilitate their edge AI projects effectively.

Regarding costs, the price of using NVIDIA Jetson devices can vary depending on the chosen model and specifications. Nonetheless, these devices are designed to be cost-effective solutions for edge AI applications. For instance, the NVIDIA Jetson Nano Developer Kit combines high performance with a competitive price, enhancing accessibility for developers.

Additionally, NVIDIA Jetson devices are compact and lightweight, making them ideal for deployment in space-constrained environments where size and weight considerations are crucial factors to address.

Edge Impulse

Edge Impulse is an end-to-end platform designed to create, train, and deploy machine learning models on edge hardware. Compared to other Edge AI platforms, Edge Impulse offers distinct advantages in terms of ease of use, flexibility, and device compatibility.

One key advantage of Edge Impulse is its user-friendly nature. The platform is designed to cater to users with varying levels of experience in machine learning and embedded systems. With its intuitive web-based interface, users can effortlessly create, train, and deploy machine learning models without requiring specialized hardware or software. This simplicity makes Edge Impulse an appealing option for enterprises and developers who seek quick prototyping and testing capabilities.

Flexibility is another noteworthy aspect of Edge Impulse. The platform supports a wide array of devices, including microcontrollers, single-board computers, and mobile devices. This versatility empowers developers to build AI-powered applications for diverse edge devices. Latency is dependent on the specific deployment device and the complexity of the machine learning model being employed.

Edge Impulse provides comprehensive data processing capabilities, encompassing data collection, preprocessing, and feature extraction. It also supports various sensor types, ensuring compatibility with a range of data sources. The platform offers multiple pricing plans for professional product development and a free option for developers working on individual projects.

The ability to shrink a model’s memory consumption while retaining its accuracy and the resulting efficiency achieved are core considerations in Edge Impulse's design, enabling inference to be performed on low-power devices. The platform seamlessly integrates with a wide range of edge devices, including microcontrollers, single-board computers, and mobile devices. The size of the deployment depends on the chosen device and the dimensions of the machine-learning model. 

Integration with popular development tools and frameworks, such as Arduino, TensorFlow, and PyTorch, is a seamless process with Edge Impulse. This ensures a smooth incorporation of machine-learning capabilities into existing workflows, streamlining development processes for users and organizations.

The Edge Impulse FOMO (Fast Objects, More Objects) enables microcontrollers to perform object detection while running faster and consuming far less RAM. Image credit: Hackster.io.

Caffe2

Caffe2 is an open-source deep learning framework that plays a significant role in developing Edge AI applications. Its prominent advantage lies in its high performance, enabling rapid and accurate processing of large datasets. Caffe2 is meticulously optimized for both CPUs and GPUs, and its compatibility with distributed computing environments ensures effortless scalability. This flexibility makes it adaptable to various computing scenarios.

Caffe2 stands out for its low inference latency, making it particularly suitable for real-time edge AI applications. Benchmarked tests have demonstrated its impressive performance on both CPU and GPU platforms. For instance, Facebook AI Research conducted a benchmark test that showcased mean inference times of 14.4 ms on a mobile CPU and 4.8 ms on a GPU.

Efficient processing of large datasets is made possible by Caffe2's support for distributed training and automatic parallelization of operations. Its versatility also extends to data formats, with comprehensive support for images, audio, and video. Consequently, Caffe2 is an excellent choice for training and deploying large-scale machine-learning models.

Being an open-source framework, Caffe2 is freely available, making it an affordable option for developers and organizations seeking to build edge AI applications without incurring substantial software costs. However, it's important to note that Caffe2's power consumption tends to be higher due to its extensive use of GPUs.

Caffe2 boasts a compact code footprint, making it easy to deploy on edge devices with limited storage capacity. The library size of Caffe2 is less than 3 MB, making it an ideal choice for resource-constrained devices. Furthermore, Caffe2 is optimized to minimize heat generation during inference, making it suitable for edge devices operating in demanding environments. By efficiently dissipating heat, Caffe2 maintains peak performance even in high-temperature conditions.

Lastly, Caffe2 offers exceptional flexibility, allowing seamless integration with a wide range of hardware platforms, programming languages, and software frameworks. This versatility ensures compatibility and ease of use across diverse development environments.

MXNet

MXNet has gained significant popularity in machine learning due to its exceptional flexibility, scalability, and high-performance capabilities. One of MXNet's notable advantages is its extensive support for multiple programming languages, including Python, C++, and Julia. This feature empowers developers to work with their language of choice and facilitates seamless integration of MXNet into existing software systems.

Efficient data processing is another compelling aspect of MXNet. The framework's data processing engine is meticulously designed to handle large datasets efficiently. MXNet also supports data and model parallelism, enabling developers to scale their models across multiple GPUs or machines. Additionally, MXNet provides a rich collection of pre-built models and modules that accelerate the development of Edge AI applications.

MXNet boasts low latency and high-performance capabilities. Its advanced memory management system and deep learning optimization algorithms enable the swift execution of complex models. MXNet's focus on performance ensures that inference and training tasks are carried out with remarkable efficiency.

From a cost perspective, MXNet is an open-source framework, making it freely available for use and distribution. This characteristic makes MXNet an affordable option for organizations seeking to leverage machine learning capabilities without incurring substantial expenses.

Regarding power consumption, MXNet's optimized algorithms and memory management system contribute to its energy efficiency. This attribute is crucial for Edge AI applications that rely on battery or renewable energy sources.

MXNet also excels in terms of size and weight. With its lightweight nature, MXNet can run efficiently on low-power devices like Raspberry Pi. This makes it an attractive choice for Edge AI applications that demand compact and portable systems.

Furthermore, MXNet's functional integration capabilities are noteworthy. It can seamlessly integrate with other deep learning frameworks like TensorFlow and PyTorch, allowing developers to leverage their preferred framework while still benefiting from MXNet's unique features and capabilities.

Edge AI platforms play a crucial role in enabling the development and deployment of machine learning models on edge devices. These are just a few examples of the diverse platforms available to developers. By carefully considering factors such as project requirements, performance capabilities, level of expertise, support, ecosystem compatibility, and cost, developers can select the most suitable platform for their specific Edge AI projects. It is important to remember that Edge AI is constantly evolving, and new platforms and technologies are continuously emerging. Staying updated with the latest advancements and conducting thorough evaluations will enable developers to harness the full potential of Edge AI platforms and drive innovation in this exciting field. With the right platform at their disposal, developers can unlock new possibilities and create intelligent edge applications that revolutionize industries and enhance our daily lives.

This chapter was contributed to by report sponsors BrainChip and Edge Impulse.

Gain a broader perspective on Edge AI by revisiting our comprehensive guide.

The 2023 Edge AI Technology Report

The guide to understanding the state of the art in hardware & software in Edge AI.

Click through to read each of the report's chapters. 

Report Introduction

Chapter I: Overview of Industries and Application Use Cases

Chapter II: Advantages of Edge AI

Chapter III: Edge AI Platforms

Chapter IV: Hardware and Software Selection

Chapter V: Tiny ML

Chapter VI: Edge AI Algorithms

Chapter VII: Sensing Modalities

Chapter VIII: Case Studies

Chapter IX: Challenges of Edge AI

Chapter X: The Future of Edge AI and Conclusion

The report sponsors: