Machine Learning Powered Home Scenes: A Blueprint for Intelligent Home Automation

Together with Digi-Key, a distributor of electronic components, we are creating a series of articles about the technologies that make IoT possible.  This second article examines the use of machine learning to power scenes in smart homes. Future articles dive into IoT security and connected transportation. 

For over two decades, smart homes and home automation applications have advanced at a rapid pace. Early smart homes comprised a limited number of connected devices and offered very basic automation functionalities, which were typically limited to controlling the status of smart devices. During the past decade, home automation functionalities have proliferated and evolved in terms of intelligence and sophistication. Specifically, smart homes platforms like Amazon Echo, Google Home and Apple’s Home Pod enable integrated and intelligent control of a larger number of devices. The latter can be configured in many ways to match the end-users’ preferences and needs.     

Introducing Smart Home Scenes

The multitude of home automation devices and functionalities in modern smart homes has given rise to the concept of “scenes”. Scenes enable smart home users to control multiple devices based on one or more scripts of well-defined actions. The latter are scheduled and executed in-line with the end-users’ preferences.  In practice, scenes consist of triggers and actions. 

Triggers are events that initiate configuration and control actions. For example, configuration actions can be triggered when the value of a sensor exceeds a threshold or based on the manual push of a button. Actions alter the status of one or more devices, which subsequently changes the state of the smart home.  The triggers and actions that comprise a scene are stored in the smart home platform as a configuration option. This option can be conveniently activated or deactivated at the end-user’s will. Smart home users can therefore configure multiple scenes and use them at different times.

Overall, scenes facilitate homeowners to control their smart devices flexibly and automatically. Using scenes, homeowners save precious time. Specifically, they no longer need to remember the different configuration options of their devices, allowing them to spend less or no time in implementing a series of configuration and control actions to ensure that their home environment matches their preferences.  Furthermore, scenes enable homeowners to control their home using their most preferred modality including remote controls, touch-panels and smartphone apps.

Prominent Scenes Examples  

Some of the most popular examples of smart home scenes include:

• The Morning Scene: This is typically a scene that configures a smart home to the owner’s desired morning state. It typically involves switching on certain lights, opening curtains, turning on the TV or radio in some rooms (e.g., the bathroom), and configuring certain appliances (e.g., toasters, boilers) to the desired state. Different instances of the morning scene can be implemented for different members of the household. For instance, different morning scenes can be implemented in the various bedrooms or bathrooms of the smart home.

• The Night Scene: This scene reverses the state of the morning scene and prepares the home for the night. For instance, some lights are switched off, the radio and the TV are turned off, and the home is locked up and secured.

• The Away Scene: In this case, the house is configured to the state preferred by homeowners when they are not present. As a prominent example, interior lights and media devices are turned off, shades are closed, exterior lights are switched on, and the alarm is set. 

• The In-Home Scene: This configures the home to the users’ desired state when one or more household members are in the house. The configuration of the scene can be personalized to the preferences of each individual member. For instance, a personalized configuration can define specific states for the lighting of the living room and the office, the ambient temperature of certain rooms, as well as the status of the TV and radio devices.

• The Party Scene:  This is about preparing the home for a social event like a party. Among other things, it can configure the lighting, the shades, the type and volume of the music, and the temperature towards creating a pleasant and effective environment for the event. The selected configuration can be activated with a click of a button on a remote control or smartphone app.  

Smart Home Scenes: Technology Enablers and Trends

The implementation and deployment scenes are enabled by cutting-edge smart home technologies such as intelligent sensors, cloud computing, and the Internet of Things (IoT). In particular:

• Intelligent Sensors and Actuators: Scenes leverage intelligent semiconductors and microsystems, which are configured remotely based on popular protocols like Bluetooth, WiFi, Zigbee, and KNX. Intelligent semiconductors are of two main types: Sensors that perceive the smart-home environment (e.g., a room’s occupancy patterns) and Actuators that implement automatic control functionalities (e.g., light control).
• IoT Controllers: Most scenes are configured, stored, deployed, and executed by IoT controllers i.e., gateway devices that control sensors and actuators from a single-entry point. IoT controllers support one or more networking protocols, which facilitate their seamless communication with sensors and actuators. 
• Cloud Computing: Modern smart home platforms and their IoT controllers are integrated with cloud computing infrastructures. Cloud integration enables smart home applications to benefit from the scalability, capacity, elasticity, and quality of service of cloud computing. For instance, the cloud facilitates the collection, management, and analysis of large volumes of data about the activity of the inhabitants. Likewise, cloud computing enables smart home applications to access storage and computing resources in flexible, scalable, and cost-effective ways.
• Edge Computing: In recent years, there is a shift of smart home data from the cloud to the edge of the network, i.e., close to end-users. Many data processing and data analytics applications are deployed close to the end-users. This reduces application latency as processing takes place in the local network.  Moreover, it economizes network bandwidth and boosts data protection as most data need not travel to the cloud. In most cases, edge computing is deployed in conjunction with cloud computing, i.e., real-time functionalities are deployed at the edge, while functionalities that ask for many data points and heavier processing are implemented in the cloud.

These technologies are the main technology enablers of smart home applications and scenes. Their evolution is driving improvements in automation, intelligence, and sustainability. Some of the top trends include:  

• Tiny Sensors: Sensors are gradually shrinking in size, which increases their versatility and pervasiveness. In this direction, modern sensing systems come with zero-touch configuration capabilities as well. Overall, the trend is towards pervasively deploying sensors without substantial effort for setting them up and operating them.      
• Energy Efficiency: Nowadays, consumers and enterprises are concerned with improving the environmental performance of their activities. This trend is extending to smart home applications, which are designed and engineered towards reducing CO2 emissions and optimizing environmental performance. For example, occupancy monitoring functionalities are increasingly used to optimize HVAC (Heating Ventilation and Air Conditioning) systems emissions. Likewise, smart homes are increasingly deploying energy-efficient sensors that live longer and exhibit optimal environmental performance. 
• Cyber-Physical Systems and Robotic Innovations: Future smart homes will benefit from Cyber-Physical Systems (CPS), i.e., specialized IoT devices that comprise both a cyber and a physical part. They will comprise various robotics devices and will be used to provide assistive functions to the tenants. Prominent examples include socially assistive robots, drones for parcel picking, as well as smart actuators that react to water leakages.  
• Strong Cybersecurity and Reliability: As smart home scenes evolve in number and functional sophistication, there will be a need for stronger cyber-security and reliability. The latter must be offered in a user-friendly way that enables even the most naïve users to configure and deploy security functions.  
• Increased Automation and Real-Time Operations: Scenes represent an important step in increasing smart home automation. In the coming years, scenes will become even more automated and intelligent. Moreover, they will comprise an increased number of real-time functions such as real-time monitoring of the status of rooms and real-time actuation in emergency cases (e.g., switching off gas or electricity upon the detection of earthquake signals).  

Sensors are becoming smaller and smarter: Image credit: Nest.

Nevertheless, the most disruptive innovations in the areas of smart home scenes will come from the deployment of Artificial Intelligence (AI), notably from Machine Learning (ML) which represents the most popular class of AI techniques. 

Smarter Scenes with Machine Learning and Artificial Intelligence

The blending of ML functions into smart home scenes is the next evolutionary step in home automation. Most of the state-of-the-art scenes are based on simple if-then-else rules over triggers and actions. This deterministic logic boosts automation, yet it provides very limited intelligence.  For example, simple rules do not consider the evolution of users’ habits and behaviors. ML alleviates these limitations by enabling scenes that learn the residents’ behavioral patterns and use them to optimize smart home operations. In this direction, ML algorithms can be trained using past data about the users’ activities in their home environment. 

Leveraging such algorithms, scenes can be enhanced in terms of dynamicity and intelligence. Specifically, rather than always applying the same configuration actions during a given time interval, ML-enabled scenes can dynamically change the actions in-line with the evolution of the residents’ behavior. Moreover, the use of ML can lift the limitations of choosing between a few pre-configured scenes for different times of the day (e.g., morning, afternoon, evening). This is because ML-based scenes can continually and dynamically adapt to the users’ habits and behavior, even when the latter changes several times during the day.     

Prominent examples of user-centric ML-based scenes include:

• Personalized Automated Home Ambience – Learning Users’ Preferences: In this case, the scenes automatically learn the users’ preferences about the ambient environment, i.e., about the lighting system, the operation of the air-conditioning, the configuration of the music player, the personal assistance services, the shades, the doors and more. Accordingly, they dynamically personalize the ambient environment to the users’ preferences.
• Occupancy Monitoring for Cost and Carbon Footprint Reduction: This type of scene leverages machine learning and computer vision recognition systems to extract occupancy patterns in the different rooms of the residential environment. Leveraging the identified patterns, the scenes configure the home’s operations (e.g., lighting, air conditioning) in ways that minimize energy costs and optimize CO2 emissions.
• Intelligent Recommendations of “Optimal” Scenes – Learning Scenes Dynamically: Leveraging past data about the users’ in-home behavior and the operation of the different devices, it is possible to construct scenes (i.e., smart home configurations) that economize on costs and optimize environmental performance. These configurations can be recommended to the end-users. Following the end-users approval, they can be stored and deployed in the smart home. In this case, ML enables the dynamic construction of “optimal” scenes rather than relying on pre-configured ones.  

Summary

The implementation of ML-based scenes hinges on the collection and processing of large volumes of data about operations, triggers, and actions, including users’ activities and behaviors within the residency. These data can enable the training of different types of ML algorithms, including traditional ML techniques (e.g., Decision Trees, Random Forests and SVM (Support Vector Machines)) and Deep Learning algorithms (e.g., Artificial Neural Networks (ANN) and Recurrent Neural Networks (RNN)). In cases where very large amounts of past data are available, ML techniques provide better performance and accuracy in understanding behavioral patterns and tailoring scenes’ configurations accordingly.

Scenes are nowadays an integral element of smart home intelligence. They boost home automation in ways that provide convenience to end-users and help them apply their desired configuration actions faster. Nevertheless, state-of-the-art scenes are quite simple and static. In this context, the emerging wave of ML-enabled scenes are set to disrupt the smart home applications landscape by enabling the development of more dynamic and intelligent scenes. 

ML is the next evolutionary step in smart home scenes. It will act as a catalyst for realizing the shift from existing rule-based scenes (Scenes 1.0) to the next generation of dynamic and highly personalized scenes (Scenes 2.0). Smart home deployers must therefore plan for integrating ML functions into their solutions to provide consumers with new levels of comfort and personalization.  

Article one set the foundations of a smart city infrastructure.

Article two looked into what makes a Smart home, smart.

Article three explored how Smart ML enables scenes for new seamless ways of interacting with smart home devices.

Article four explained how 5G and 6G will assist in delivering smarter communication.

Article five discussed about Smart Home security.

About the sponsor: Digi-Key

Digi-Key is one of the fastest-growing distributors of electronic components in the world. Since its founding in 1972, Digi-Key has been committed to offering the broadest selection of in-stock electronic components, as well as providing the best service possible to its customers, aiding engineers through the entire design process, from Prototype to Production®.