AI recognizes emotions in real sports situations

This article was discussed in our Next Byte podcast. 

The full article will continue below.

Researchers from the Karlsruhe Institute of Technology (KIT) and the University of Duisburg Essen have managed to use computer-aided neural networks to precisely identify affective states based on the body language of tennis players during the game. For the first time, a model based on artificial intelligence (AI) was trained with data from real competition scenes. The study, published in the journal Knowledge-Based Systems, shows that AI can evaluate body language and emotions with similar accuracy to a human, but it also points to ethical aspects. ( DOI: 10.1016/j.knosys.2024.111856 )

For the study “Detection of emotional states from the expressive behavior of tennis players using convolutional neural networks,” researchers from sports science, software development and computer science at KIT and the University of Duisburg Essen have developed a special AI model. Using computer-based pattern recognition programs, they analyzed video recordings of tennis players during real competitions. 

68.9 percent hit rate

“Our model detects affective states with an accuracy of up to 68.9 percent. This is comparable to, and in some cases even superior to, the assessment by human observers as well as to previous automated methods,” explains study leader Professor Darko Jekauc from the Institute for Sport and Sports Science (IfSS) at KIT. 

An important unique feature of the study is that the project team did not train the AI system using simulated or posed situations, but rather using real scenes. To do this, the researchers recorded video sequences of 15 tennis players in a fixed setting. The focus was on body language when a point was lost or gained. The videos showed players with so-called cues, i.e. with cues such as a lowered head, arms thrown in the air in celebration, a hanging racket or differences in walking speed, which could be used to identify the players' affective state. 

Fed with this data, the AI learned to assign the body language signals to different affective reactions and to recognize whether a point was won (positive body language) or lost (negative body language). "Training in natural contexts represents a significant advance in recognizing real emotional states and enabling predictions in real-world scenarios," explains Jekauc.

Humans and machines recognize negative emotions better than positive ones

The research not only shows that AI algorithms could outperform human observers in emotion recognition in the future. The evaluation revealed another interesting aspect: both AI and humans are better at recognizing negative emotions. “This may be because they are easier to identify due to their clearer expressions,” explains Jekauc. “Psychological theories suggest that humans are evolutionarily more attuned to perceiving negative emotional expressions. For example, because it is crucial for the social context to defuse conflicts in a timely manner.”

Before use: Clarification of ethical aspects necessary

The study sees numerous areas of application for realistic emotion recognition in sport, from optimizing training methods, team dynamics and improving performance to preventing burnout. In addition, other areas could benefit from reliable early detection of emotional states, such as healthcare, educational institutions, customer service and automotive safety. 

“Although this technology offers significant benefits in the future, the potential risks associated with it must be taken into account, especially with regard to privacy and data misuse,” emphasizes Jekauc. “Our study was strictly based on existing ethical guidelines and data protection regulations. Even with a view to future applications of such technology in practice, it is essential to clarify ethical and legal questions in advance.” 

Original publication

Darko Jekauc, Diana Burkart, Julian Fritsch, Marc Hesenius, Ole Meyer, Saquib Sarfraz,
Rainer Schuhehagen: Recognizing affective states from the expressive behavior of tennis players using convolutional neural networks. Knowledge-Based Systems, Vol. 295, 2024. DOI: 10.1016/j.knosys.2024.111856