Artificial intelligence (AI) can reliably detect emotions based on facial expressions in psychotherapeutic situations. These are the findings of a feasibility study by researchers from the Faculty of Psychology and the University Psychiatric Clinics (UPK) at the University of Basel. The AI system is also able to reliably predict therapeutic success in patients with borderline personality pathology.
The face is a mirror for a person’s emotional state. The interpretation of facial expressions as part of psychotherapy or psychotherapeutic research, for example, is a very effective way of characterizing how a person is feeling in that particular moment. Back in the 1970s, psychologist Paul Ekmann developed a standardized coding system to assign basic emotions such as happiness, disgust or sadness to a facial expression in an image or video sequence.
“Ekman’s system is very widespread, and represents a standard in psychological emotion research,” says Dr. Martin Steppan, psychologist at the Faculty of Psychology at the University of Basel.
But the process of analyzing and interpreting recorded facial expressions as part of research projects or psychotherapy is extremely time-consuming, which is why psychiatry specialists often use less reliable, indirect methods such as skin conductance measurements, which can also be a measure of emotional arousal.
“We wanted to find out whether AI systems can reliably determine the emotional states of patients in video recordings,” says Martin Steppan, who developed the study together with emeritus Professor Klaus Schmeck, Dr. Ronan Zimmermann and Dr. Lukas Fürer from the UPK. The researchers published their findings in the Psychopathology journal.
No facial expression can escape AI
The researchers used freely available artificial neural networks that were trained in the detection of six basic emotions (happiness, surprise, anger, disgust, sadness and fear) using over 30,000 facial photos. This AI system then analyzed video data from therapy sessions with a total of 23 patients with borderline personality pathology at the Center for Scientific Computing at the University of Basel. The high-performance computer had to process over 950 hours of video recordings for this study.
The results were astonishing: statistical comparisons between the analysis of three trained therapists and the AI system showed a remarkable level of agreement. The AI system assessed the facial expressions as reliably as a human but was also able to detect even the most fleeting of emotions within the millisecond range, such as a brief smile or expression of disgust.
