Change search
ReferencesLink to record
Permanent link

Direct link
Decision-making content of an agent affected by emotional feedback provided by capture of human’s emotions through a Bimodal System
Stockholm University, Faculty of Social Sciences, Department of Computer and Systems Sciences.
2015 (English)In: International Journal of Computer Science Issues, ISSN 1694-0784, E-ISSN 1694-0814, Vol. 12, no 6Article in journal (Refereed) Accepted
Abstract [en]

Affective computing allows for widening the view of the complex world in human-machine interaction through the comprehension of emotions, which allows an enriched coexistence of natural interactions between them. Corporal features such as facial expression, kinetics, structural components of the voice or vision, to mention just a few, provide us with valid information of how a human behaves. Among all the carriers of emotional information we may point out two, voice and facial gestures as holders of an ample potential for identifying emotions with a high degree of accuracy. This paper focuses on the development of a system that will track a human’s affective state using facial expressions and speech signals with the purpose of modifying the actions of an autonomous agent. The system uses a fusion of two baseline unimodal classifiers based on bayes Net giving rise to a multi-classifier. The union of the three classifiers forms a bimodal scheme of emotion classification. The outputs from the baseline unimodal classifiers are combined together through a probability fusion framework applied in the general multi-classifier. The system classifies six universal basic emotions using audiovisual data extracted from the eNTERFACE05 audiovisual emotion database. The emotional information obtained could provide an agent with the basis for taking an affective decision. It is shown by experimental results that the proposed system can detect emotions with good accuracy achieving the change of the emotional behavior of the agent faced with a human.

Place, publisher, year, edition, pages
2015. Vol. 12, no 6
Keyword [en]
Affective Computing, Machine Learning, Adversarial Risk Analysis, Broaden and Build Theory, Facial Expression Recognition, Speech Emotion Recognition, Detection of Emotional Information, Emotional self-regulation
National Category
Computer Science
Research subject
Computer and Systems Sciences
URN: urn:nbn:se:su:diva-122142OAI: diva2:865124
Available from: 2015-10-26 Created: 2015-10-26 Last updated: 2015-11-11Bibliographically approved
In thesis
1. Decisional-Emotional Support System for a Synthetic Agent: Influence of Emotions in Decision-Making Toward the Participation of Automata in Society
Open this publication in new window or tab >>Decisional-Emotional Support System for a Synthetic Agent: Influence of Emotions in Decision-Making Toward the Participation of Automata in Society
2015 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Emotion influences our actions, and this means that emotion has subjective decision value. Emotions, properly interpreted and understood, of those affected by decisions provide feedback to actions and, as such, serve as a basis for decisions. Accordingly, "affective computing" represents a wide range of technological opportunities toward the implementation of emotions to improve human-computer interaction, which also includes insights across a range of contexts of computational sciences into how we can design computer systems to communicate and recognize the emotional states provided by humans. Today, emotional systems such as software-only agents and embodied robots seem to improve every day at managing large volumes of information, and they remain emotionally incapable to read our feelings and react according to them. From a computational viewpoint, technology has made significant steps in determining how an emotional behavior model could be built; such a model is intended to be used for the purpose of intelligent assistance and support to humans. Human emotions are engines that allow people to generate useful responses to the current situation, taking into account the emotional states of others. Recovering the emotional cues emanating from the natural behavior of humans such as facial expressions and bodily kinetics could help to develop systems that allow recognition, interpretation, processing, simulation, and basing decisions on human emotions. Currently, there is a need to create emotional systems able to develop an emotional bond with users, reacting emotionally to encountered situations with the ability to help, assisting users to make their daily life easier. Handling emotions and their influence on decisions can improve the human-machine communication with a wider vision. The present thesis strives to provide an emotional architecture applicable for an agent, based on a group of decision-making models influenced by external emotional information provided by humans, acquired through a group of classification techniques from machine learning algorithms. The system can form positive bonds with the people it encounters when proceeding according to their emotional behavior. The agent embodied in the emotional architecture will interact with a user, facilitating their adoption in application areas such as caregiving to provide emotional support to the elderly. The agent's architecture uses an adversarial structure based on an Adversarial Risk Analysis framework with a decision analytic flavor that includes models forecasting a human's behavior and their impact on the surrounding environment. The agent perceives its environment and the actions performed by an individual, which constitute the resources needed to execute the agent's decision during the interaction. The agent's decision that is carried out from the adversarial structure is also affected by the information of emotional states provided by a classifiers-ensemble system, giving rise to a "decision with emotional connotation" included in the group of affective decisions. The performance of different well-known classifiers was compared in order to select the best result and build the ensemble system, based on feature selection methods that were introduced to predict the emotion. These methods are based on facial expression, bodily gestures, and speech, with satisfactory accuracy long before the final system.

Place, publisher, year, edition, pages
Stockholm: Department of Computer and Systems Sciences, Stockholm University, 2015. 146 p.
Report Series / Department of Computer & Systems Sciences, ISSN 1101-8526 ; 15-019
Affective Computing; Machine Learning; Adversarial Risk Analysis; Broaden and Build Theory; Facial Expression Recognition; Speech Emotion Recognition; Detection of Emotional Information; Emotional self-regulation
National Category
Human Computer Interaction
Research subject
Computer and Systems Sciences
urn:nbn:se:su:diva-122084 (URN)978-91-7649-291-8 (ISBN)
Public defence
2015-12-14, room L70, NOD Building, Borgarfjordsgatan 12, Kista, 13:00 (English)

At the time of the doctoral defense, the following paper was unpublished and had a status as follows: Paper 8: Accepted.

Available from: 2015-11-20 Created: 2015-10-23 Last updated: 2015-11-30Bibliographically approved

Open Access in DiVA

fulltext(1693 kB)82 downloads
File information
File name FULLTEXT01.pdfFile size 1693 kBChecksum SHA-512
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Guerrero, Javier
By organisation
Department of Computer and Systems Sciences
In the same journal
International Journal of Computer Science Issues
Computer Science

Search outside of DiVA

GoogleGoogle Scholar
Total: 82 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 394 hits
ReferencesLink to record
Permanent link

Direct link