Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Validating User Engagement and Effectiveness of Training Simulations: A mixed-methods approach informed by embodied cognition and psychophysiological measures
Stockholm University, Faculty of Social Sciences, Department of Computer and Systems Sciences. University of Colombo School of Computing.
2015 (English)Doctoral thesis, comprehensive summary (Other academic)Alternative title
Validering av användarengagemang och effektivitet hos träningssimulatorer : En kombinerad metodansats informerad av kroppslig kognition och psyko-fysiologiska mått (Swedish)
Abstract [en]

Simulation-based training has gained widespread attention recently as a response to drawbacks associated with traditional training approaches, such as high training costs (instructors, equipment, etc.), high risks (e.g. pilot training), and ethical issues (e.g. medical training), as well as a lack of availability of certain training environments (e.g. space exploration). Apart from their target training domains, many of aspects of simulations differ, such as their degree of physical realism (fidelity), scenarios (e.g. story), and pedagogical aspects (e.g. after-action reviews and collaborative learning). Among those aspects, designers have mostly focused on developing high-fidelity simulations with the expectation of increasing the effectiveness of training. However, some authors suggest that the above belief is a myth as researchers have failed to identify a linear relationship between the (physical) fidelity and training effectiveness of simulations.  Most researchers have therefore evaluated the correspondence between the behaviours of trainees in both real world and simulated contexts, however, the existing methods of simulation validation using behavioural measures have a number of drawbacks, such as the fact that they do not address certain complex phenomena of skills acquisition.

Bridging the above knowledge gap, this research reports on empirical investigations using an improved methodology for validating training simulations. This research includes an investigation of the user experience of trainees, with respect to the acceptance of virtual scenarios provoking a similar psychophysiological response as in real world scenarios, and the training potential of simulations with respect to the positive transfer of training from a simulator to real world operational contexts. The most prominent features of the proposed methodology include the use of psychophysiological measures in addition to traditional behavioural measures and the use of natural (quasi-) experiments. Moreover, its conceptual framework was influenced by contemporary theories in cognitive science (e.g. constructivism and embodied cognition). The results of this research have several important theoretical and methodological implications, involving, for example, the dependency of the effectiveness of simulations on the perceived realism of trainees, which is more embodied than has been predicted by previous researchers, and the requirement of several different types/levels of adaptive training experience, depending on the type of trainee.

Abstract [sv]

Träning i simulatorer har på senare år fått ökad uppmärksamhet som en respons på problem och svårigheter förknippade med traditionella träningsansatser, såsom höga kostnader (instruktörer och utrustning, etc.), hög risk (t.ex. träning av piloter), och etiska aspekter (t.ex. träning av kirurger), likaväl som avsaknaden av träningsmöjligheter och miljöer (t.ex. forskning om rymden). Bortsett från vad som specifikt tränas så skiljer sig simuleringar åt i ett flertal olika aspekter såsom fysisk realism (eng. fidelity), scenarier (handling) och pedagogiska aspekter (t.ex. genomgång efter övning och kollaborativt lärande).  Bland dessa aspekter så har designers ofta fokuserat att utveckla simuleringar med hög realism med förväntningen att detta ska göra träningen mer effektiv. Litteraturen antyder dock att denna föreställning inte stämmer och att de flesta simuleringar med hög realism inte har lyckats uppnå denna målsättning. En slutsats är därför att det finns ett behov av metoder som kan validera potentialen hos simuleringar avsedda att stödja träning – redan innan dessa används.

Enligt litteraturen så är utbildningspotentialen hos en simulering starkt kopplad till hur väl den psykologiska effekten en simulering har, stämmer överens med en verklig upplevelse. Forskning har emellertid identifierat ett flertal svagheter hos existerande ansatser för att validera simuleringar; de är oftast baserade på prestations- och/eller subjektiva mätningar; de har fokuserat en eller ett fåtal psykologiska aspekter; och de bygger på traditionella teorier. Baserat på resultat från studier av en kör-simulator presenteras och föreslås i denna avhandling ett förbättrat ramverk för utvärdering. De mest centrala egenskaperna hos det föreslagna ramverket inbegriper användandet av psyko-fysiologiska mått tillsammans med mer traditionella mått; det konceptuella ramverket bygger på samtida teoretiska ansatser (tex konstruktivism och kroppslig kognition); samt användandet av fält (kvasi-) experiment. Utöver uppnåendet av uppsatta mål för forskningen så har resultaten ett flertal teoretiska och metodologiska implikationer. Bland dessa återfinns beroendet mellan effektiviteten hos en simulering och den upplevelse av realitet som de tränade har, vilken är mer grundläggande än vad som rapporterats i tidigare forskning, samt kravet på flera och olika typer av anpassning av träningsupplevelse för den tränade för att förhöja potentialen hos träningssimulatorer.

Place, publisher, year, edition, pages
Stockholm: Department of Computer and Systems Sciences, Stockholm University , 2015. , 125 p.
Series
Report Series / Department of Computer & Systems Sciences, ISSN 1101-8526 ; 15-015
Keyword [en]
training simulators, simulation validation, psychological fidelity, psychophysiological measures, embodied cognition, electroencephalography (EEG), galvanic skin response (GSR)
Keyword [sv]
träningssimulatorer, validering av simuleringar, psykologisk realism, psyko-fysiologiska mått, kroppslig kognition, elektroencefalografi (EEG), galvanisk hudrespons (GSR)
National Category
Human Computer Interaction
Research subject
Computer and Systems Sciences
Identifiers
URN: urn:nbn:se:su:diva-122546ISBN: 978-91-7649-305-2 (print)OAI: oai:DiVA.org:su-122546DiVA: diva2:867201
Public defence
2015-12-09, L30, NOD-huset, Borgarfjordsgatan 12, Kista, 13:00 (English)
Opponent
Supervisors
Projects
SIDA Funded National e-Learning Centre Project at the University of Colombo School of Computing, Sri Lanka
Available from: 2015-11-17 Created: 2015-11-04 Last updated: 2015-11-09Bibliographically approved
List of papers
1. Determining the Psychological Involvement in Multimedia Interactions
Open this publication in new window or tab >>Determining the Psychological Involvement in Multimedia Interactions
2009 (English)In: The International Journal on Advances in ICT for Emerging Regions, ISSN 1800-4156, Vol. 2, no 1Article in journal (Refereed) Published
Abstract [en]

Human computer interaction (HCI) is currently aimed at the design of interactive computer applications for human use while preventing user frustration. When considering the nature of modern computer applications, such as e-learning systems and computer games, it appears that human involvement cannot be improved only by using traditional approaches, such as nice user interfaces. For a pleasant human involvement, these computer applications require that the computers should have the ability to naturally adapt to their users and this requires the computers to have the ability to recognize user emotions. For recognizing emotions currently most preferred research approach is aimed at facial expression based emotion recognition, which seems to have many limitations. Therefore, in this paper, we propose a method to determine the psychological involvement of a human during a multimedia interaction session using the eye movement activity and arousal evaluation. In our approach we use a low cost hardware/software combination, which determines eye movement activity based on electrooculogram (EOG) signals and the level of arousal using galvanic skin response (GSR) signals. The results obtained using six individuals show that the nature of involvement can be recognized using these affect signals as optimal levels and distracted conditions.

Keyword
Arousal, Attention, Cognition, Emotion, EOG, Eye Movement Activity, GSR, HCI, Human Involvement, Multimedia Interactions
National Category
Computer Science
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-122475 (URN)10.4038/icter.v2i1.1400 (DOI)
Available from: 2015-11-03 Created: 2015-11-03 Last updated: 2017-12-01Bibliographically approved
2. Game Interaction State Graphs for Evaluation of User Engagement in Explorative and Experience-based Training Games
Open this publication in new window or tab >>Game Interaction State Graphs for Evaluation of User Engagement in Explorative and Experience-based Training Games
Show others...
2010 (English)In: 2010 International Conference on Advances in ICT for Emerging Regions (ICTer), 2010, 40-44 p.Conference paper, Published paper (Refereed)
Abstract [en]

There is an increasing interest to use computer games for non-traditional education, such as for training purposes. For training education, simulators are considered as offering more realistic learning environments to experience situations that are similar to real world. This type of learning is more beneficial for practicing critical situations which are difficult or impossible in real world training, for instance experience the consequences of unsafe driving. However, the effectiveness of simulation-based learning of this nature is dependent upon the learner's engagement and explorative behaviour. Most current learner evaluation systems are unable to capture this type of learning. Therefore, in this paper we introduce the concept of game interaction state graphs (GISGs) to capture the engagement in explorative and experience-based training tasks. These graphs are constructed based on rules which capture psychologically significant learner behaviours and situations. Simple variables reflecting game state and learner's controller actions provide the ingredients to the rules. This approach eliminates the complexity involved with other similar approaches, such as constructing a full-fledged cognitive model for the learner. GISGs, at minimum, can be used to evaluate the explorative behaviour, the training performance and personal preferences of a learner.

Keyword
driving simulator training, engagement, experience-based systems, game interaction, serious games
National Category
Information Science
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-51910 (URN)10.1109/ICTER.2010.5643272 (DOI)978-1-4244-9030-4 (ISBN)
Conference
ICTer 2010: International Conference on Advances in ICT for Emerging Regions, Sept. 29 - Oct. 1, 2010
Available from: 2011-01-12 Created: 2011-01-12 Last updated: 2015-11-11Bibliographically approved
3. Assessing Performance Competence in Training Games
Open this publication in new window or tab >>Assessing Performance Competence in Training Games
Show others...
2011 (English)In: Affective Computing and Intelligent Interaction: Proceedings, Part II / [ed] Sidney D’Mello, Arthur Graesser, Björn Schuller, Jean-Claude Martin, Springer Berlin/Heidelberg, 2011, 518-527 p.Conference paper, Published paper (Refereed)
Abstract [en]

In-process assessment of trainee learners in game-based simulators is a challenging activity. This typically involves human instructor time and cost, and does not scale to the one tutor per learner vision of computer-based learning. Moreover, evaluation from a human instructor is often subjective and comparisons between learners are not accurate. Therefore, in this paper, we propose an automated, formula-driven quantitative evaluation method for assessing performance competence in serious training games. Our proposed method has been empirically validated in a game-based driving simulator using 7 subjects and 13 sessions, and accuracy up to 90.25% has been achieved when compared to an existing qualitative method. We believe that by incorporating quantitative evaluation methods like these future training games could be enriched with more meaningful feedback and adaptive game-play so as to better monitor and support player motivation, engagement and learning performance.

Place, publisher, year, edition, pages
Springer Berlin/Heidelberg, 2011
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 6975
Keyword
Serious games, Performance, Evaluation, Motivation, Driver training
National Category
Information Science
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-65043 (URN)10.1007/978-3-642-24571-8_65 (DOI)978-3-642-24570-1 (ISBN)978-3-642-24571-8 (ISBN)
Conference
Fourth International Conference, ACII 2011, Memphis, TN, USA, October 9–12, 2011
Available from: 2011-12-01 Created: 2011-12-01 Last updated: 2015-11-11Bibliographically approved
4. Affective Realism of Animated Films in the Development of Simulation-Based Tutoring Systems
Open this publication in new window or tab >>Affective Realism of Animated Films in the Development of Simulation-Based Tutoring Systems
Show others...
2013 (English)In: International Journal of Distance Education Technologies, ISSN 1539-3100, E-ISSN 1539-3119, Vol. 11, no 2, 96-109 p.Article in journal (Refereed) Published
Abstract [en]

This paper presents a study focused on comparing real actors based scenarios and animated characters based scenarios with respect to their similarity in evoking psychophysiological activity for certain events by measuring galvanic skin response (GSR). In the experiment, one group (n=11) watched the real actors’ film whereas another group (n=7) watched the animated film, which had the same story and dialogue as the real actors’ film. The results have shown that there is no significant difference in the skin conductance response (SCR) scores between the two groups; however, responses significantly differ when SCR amplitudes are taken into account. Moreover, Pearson’s correlation reported as high as over 80% correlation between the two groups’ SCRs for certain time intervals. The authors believe that this finding is of general importance for the domain of simulation-based tutoring systems in development of and decisions regarding use of animated characters based scenarios.

Keyword
Affective Realism, Animated Scenarios, Emotion, Game Simulator, Psychophysiology, Skin
National Category
Information Systems
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-122526 (URN)
Note

Special issue on Emotional Intelligence for Online Learning.

Available from: 2015-11-03 Created: 2015-11-03 Last updated: 2017-12-01Bibliographically approved
5. Comparing Expert Driving Behavior in Real World and Simulator Contexts
Open this publication in new window or tab >>Comparing Expert Driving Behavior in Real World and Simulator Contexts
Show others...
2013 (English)In: International Journal of Computer Games Technology, ISSN 1687-7047, E-ISSN 1687-7055, Vol. 2013, 891431Article in journal (Refereed) Published
Abstract [en]

Computer games are increasingly used for purposes beyond mere entertainment, and current hi-tech simulators can provide quite, naturalistic contexts for purposes such as traffic education. One of the critical concerns in this area is the validity or transferability of acquired skills from a simulator to the real world context. In this paper, we present our work in which we compared driving in the real world with that in the simulator at two levels, that is, by using performance measures alone, and by combining psychophysiological measures with performance measures. For our study, we gathered data using questionnaires as well as by logging vehicle dynamics, environmental conditions, video data, and users' psychophysiological measurements. For the analysis, we used several novel approaches such as scatter plots to visualize driving tasks of different contexts and to obtain vigilance estimators from electroencephalographic (EEG) data in order to obtain important results about the differences between the driving in the two contexts. Our belief is that both experimental procedures and findings of our experiment are very important to the field of serious games concerning how to evaluate the fitness of driving simulators and measure driving performance.

National Category
Information Systems
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-95582 (URN)10.1155/2013/891431 (DOI)
Available from: 2013-10-31 Created: 2013-10-31 Last updated: 2017-12-06Bibliographically approved
6. Comparing Expert and Novice Driving Behaviour in a Driving Simulator
Open this publication in new window or tab >>Comparing Expert and Novice Driving Behaviour in a Driving Simulator
Show others...
2013 (English)In: ID&A Interaction design & architecture(s), ISSN 1826-9745, E-ISSN 2283-2998, no 19, 115-131 p.Article in journal (Refereed) Published
Abstract [en]

This paper presents a study focused on comparing driving behavior of expert and novice drivers in a mid-range driving simulator with the intention of evaluating the validity of driving simulators for driver training. For the investigation, measurements of performance, psychophysiological measurements, and self-reported user experience under different conditions of driving tracks and driving sessions were analyzed. We calculated correlations between quantitative and qualitative measures to enhance the reliability of the findings. The experiment was conducted involving 14 experienced drivers and 17 novice drivers. The results indicate that driving behaviors of expert and novice drivers differ from each other in several ways but it heavily depends on the characteristics of the task. Moreover, our belief is that the analytical framework proposed in this paper can be used as a tool for selecting appropriate driving tasks as well as for evaluating driving performance in driving simulators.

Keyword
Educational game design, simulation-based training, playful learning, emotion in games, driving simulator, simulator validity, evaluation, driving performance, psychophysiology, EEG, Emotiv EPOC, player experience
National Category
Information Systems
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-110958 (URN)
External cooperation:
Available from: 2014-12-19 Created: 2014-12-19 Last updated: 2017-12-05Bibliographically approved

Open Access in DiVA

fulltext(1964 kB)168 downloads
File information
File name FULLTEXT02.pdfFile size 1964 kBChecksum SHA-512
3138b6e1faebe43cab8366112e9a08437aa0f3e2dc1c37ec8bfd4a128513f94d561f5170834b561beb3c9677d2393bbd79437f454193a00fa94ada2133cf59a4
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Ekanayake, Hiran B.
By organisation
Department of Computer and Systems Sciences
Human Computer Interaction

Search outside of DiVA

GoogleGoogle Scholar
Total: 168 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 596 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf