Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
User Modeling for Adaptive Virtual Reality Experiences: Personalization from Behavioral and Physiological Time Series
Stockholm University, Faculty of Social Sciences, Department of Computer and Systems Sciences.ORCID iD: 0000-0002-6047-2793
2023 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Research in human-computer interaction (HCI) has focused on designing technological systems that serve a beneficial purpose, offer intuitive interfaces, and adapt to a person's expectations, goals, and abilities. Nearly all digital services available in our daily lives have personalization capabilities, mainly due to the ubiquity of mobile devices and the progress that has been made in machine learning (ML) algorithms. Web, desktop, and smartphone applications inherently gather metrics from the system and users' activity to improve the attractiveness of their products and services. Meanwhile, the hardware, input interfaces, and algorithms currently under development guide the designs of upcoming interactive systems that may become pervasive in society, such as immersive virtual reality (VR) or physiological wearable sensing systems. These technological advancements have led to multiple questions regarding the personalization capabilities of modern visualization mediums and fine-grained body measurements. How does immersive VR enable new pathways for understanding the context in which a user interacts with a system? Can the user's behavioral and physiological data improve the accuracy of ML models estimating human factors? What are the challenges and risks of designing personalized systems that transcend current setups with a 2D-based display, touchscreen, keyboard, and mouse? This thesis provides insights into how human behavior and body responses can be incorporated into immersive VR applications to enable personalized adaptations in 3D virtual environments. The papers contribute frameworks and algorithms that harness multimodal time-series data and state-of-the-art ML classifiers in user-centered VR applications. The multimodal data include motion trajectories and body measurements from the user's brain and heart, which are used to capture responses elicited by virtual experiences. The ML algorithms exploit the temporality of large datasets to perform automatic data analysis and provide interpretable explanations about signals that correlate with the user's skill level or emotional states. Ultimately, this thesis provides an outlook on how the combination of recent hardware and algorithms may unlock unprecedented opportunities to create 3D experiences tailored to each user and to help them attain specific goals with VR-based systems, framed using the overarching topic of context-aware systems and discussing the ethical risks related to personalization based on behavioral and physiological time-series data in immersive VR experiences.

Abstract [sv]

Forskning inom människa-datorinteraktion (på engelska, human-computer interaction -- HCI) har fokuserat på att designa tekniska system som tjänar ett fördelaktigt syfte, erbjuder intuitiva gränssnitt och anpassar sig till en persons förväntningar, mål och förmågor. Nästan alla digitala tjänster som är tillgängliga i vårt dagliga liv har personaliseringsmöjligheter, främst på grund av de mobila enheternas överallt och de framsteg som har gjorts inom maskininlärning (ML) algoritmer. Webb-, skrivbords- och smartphoneapplikationer samlar i sig mätvärden från systemet och användarnas aktivitet för att förbättra attraktiviteten för deras produkter och tjänster. Samtidigt styr hårdvaran, ingångsgränssnitten och algoritmerna som för närvarande är under utveckling designen av kommande interaktiva system som kan bli genomträngande i samhället, såsom immersive virtual reality (VR) eller fysiologiska bärbara avkänningssystem. Dessa tekniska framsteg har lett till flera frågor angående personaliseringsmöjligheterna hos moderna visualiseringsmedier och finkorniga kroppsmått. Hur möjliggör uppslukande VR nya vägar för att förstå sammanhanget där en användare interagerar med ett system? Kan användarens beteendemässiga och fysiologiska data förbättra noggrannheten hos ML-modeller som uppskattar mänskliga faktorer? Vilka är utmaningarna och riskerna med att designa personliga system som överskrider nuvarande inställningar med en 2D-baserad skärm, pekskärm, tangentbord och mus? Den här avhandlingen ger insikter i hur mänskligt beteende och kroppssvar kan införlivas i uppslukande VR-applikationer för att möjliggöra personliga anpassningar i virtuella 3D-miljöer. Artiklarna bidrar med ramverk och algoritmer som utnyttjar multimodala tidsseriedata och toppmoderna ML-klassificerare i användarcentrerade VR-applikationer. De multimodala data inkluderar rörelsebanor och kroppsmätningar från användarens hjärna och hjärta, som används för att fånga svar som framkallas av virtuella upplevelser. ML-algoritmerna utnyttjar temporaliteten hos stora datamängder för att utföra automatisk dataanalys och ge tolkningsbara förklaringar om signaler som korrelerar med användarens kompetensnivå eller känslomässiga tillstånd. I slutändan ger denna avhandling en syn på hur kombinationen av nyare hårdvara och algoritmer kan låsa upp oöverträffade möjligheter att skapa 3D-upplevelser skräddarsydda för varje användare och hjälpa dem att uppnå specifika mål med VR-baserade system, inramade med hjälp av det övergripande ämnet sammanhangsmedvetna system och diskutera de etiska riskerna relaterade till personalisering baserat på beteendemässiga och fysiologiska tidsseriedata i uppslukande VR-upplevelser.

Place, publisher, year, edition, pages
Stockholm: Department of Computer and Systems Sciences, Stockholm University , 2023. , p. 84
Series
Report Series / Department of Computer & Systems Sciences, ISSN 1101-8526 ; 23-010
Keywords [en]
virtual reality, VR, machine learning, ML, user modeling, personalization, wearable, heart rate, HRV, biosensors, movement, behavioral, user experience, UX, spatial computing, metaverse, extended reality, XR
National Category
Human Computer Interaction Computer Sciences
Research subject
Computer and Systems Sciences
Identifiers
URN: urn:nbn:se:su:diva-222210ISBN: 978-91-8014-540-4 (print)ISBN: 978-91-8014-541-1 (electronic)OAI: oai:DiVA.org:su-222210DiVA, id: diva2:1804014
Public defence
2023-11-28, Lilla Hörsalen, NOD-huset, Borgarfjordsgatan 12, Kista, 09:00 (English)
Opponent
Supervisors
Available from: 2023-11-02 Created: 2023-10-11 Last updated: 2023-10-24Bibliographically approved
List of papers
1. Implementation of Mobile-Based Real-Time Heart Rate Variability Detection for Personalized Healthcare
Open this publication in new window or tab >>Implementation of Mobile-Based Real-Time Heart Rate Variability Detection for Personalized Healthcare
2019 (English)In: 2019 International Conference on Data Mining Workshops (ICDMW): Proceedings / [ed] Panagiotis Papapetrou, Xueqi Cheng, Qing He, IEEE, 2019, p. 838-846Conference paper, Published paper (Refereed)
Abstract [en]

The ubiquity of wearable devices together with areas like internet of things, big data and machine learning have promoted the development of solutions for personalized healthcare that use digital sensors. However, there is a lack of an implemented framework that is technically feasible, easily scalable and that provides meaningful variables to be used in applications for translational medicine. This paper describes the implementation and early evaluation of a physiological sensing tool that collects and processes photoplethysmography data from a wearable smartwatch to calculate heart rate variability in real-time. A technical open-source framework is outlined, involving mobile devices for collection of heart rate data, feature extraction and execution of data mining or machine learning algorithms that ultimately deliver mobile health interventions tailored to the users. Eleven volunteers participated in the empirical evaluation that was carried out using an existing mobile virtual reality application for mental health and under controlled slow-paced breathing exercises. The results validated the feasibility of implementation of the proposed framework in the stages of signal acquisition and real-time calculation of heart rate variability (HRV). The analysis of data regarding packet loss, peak detection and overall system performance provided considerations to enhance the real-time calculation of HRV features. Further studies are planned to validate all the stages of the proposed framework.

Place, publisher, year, edition, pages
IEEE, 2019
Series
IEEE International Conference on Data Mining workshops, ISSN 2375-9232, E-ISSN 2375-9259
Keywords
Heart rate variability, HRV, personalized health, framework, mobile, real-time, smartwatch
National Category
Computer Sciences
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-178338 (URN)10.1109/ICDMW.2019.00123 (DOI)
Conference
19th IEEE International Conference on Data Mining Workshops (ICDMW), Beijing, China, 8–11 November, 2019
Available from: 2020-01-24 Created: 2020-01-24 Last updated: 2023-10-11Bibliographically approved
2. A Psychophysiological Model of Firearms Training in Police Officers: A Virtual Reality Experiment for Biocybernetic Adaptation
Open this publication in new window or tab >>A Psychophysiological Model of Firearms Training in Police Officers: A Virtual Reality Experiment for Biocybernetic Adaptation
2020 (English)In: Frontiers in Psychology, E-ISSN 1664-1078, Vol. 11, article id 683Article in journal (Refereed) Published
Abstract [en]

Crucial elements for police firearms training include mastering very specific psychophysiological responses associated with controlled breathing while shooting. Under high-stress situations, the shooter is affected by responses of the sympathetic nervous system that can impact respiration. This research focuses on how frontal oscillatory brainwaves and cardiovascular responses of trained police officers (N = 10) are affected during a virtual reality (VR) firearms training routine. We present data from an experimental study wherein shooters were interacting in a VR-based training simulator designed to elicit psychophysiological changes under easy, moderate and frustrating difficulties. Outcome measures in this experiment include electroencephalographic and heart rate variability (HRV) parameters, as well as performance metrics from the VR simulator. Results revealed that specific frontal areas of the brain elicited different responses during resting states when compared with active shooting in the VR simulator. Moreover, sympathetic signatures were found in the HRV parameters (both time and frequency) reflecting similar differences. Based on the experimental findings, we propose a psychophysiological model to aid the design of a biocybernetic adaptation layer that creates real-time modulations in simulation difficulty based on targeted physiological responses.

Keywords
biocybernetic adaptation, virtual reality, psychophysiological model, electroencephalography, heart rate variability, simulation, firearms training
National Category
Psychology Computer and Information Sciences
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-181843 (URN)10.3389/fpsyg.2020.00683 (DOI)000531233200001 ()32373026 (PubMedID)
Available from: 2020-06-01 Created: 2020-06-01 Last updated: 2023-10-11Bibliographically approved
3. Excite-O-Meter: Software Framework to Integrate Heart Activity in Virtual Reality
Open this publication in new window or tab >>Excite-O-Meter: Software Framework to Integrate Heart Activity in Virtual Reality
2021 (English)In: 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE , 2021, p. 357-366Conference paper, Published paper (Refereed)
Abstract [en]

Bodily signals can complement subjective and behavioral measures to analyze human factors, such as user engagement or stress, when interacting with virtual reality (VR) environments. Enabling widespread use of (also the real-time analysis) of bodily signals in VR applications could be a powerful method to design more user-centric, personalized VR experiences. However, technical and scientific challenges (e.g., cost of research-grade sensing devices, required coding skills, expert knowledge needed to interpret the data) complicate the integration of bodily data in existing interactive applications. This paper presents the design, development, and evaluation of an open-source software framework named Excite-O-Meter. It allows existing VR applications to integrate, record, analyze, and visualize bodily signals from wearable sensors, with the example of cardiac activity (heart rate and its variability) from the chest strap Polar H10. Survey responses from 58 potential users determined the design requirements for the framework. Two tests evaluated the framework and setup in terms of data acquisition/analysis and data quality. Finally, we present an example experiment that shows how our tool can be an easy-to-use and scientifically validated tool for researchers, hobbyists, or game designers to integrate bodily signals in VR applications.

Place, publisher, year, edition, pages
IEEE, 2021
National Category
Information Systems
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-200478 (URN)10.1109/ISMAR52148.2021.00052 (DOI)978-1-6654-0158-6 (ISBN)
Conference
International Symposium on Mixed and Augmented Reality - ISMAR, Bari, Italy, October 4-8, 2021
Available from: 2022-01-05 Created: 2022-01-05 Last updated: 2023-10-11Bibliographically approved
4. Effective Classification of Head Motion Trajectories in Virtual Reality using Time-Series Methods
Open this publication in new window or tab >>Effective Classification of Head Motion Trajectories in Virtual Reality using Time-Series Methods
2021 (English)In: 2021 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), IEEE , 2021, p. 38-46Conference paper, Published paper (Refereed)
Abstract [en]

In this paper, we present a method to classify head motion trajectories using recent time-series methods. The analysis of motion data with machine learning is a common technique to solve problems in Virtual Reality (VR), such as adaptive rendering or user behavioral modeling. Motion data are initially collected as time series, but they are usually transformed into tabular features compatible with traditional feature-based classifiers. Data mining research has proposed several time-series classifiers that can directly exploit the temporal relationship of the data without requiring manual feature extraction. Nevertheless, the effectiveness of these time-series methods still requires validation on real-life problem domains. Therefore, this paper demonstrates how a pipeline that combines a recent time-series classifier with two rotation space representations (quaternion and Euler) can successfully analyze head motion in VR applications. We test the proposed method on two public datasets containing head rotations, resulting in higher prediction accuracy than other feature-based and time-series classifiers. We also discuss some limitations, guidelines for future work, and concluding remarks.

Place, publisher, year, edition, pages
IEEE, 2021
Keywords
virtual reality, 360 videos, kinematics, motion, feature, time series, classification
National Category
Information Systems
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-200473 (URN)10.1109/AIVR52153.2021.00015 (DOI)978-1-6654-3226-9 (ISBN)
Conference
International Conference on Artificial Intelligence and Virtual Reality (AIVR), 2021
Available from: 2022-01-05 Created: 2022-01-05 Last updated: 2023-10-11Bibliographically approved
5. Personalized Feature Importance Ranking for Affect Recognition From Behavioral and Physiological Data
Open this publication in new window or tab >>Personalized Feature Importance Ranking for Affect Recognition From Behavioral and Physiological Data
2023 (English)In: IEEE Transactions on Games (TG), ISSN 2475-1502Article in journal (Refereed) Epub ahead of print
Abstract [en]

Designing affect-based personalized technology involves dealing with large datasets. Machine learning (ML) algorithms are employed to predict affect and similar human factors from in-game metrics, behavioral patterns, or physiological responses. The classification performance is usually presented as a global point estimate without providing user-specific interpretations. This approach is incompatible with effective personalization in games because it disregards the variability of body responses between players. This paper proposes a methodology to classify subjective human factors from large multimodal data. A public VR dataset (CEAP-360VR) was used to extensively compare three ML classifiers and five feature importance techniques. The produced models could reduce the original feature space by 82% (from 113 to 20 features) without compromising predictive performance (F1 score). A random forest (RF) using forward sequential feature selection (fSFS) yielded the best prediction of binary valence (F1=0.761) and arousal (F1=0.748). Finally, feature importance rankings are discussed with emphasis on global and user-specific patterns that may improve affect recognition. The proposed methodology is envisioned to help game designers and researchers create customized user-centric games and VR experiences inferring possible explanations from multimodal datasets.

Keywords
Games, Feature extraction, Behavioral sciences, Physiology, Classification algorithms, Solid modeling, Skin
National Category
Computer Sciences
Research subject
Computer and Systems Sciences
Identifiers
urn:nbn:se:su:diva-216061 (URN)10.1109/TG.2023.3263070 (DOI)2-s2.0-85151511513 (Scopus ID)
Available from: 2023-03-31 Created: 2023-03-31 Last updated: 2023-10-11

Open Access in DiVA

User Modeling for Adaptive Virtual Reality Experiences: Personalization from Behavioral and Physiological Time Series(7232 kB)403 downloads
File information
File name FULLTEXT01.pdfFile size 7232 kBChecksum SHA-512
7ea22d3d7f1e4cd72aa13834e6474bb78db45137e77ec2678c049b32bbda67f9df4eea083de87e390df765ec6bc30d92d91205e10b3f22a3efe99386a5e6636c
Type fulltextMimetype application/pdf

Authority records

Quintero, Luis

Search in DiVA

By author/editor
Quintero, Luis
By organisation
Department of Computer and Systems Sciences
Human Computer InteractionComputer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 404 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1966 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf