publications

2024

1. 3D Gaze Tracking for Studying Collaborative Interactions in Mixed-Reality Environments

   Eduardo Davalos, Joyce H Fonteles, Yike Zhang, and 1 more author

   2024

   Abs PDF

   This study presents a novel framework for 3D gaze tracking tailored for mixed-reality settings, aimed at enhancing joint attention and collaborative efforts in team-based scenarios. Conventional gaze tracking, often limited by monocular cameras and traditional eye-tracking apparatus, struggles with simultaneous data synchronization and analysis from multiple participants in group contexts. Our proposed framework leverages state-of-the-art computer vision and machine learning techniques to overcome these obstacles, enabling precise 3D gaze estimation without dependence on specialized hardware or complex data fusion. Utilizing facial recognition and deep learning, the framework achieves real-time, tracking of gaze patterns across several individuals, addressing common depth estimation errors, and ensuring spatial and identity consistency within the dataset. Empirical results demonstrate the accuracy and reliability of our method in group environments. This provides mechanisms for significant advances in behavior and interaction analysis in educational and professional training applications in dynamic and unstructured environments.

2. Monocular Microscope to CT Registration using Pose Estimation of the Incus for Augmented Reality Cochlear Implant Surgery

   Yike Zhang, Eduardo Davalos, Dingjie Su, and 2 more authors

   2024

   arXiv
3. A First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments

   Joyce Fonteles, Eduardo Davalos, Ashwin T. S., and 9 more authors

   2024

   arXiv

2023

1. ChimeraPy: A Scientific Distributed Streaming Framework for Real-time Multimodal Data Retrieval and Processing

   Eduardo Davalos, Umesh Timalsina, Yike Zhang, and 3 more authors

   In 2023 IEEE International Conference on Big Data (BigData) , 2023

   PDF
2. Identifying Gaze Behavior Evolution via Temporal Fully-Weighted Scanpath Graphs

   Eduardo Davalos, Caleb Vatral, Clayton Cohn, and 5 more authors

   In LAK23: 13th International Learning Analytics and Knowledge Conference , , Arlington, TX, USA, , 2023

   Abs PDF

   Eye-tracking technology has expanded our ability to quantitatively measure human perception. This rich data source has been widely used to characterize human behavior and cognition. However, eye-tracking analysis has been limited in its applicability, as contextualizing gaze to environmental artifacts is non-trivial. Moreover, the temporal evolution of gaze behavior through open-ended environments where learners are alternating between tasks often remains unclear. In this paper, we propose temporal fully-weighted scanpath graphs as a novel representation of gaze behavior and combine it with a clustering scheme to obtain high-level gaze summaries that can be mapped to cognitive tasks via network metrics and cluster mean graphs. In a case study with nurse simulation-based team training, our approach was able to explain changes in gaze behavior with respect to key events during the simulation. By identifying cognitive tasks via gaze behavior, learners’ strategies can be evaluated to create online performance metrics and personalized feedback.

3. Prediction of Students’ Self-confidence Using Multimodal Features in an Experiential Nurse Training Environment

   Caleb Vatral, Madison Lee, Clayton Cohn, and 3 more authors

   In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky , 2023

   Abs PDF

   Simulation-based experiential learning environments used in nurse training programs offer numerous advantages, including the opportunity for students to increase their self-confidence through deliberate repeated practice in a safe and controlled environment. However, measuring and monitoring students’ self-confidence is challenging due to its subjective nature. In this work, we show that students’ self-confidence can be predicted using multimodal data collected from the training environment. By extracting features from student eye gaze and speech patterns and combining them as inputs into a single regression model, we show that students’ self-rated confidence can be predicted with high accuracy. Such predictive models may be utilized as part of a larger assessment framework designed to give instructors additional tools to support and improve student learning and patient outcomes.

4. A Tale of Two Nurses: Studying Groupwork in Nurse Training by Analyzing Taskwork Roles, Social Interactions, and Self-Efficacy

   Caleb Vatral, Clayton Cohn, Eduardo Davalos, and 5 more authors

   Computer-Supported Collaborative Learning Conference, CSCL, 2023

   Abs PDF

   Modern healthcare requires the coordination of a team of professionals with complementary skillsets. To help facilitate teamwork, healthcare professionals, such as nurses, undergo rigorous training of their clinical skills in team settings. In this paper, we analyze a mixed-reality, simulation-based training exercise involving three nurses in a hospital room. We perform multimodal interaction analysis to contrast strategies used in two cases where the patient expressed doubts about their medical care. By analyzing these strategies and comparing them to the student nurses’ self-reflections, we show connections among the nurses’ clinical roles, their self-efficacy, and their teamwork.

2022

1. Adaptive Scaffolding to Support Strategic Learning in an Open-Ended Learning Environment

   Anabil Munshi, Gautam Biswas, Eduardo Davalos, and 2 more authors

   2022

   Abs PDF

   An important goal for intelligent learning environments is to provide adaptive support for learners. This paper presents an adaptive scaffolding framework designed to support strategic learning in Betty’s Brain, an open-ended learning-by-teaching environment. The framework includes an online strategy detector that identifies students’ sub-optimal use of cognitivemetacognitive strategies, and a conversational adaptive feedback mechanism to help students overcome their difficulties. We ran a pilot study with undergraduate students to determine how well they applied the adaptive scaffolds provided by our framework. Our qualitative case study analysis used students’ activity logs, one-on-one interactions with researchers, and eye-gaze patterns on the Betty’s Brain screens, to infer their learning behaviors and performance. Findings suggest that some scaffolds helped students develop more effective and strategic behaviors to overcome their difficulties. The results also suggest a roadmap to improve scaffold design and better support long-term learning strategies in Betty’s Brain.

2. Using the DiCoT framework for integrated multimodal analysis in mixed-reality training environments

   C. Vatral, G. Biswas, C. Cohn, and 2 more authors

   Frontiers in Artificial Intelligence, 2022

   PDF

2021

1. FastPoseCNN: Real-Time Monocular Category-Level Pose and Size Estimation Framework

   Eduardo Davalos, and Merhan Aminian

   2021

   Abs PDF

   The primary focus of this paper is the development of a framework for pose and size estimation of unseen objects given a single RGB image - all in real-time. In 2019, [1] proposed the first category-level pose and size estimation framework alongside two novel datasets called CAMERA and REAL. However, the novel method proposed by [1] was restricted from practical use because of its long inference time (2-4 fps). Their approach’s inference had significant delays because they used the computationally expensive MaskedRCNN framework and Umeyama algorithm. To optimize our method and yield real-time results, our framework uses the efficient ResNet-FPN framework alongside decoupling the translation, rotation, and size regression problem by using distinct decoders. Moreover, our methodology performs pose and size estimation in a global context - i.e., estimating the involved parameters of all captured objects in the image all at once. We perform extensive testing to fully compare the performance in terms of precision and speed to demonstrate the capability of our method - FastPoseCNN.