Multimodal learning environments afford an abundance of observable interaction for informing learning science research. Among this interaction are haptic feedback and gaze, which, when analyzed via theoretical approaches such as embodied cognition, enactivism, and ecological dynamics, hold the potential to reveal important insights into the development of sensorimotor-facilitated conceptual learning. By analyzing how students’ sensorimotor behavior shifts over time in response to task-based perception, behavioral patterns are revealed that indicate phase shifts in conceptual understanding, reflecting the complex and dynamic nature of embodied content-based learning. In applying predictive machine learning techniques to telemetry data, multimodal learning analytics aims to contribute to research-informed education design, as well as improve educational support interventions such as adaptive tutoring systems.
The motivation for our research is to expand on previous approaches to gaze- and bimodal-based multimodal learning analytics in the context of an embodied design framework for learning proportionality. We seek to either reproduce or challenge expert-defined behavioral stages via unsupervised clustering using skip-grams and recurrent quantification analysis (RQA). Furthermore, we apply a long-short term memory recurrent neural network (LSTM RNN) in order to predict future behavior given a behavioral sequence, and, in a similar fashion, apply deep knowledge tracing (DKT) to predict future performance. Finally, we model this motor-coordination task as a skill-based situation, and explore the application of bayesian knowledge tracing (BKT) in predicting skill-based development over time. For each analytical technique, we report on our findings and discuss implications and limitations of our findings in educational interventions.
In collaboration with Julien Putz.
This was a final project for EDUC C260: Machine Learning in Education, taught by Professor Zachary Pardos in Spring 2022 at UC Berkeley School of Education.