Multimodal LLM-based approach for detecting Disengagement from Task Goal (DTG) in Math problem solving

Nidhi Nasiar, Stanford University
Ryan S. Baker
Jaclyn L. Ocumpaugh
Zeyneb S. Sarioglan
Caitlin Mills
Samuel Rhodes
Date
Publication
Proceedings of the International Conference on Educational Data Mining (EDM 2026)

Learner disengagement has been widely studied and modeled in math learning environments to enable timely supports and promote engagement. Recent advances in multimodal large language models (MLLMs) offer new opportunities for building such detectors. This study investigates the potential of an MLLM-based approach to detect disengagement from the task goal (DTG) in screen-recording videos of students solving math problems in CueThink. DTG is operationalized as actions unrelated to the intended learning task. We develop and validate a DTG detector using LLaVA-Video-7B-Qwen2(Large Language and Vision Assistant) using various prompting strategies, and analyze themes across the few model errors. We also examine associations between DTG behavior and learning gains, as well as students' math beliefs, anxiety, and metacognition. Results indicate that the model, using role assignment and chain-of-thought prompting, identifies DTG behavior with good performance (κ = 0.71, precision = 77.6\%, recall = 76.4\%). Performance limitations that emerged related to low-fidelity data, limited contextual knowledge, and following instructions. DTG behavior was marginally negatively correlated with normalized learning gains later in the academic year (winter to spring), but not during the earlier fall-to-winter period. There were no significant correlations to beliefs or anxiety.

Search Repository

SCALE Authors