LLMs as Educational Analysts: Transforming Multimodal Data Traces into Actionable Reading Assessment Reports

Eduardo Davalos, Vanderbilt University

Yike Zhang, Vanderbilt University

Namrata Srivastaval

Jorge Alberto Salas, Vanderbilt University

Sara McFadden, Vanderbilt University

Sun-Joo Cho, Vanderbilt University

Gautam Biswas, Vanderbilt University

Amanda Goodwin, Vanderbilt University

Date

03/2025

AI Overview

Why it matters

Traditional reading assessments rely on unimodal test scores that overlook cognitive processes. While eye-tracking reveals reader behaviors, its complexity makes raw gaze metrics hard for teachers to interpret and use for instruction without structured feedback.

Study at a glance

Question: Can AI effectively translate complex student eye-tracking and reading data into useful assessment reports for teachers?
Setting and data: Fifth-grade students in a southeastern U.S. school completing three reading assessments (sample sizes N = 36, N = 36, and N = 46), and five educators.
Method: Unsupervised clustering of gaze data, AI-generated report synthesis and evaluation, and a teacher survey with Likert ratings.
AI tools: GPT-4o

Key findings

K-Means clustering grouped students into four distinct reading profiles, with 8 out of 10 gaze features differing significantly across clusters.
Surveyed teachers rated student behavior clusters as the most useful report section (average 4.20 out of 5), followed by outlier student identification (3.80).
An AI evaluator agent consistently gave generated reports an average rating of 4 out of 5 for clarity, relevance, and instructional support across assessments.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Reading assessments are essential for enhancing students' comprehension, yet many EdTech applications focus mainly on outcome-based metrics, providing limited insights into student behavior and cognition. This study investigates the use of multimodal data sources -- including eye-tracking data, learning outcomes, assessment content, and teaching standards -- to derive meaningful reading insights. We employ unsupervised learning techniques to identify distinct reading behavior patterns, and then a large language model (LLM) synthesizes the derived information into actionable reports for educators, streamlining the interpretation process. LLM experts and human educators evaluate these reports for clarity, accuracy, relevance, and pedagogical usefulness. Our findings indicate that LLMs can effectively function as educational analysts, turning diverse data into teacher-friendly insights that are well-received by educators. While promising for automating insight generation, human oversight remains crucial to ensure reliability and fairness. This research advances human-centered AI in education, connecting data-driven analytics with practical classroom applications.