Date
AI Overview
Why it matters
Existing student assessment models rely on limited numerical mastery scores or uninterpretable black boxes. LLM-based approaches also lack rigorous interpretability, preventing educators from getting grounded, trustworthy summaries of student knowledge and specific misconceptions.
Study at a glance
- Question: Can language models accurately predict student performance using interpretable, natural-language summaries of their knowledge and misconceptions?
- Setting and data: Synthetic math dataset (2,000 simulated students), filtered Eedi dataset (623 students), and filtered Chinese/English XES3G5M dataset (996 students).
- Method: Model or benchmark evaluation comparing proposed two-stage language bottleneck models against 14 cognitive diagnosis and knowledge tracing baselines.
- AI tools: GPT-5, GPT-4o, GPT-4o-mini, Qwen 2.5, Gemma 3
Key findings
- Language Bottleneck Models matched or rivaled the accuracy of top traditional models while using substantially less training data and generating readable explanations of student misconceptions.
- A fine-tuned Gemma-12B decoder paired with a GPT-5 encoder achieved the highest AUC on the Eedi dataset among all tested models.
- Using strong models as encoders rather than decoders improved accuracy by 5 to 10 percentage points, showing that extracting knowledge states is harder than predicting future answers.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Accurately assessing student knowledge is central to education. Cognitive Diagnosis (CD) models estimate student proficiency at a fixed point in time, while Knowledge Tracing (KT) methods model evolving knowledge states to predict future performance. However, existing approaches either provide quantitative concept mastery estimates with limited expressivity (CD, probabilistic KT) or prioritize predictive accuracy at the cost of interpretability (deep learning KT). We propose Language Bottleneck Models (LBMs), where an encoder LLM produces textual knowledge state summaries, which a decoder LLM uses to predict future performance. This produces interpretable summaries that can express nuanced insights--such as misconceptions--that CD and KT models cannot capture. Extensive validation across synthetic and real-world datasets shows LBMs reveal qualitative insights beyond what CD and KT models can capture, while achieving competitive accuracy with improved sample efficiency. We demonstrate that the encoder and decoder can be fine-tuned with reinforcement learning and supervised fine-tuning respectively to improve both summary quality and predictive performance.
Research Categories
What is the application?
Who is the user?
Study design
AI tools
Publication Source
Publisher
arXiv
