Search and Filter

Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform Llms

Date

AI Overview

Why it matters

Predicting student responses enables educational platforms to anticipate misconceptions and personalize support. It remains unclear whether general-purpose large language models can match specialized knowledge tracing models on this task at scalable speed and cost.

Study at a glance
  • Question: Can general-purpose language models predict student answers more accurately, cheaply, and quickly than specialized knowledge tracing models?
  • Setting and data: Online learning platform data with 14,400 students answering up to 50 mathematics questions across 4,252 questions.
  • Method: Benchmark evaluation comparing predictive performance, latency, and deployment costs across models
  • AI tools: GPT-4o-mini, Gemini-2.5-flash-lite, Qwen2.5-7B-Instruct, Llama-1B, Qwen 3 0.6B, LLM KT
Key findings
  • Specialized knowledge tracing models achieved higher predictive accuracy (71.8% to 72.8%) and F1 scores than general-purpose language models (33.5% to 71.0%).
  • Knowledge tracing models were orders of magnitude faster, generating predictions in under 0.25 seconds per student compared to 3.1 seconds to nearly 55 minutes for language models.
  • Specialized knowledge tracing models cost less than $2 annually to serve 100,000 students, making them 615 to 12,400 times cheaper to deploy than language models.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Predicting future student responses to questions is particularly valuable for educational learning platforms where it enables effective interventions. One of the key approaches to do this has been through the use of knowledge tracing (KT) models. These are small, domain-specific, temporal models trained on student question-response data. KT models are optimised for high accuracy on specific educational domains and have fast inference and scalable deployments. The rise of Large Language Models (LLMs) motivates us to ask the following questions: (1) How well can LLMs perform at predicting students' future responses to questions? (2) Are LLMs scalable for this domain? (3) How do LLMs compare to KT models on this domain-specific task? In this paper, we compare multiple LLMs and KT models across predictive performance, deployment cost, and inference speed to answer the above questions. We show that KT models outperform LLMs with respect to accuracy and F1 scores on this domain-specific task. Further, we demonstrate that LLMs are orders of magnitude slower than KT models and cost orders of magnitude more to deploy. This highlights the importance of domain-specific models for education prediction tasks and the fact that current closed source LLMs should not be used as a universal solution for all tasks.