Date
Publisher
arXiv
Accurately modeling student cognition is crucial for developing effective
AI-driven educational technologies. A key challenge is creating realistic
student models that satisfy two essential properties: (1) accurately
replicating specific misconceptions, and (2) correctly solving problems where
these misconceptions are not applicable. This dual requirement reflects the
complex nature of student understanding, where misconceptions coexist with
correct knowledge. This paper investigates whether Large Language Models (LLMs)
can be instruction-tuned to meet this dual requirement and effectively simulate
student thinking in algebra. We introduce MalAlgoPy, a novel Python library
that generates datasets reflecting authentic student solution patterns through
a graph-based representation of algebraic problem-solving. Utilizing MalAlgoPy,
we define and examine Cognitive Student Models (CSMs) - LLMs instruction tuned
to faithfully emulate realistic student behavior. Our findings reveal that LLMs
trained on misconception examples can efficiently learn to replicate errors.
However, the training diminishes the model's ability to solve problems
correctly, particularly for problem types where the misconceptions are not
applicable, thus failing to satisfy second property of CSMs. We demonstrate
that by carefully calibrating the ratio of correct to misconception examples in
the training data - sometimes as low as 0.25 - it is possible to develop CSMs
that satisfy both properties. Our insights enhance our understanding of
AI-based student models and pave the way for effective adaptive learning
systems.
Study at a glance
- Question: Can AI models accurately simulate student algebra mistakes while still correctly solving other math problems?
- Setting and data: Generated datasets representing student algebra problem-solving patterns.
- Method: Computational experiment testing AI models trained on varying ratios of correct solutions and misconceptions (instruction tuning).
Why it matters
Accurately modeling student cognition is crucial for developing effective AI-driven educational technologies. A key challenge is creating realistic models that accurately replicate specific misconceptions while still correctly solving problems where those misconceptions do not apply.
Key findings
- In algebra, AI models can realistically simulate student misconceptions while correctly solving other problems when trained with a correct-to-misconception ratio as low as 0.25.
- AI models trained on student misconceptions efficiently learned to replicate their errors.
- Learning to replicate misconceptions reduced the AI models' ability to correctly solve problems where those misconceptions did not apply.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
