Date
Publisher
arXiv
Student mistakes in mathematics are often systematic: a learner applies a coherent but wrong procedure and repeats it across contexts. We introduce MalruleLib, a learning-science-grounded framework that translates documented misconceptions into executable procedures, drawing on 67 learning-science and mathematics education sources, and generates step-by-step traces of malrule-consistent student work. We formalize a core student-modeling problem as Malrule Reasoning Accuracy (MRA): infer a misconception from one worked mistake and predict the student's next answer under cross-template rephrasing. Across nine language models (4B-120B), accuracy drops from 66% on direct problem solving to 40% on cross-template misconception prediction. MalruleLib encodes 101 malrules over 498 parameterized problem templates and produces paired dual-path traces for both correct reasoning and malrule-consistent student reasoning. Because malrules are executable and templates are parameterizable, MalruleLib can generate over one million instances, enabling scalable supervision and controlled evaluation. Using MalruleLib, we observe cross-template degradations of 10-21%, while providing student step traces improves prediction by 3-15%. We release MalruleLib as infrastructure for educational AI that models student procedures across contexts, enabling diagnosis and feedback that targets the underlying misconception.
Study at a glance
- Question: Can language models infer student math misconceptions from mistakes and predict their answers on new problems?
- Setting and data: Benchmark of 4,991 problem instances generated across 101 malrules and 498 templates spanning elementary through early high school mathematics.
- Method: Benchmark evaluation comparing nine language models on correct problem solving and misconception prediction across prompt and template conditions.
Why it matters
Tutoring requires inferring student misconceptions and predicting how they reappear across contexts. However, educational AI lacks computational infrastructure to model systematic student error procedures and test whether models can predict misconception-driven errors under cross-template shifts.
Key findings
- Language models correctly solved math problems 65.7% of the time, but their cross-template misconception prediction accuracy dropped to 40.5% without steps.
- Providing step-by-step reasoning traces improved cross-template misconception prediction by 3 to 15 percentage points across models, reaching 46.5% on average.
- Models predicted student errors better within the same problem template than across different templates, where performance degraded by 10 to 21 percentage points.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
