Date
AI Overview
Why it matters
Modelling scenarios with mathematical formalisms is challenging for STEM students. Identifying common mistakes from large data sets helps educators understand student struggles and provide targeted advice and automated feedback in learning systems.
Study at a glance
- Question: How can common student mistakes in mathematical modelling be identified from large educational data sets?
- Setting and data: Introductory logic courses (summer 2025 and 2026) and prior datasets, covering propositional, modal, and computation tree logic (up to 12,482 formula pairs).
- Method: Tool development combining iterative language model generation of bug-fixing transformations, algorithmic validation, and graph-based clustering with expert inspection.
- AI tools: GPT-OSS-120B
Key findings
- On a propositional logic data set of 6,106 student pairs, the workflow discovered transformation clusters explaining 84.44% of student mistakes, including 15.03% not captured by previous hand-picked analyses.
- The method scaled beyond propositional logic, discovering clusters that explained 79.39% of 12,482 student errors in modal logic and 35.89% of 7,210 errors in computation tree logic.
- Expert educator review indicated that main cluster transformations in propositional and computation tree logic were generally suitable for feedback, though specific sub-transformations were sometimes required for targeted advice.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Modelling with mathematical formalisms like logical formulas, mathematical equations, or regular expressions is an important yet challenging task for students of computer science and other STEM disciplines. Identifying common mistakes occurring in this context is an important step towards helping struggling students by providing targeted high-quality feedback, e.g. in interactive learning systems.
We present a tool-supported workflow that allows to (1) identify candidates for common mistakes that explain many student mistakes in large educational data sets, (2) cluster candidates according to similarities, and (3) visualize resulting clusters for instructors and CS education researchers. The visualization is designed to help researchers to identify common modelling mistakes. The candidates for common mistakes are represented by bug fixing transformations that translate incorrect formalizations into correct formalizations; they are generated by an LLM and validated algorithmically.
We show that this approach works well by reproducing common mistakes in propositional logic modelling that were identified by hand in the literature; showing that, unlike other algorithmic approaches, the LLM-based approach is suitable for very large sets of data; and applying it to multiple other formalisms to showcase it generalizes beyond propositional logic.
Research Categories
What is the application?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
