Date
Publisher
arXiv
To enhance LLMs' impact on math education, we need data on their mathematical prowess and biases across prompts. To fill this gap, we introduce MEDS (Math Education Digital Shadows) as a dataset mapping how large language models reason about and report mathematics across human- and AI-like conditions. MEDS involves 28,000 personas from 14 LLMs (from families like Mistral, Qwen, DeepSeek, Granite, Phi and Grok) shadowing either humans or AI assistants. Each record/shadow includes a set of prompts along with psychological/sociodemographic persona metadata and four types of math tasks: (i) open math interview, (ii) three psychometric tests about math perceptions with explanations, (iii) cognitive networks capturing math attitudes, and (iv) 18 high-school math test questions together with their reasoning and confidence scores. MEDS differs from traditional score-only math benchmarks because it integrates concepts of self-efficacy, math anxiety, and cognitive network science besides math proficiency scores. Data validation shows that the sampled LLMs exhibit schema integrity and consistent personas, together with family-specific peculiarities like human-like negative math attitudes, logical fallacies, and math overconfidence. MEDS will benefit learning analytics experts, cognitive scientists, and developers of safer AI tutors in mathematics.
Study at a glance
- Question: Can large language models convey mathematical knowledge effectively without expressing anxiety or disruptive biases across different prompting conditions?
- Setting and data: 28,000 automated runs across 14 large language models, evaluated under simulated human-shadow and AI-assistant prompting conditions.
- Method: Model evaluation using prompt-conditioned simulations across qualitative interviews, psychometric surveys, semantic networks, and multiple-choice problem solving.
Why it matters
Schools are slowly adopting large language models as mathematics tutors, partly due to a lack of clear evidence about the cognitive repercussions, including whether models express mathematics anxiety, overconfidence, or disruptive biases.
Key findings
- Several models displayed myopic overconfidence, with confidence scores exceeding 0.90 despite actual problem-solving accuracies remaining near 0.55.
- Prompted as AI assistants, models consistently reported an artificially inflated competence profile marked by high self-efficacy and low mathematics anxiety compared to human personas.
- Models mirrored human negative attitudes and cognitive dissonance toward mathematics, associating science positively while linking mathematics to stressful concepts like difficulty and frustration.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
