Baseline Performance Of AI Tools In Classifying Cognitive Demand Of Mathematical Tasks

Danielle Fox, University of Pittsburgh

Brenda Robles, University of Pittsburgh

Elizabeth DiPietro Brovey, University of Pittsburgh

Christian D. Schunn, University of Pittsburgh

Date

03/2026

AI Overview

Why it matters

Teachers need time and expertise to adapt math tasks while maintaining rigor. If AI cannot reliably evaluate cognitive demand, its utility for modifying or creating instructional tasks is undermined.

Study at a glance

Question: Can AI tools accurately classify the cognitive demand of mathematics tasks using an established framework?
Setting and data: 12 expert-categorized mathematics tasks evaluated across 11 AI tools, yielding 133 total classifications.
Method: Benchmark evaluation comparing AI classifications and reasoning against human expert ratings using the Task Analysis Guide.
AI tools: ChatGPT Free GPT 5, Claude Free, DeepSeek DeepSeek-V3.2-Exp, Gemini Free 2.5 Flash, Grok Free Version 4, Khanmigo Free

Key findings

AI tools accurately classified the cognitive demand of mathematics tasks in only 62% of cases, with accuracy ranging from 9% to 100% across tasks.
Education-specific tools were not significantly more accurate than general-purpose tools (63% versus 61%), and tool accuracy ranged from 50% to 83%.
All tools struggled with extreme cognitive demand categories, exhibiting a systematic central-tendency bias toward middle procedural categories and over-weighting surface textual features.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Teachers face increasing demands on their time, particularly in adapting mathematics curricula to meet individual student needs while maintaining cognitive rigor. This study evaluates whether AI tools can accurately classify the cognitive demand of mathematical tasks, which is important for creating or adapting tasks that support student learning. We tested eleven AI tools: six general-purpose (ChatGPT, Claude, DeepSeek, Gemini, Grok, Perplexity) and five education-specific (Brisk, Coteach AI, Khanmigo, Magic School, School$.$AI), on their ability to categorize mathematics tasks across four levels of cognitive demand using a research-based framework. The goal was to approximate the performance teachers will achieve with straightforward prompts. On average, AI tools accurately classified cognitive demand in only 63% of cases. Education-specific tools were not more accurate than general-purpose tools, and no tool exceeded 83% accuracy. All tools struggled with tasks at the extremes of cognitive demand (Memorization and Doing Mathematics), exhibiting a systematic bias toward middle-category levels (Procedures with/without Connections). The tools often gave plausible-sounding explanations likely to be persuasive to novice teachers. Error analysis of AI tools' misclassification of the broad level of cognitive demand (high vs. low) revealed that tools consistently overweighted surface textual features over underlying cognitive processes. Further, AI tools showed weaknesses in reasoning about factors that make tasks higher vs. lower cognitive demand. Errors stemmed not from ignoring relevant dimensions, but from incorrectly reasoning about multiple task aspects. These findings carry implications for AI integration into teacher planning workflows and highlight the need for improved prompt engineering and tool development for educational applications.