Date
Publisher
arXiv
To ensure that math curriculum is grade-appropriate and aligns with critical skills or concepts in accordance with educational standards, pedagogical experts can spend months carefully reviewing published math problems. Drawing inspiration from this process, our work presents a novel angle for evaluating language models' (LMs) mathematical abilities, by investigating whether they can discern skills and concepts enabled by math content. We contribute two datasets: one consisting of 385 fine-grained descriptions of K-12 math skills and concepts, or standards, from Achieve the Core (ATC), and another of 9.9K math problems labeled with these standards (MathFish). We develop two tasks for evaluating LMs' abilities to assess math problems: (1) verifying whether a problem aligns with a given standard, and (2) tagging a problem with all aligned standards. Working with experienced teachers, we find that LMs struggle to tag and verify standards linked to problems, and instead predict labels that are close to ground truth, but differ in subtle ways. We also show that LMs often generate problems that do not fully align with standards described in prompts, suggesting the need for careful scrutiny on use cases involving LMs for generating curricular materials. Finally, we categorize problems in GSM8k using math standards, allowing us to better understand why some problems are more difficult to solve for models than others.
Study at a glance
- Question: Can artificial intelligence models accurately match math problems to the specific educational standards and skills they teach?
- Setting and data: 385 K-12 math standards, 9,900 standard-labeled math problems, problems from GSM8k, and experienced teachers.
- Method: Tests how well language models verify and tag math problems with educational standards, working with experienced teachers.
Why it matters
Pedagogical experts spend months reviewing math problems to ensure curriculum is grade-appropriate and aligns with educational standards. Investigating whether language models can discern skills and concepts in math content introduces a new angle for evaluating their mathematical abilities.
Key findings
- In K-12 math, language models struggle to tag and verify curriculum standards, predicting labels that differ in subtle ways from the truth.
- Language models often generate math problems that do not fully align with the educational standards described in user prompts.
- Curriculum standards help explain why certain math problems are more difficult for language models to solve than others.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Why use AI?
Study design
