Date
Publisher
arXiv
The evolution of mathematics has been guided in part by interestingness. From researchers choosing which problems to tackle next, to students deciding which ones to engage with, people's choices are often guided by judgments about how interesting or challenging problems are likely to be. As AI systems, such as LLMs, increasingly participate in mathematics with people -- whether for advanced research or education -- it becomes important to understand how well their judgments align with human ones. Our work examines this alignment through two empirical studies of human and LLM assessment of mathematical interestingness and difficulty, spanning a range of mathematical experience. We study two groups: participants from a crowdsourcing platform and International Math Olympiad competitors. We show that while many LLMs appear to broadly agree with human notions of interestingness, they mostly do not capture the distribution observed in human judgments. Moreover, most LLMs only somewhat align with why humans find certain math problems interesting, showing weak correlation with human-selected interestingness rationales. Together, our findings highlight both the promises and limitations of current LLMs in capturing human interestingness judgments for mathematical AI thought partnerships.
Study at a glance
- Question: How well do AI models' judgments of math problem interestingness and difficulty match human judgments?
- Setting and data: Crowdsourcing platform participants and International Math Olympiad competitors.
- Method: Two empirical studies comparing human and AI assessments of mathematical interestingness and difficulty.
Why it matters
People's choices in mathematics are often guided by how interesting or challenging problems are. As AI systems increasingly participate in math education and research, understanding how well their judgments align with human ones becomes important.
Key findings
- Across crowdsourced participants and Math Olympiad competitors, many language models appear to broadly agree with human judgments of math problem interestingness.
- Language models mostly do not capture the distribution of interestingness judgments observed among humans.
- Most language models align only somewhat with why humans find math problems interesting, showing weak correlation with human-selected rationales.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
