Date
Publisher
arXiv
Large Language Models (LLMs) are increasingly used for educational support, yet their response quality varies depending on the language of interaction. This paper presents an automated multilingual pipeline for generating, solving, and evaluating math problems aligned with the German K-10 curriculum. We generated 628 math exercises and translated them into English, German, and Arabic. Three commercial LLMs (GPT-4o-mini, Gemini 2.5 Flash, and Qwen-plus) were prompted to produce step-by-step solutions in each language. A held-out panel of LLM judges, including Claude 3.5 Haiku, evaluated solution quality using a comparative framework. Results show a consistent gap, with English solutions consistently rated highest, and Arabic often ranked lower. These findings highlight persistent linguistic bias and the need for more equitable multilingual AI systems in education.
Study at a glance
- Question: How does the language of interaction affect the quality of math solutions generated by large language models?
- Setting and data: 628 K-10 math exercises aligned with the German curriculum, translated into English, German, and Arabic.
- Method: Automated multilingual evaluation using a held-out panel of language models to rank solution quality.
Why it matters
Commercial language models are trained on English-centric data, raising concerns that students who interact with them in other languages receive math explanations with varying clarity, accuracy, and pedagogical value.
Key findings
- Across all three models, English math solutions were most frequently ranked highest, German occupied the middle range, and Arabic was generally ranked lowest.
- GPT-4o-mini displayed the strongest linguistic disparity, rating English highest in 493 exercises and Arabic worst in 403 exercises.
- Judges favored solutions with explicit reasoning and clear pedagogy rather than simple correctness, praising English for structure and critiquing Arabic for lacking depth.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
