Evaluating Gemini in an Arena for Learning

Authors

LearnLM Team,

Google

Date

05/2025

Publisher

arXiv

Link

http://arxiv.org/pdf/2505.24477v1

Artificial intelligence (AI) is poised to transform education, but the research community lacks a robust, general benchmark to evaluate AI models for learning. To assess state-of-the-art support for educational use cases, we ran an "arena for learning" where educators and pedagogy experts conduct blind, head-to-head, multi-turn comparisons of leading AI models. In particular, $N = 189$ educators drew from their experience to role-play realistic learning use cases, interacting with two models sequentially, after which $N = 206$ experts judged which model better supported the user's learning goals. The arena evaluated a slate of state-of-the-art models: Gemini 2.5 Pro, Claude 3.7 Sonnet, GPT-4o, and OpenAI o3. Excluding ties, experts preferred Gemini 2.5 Pro in 73.2% of these match-ups -- ranking it first overall in the arena. Gemini 2.5 Pro also demonstrated markedly higher performance across key principles of good pedagogy. Altogether, these results position Gemini 2.5 Pro as a leading model for learning.

Study at a glance

Question: Which leading AI model best supports learning goals in realistic educational scenarios?
Setting and data: 189 educators role-playing learning interactions and 206 pedagogy experts evaluating four leading AI models.
Method: Blind, head-to-head comparisons where educators role-play learning interactions with AI models and experts judge which model better supports learning.

Why it matters

Artificial intelligence is poised to transform education, but the research community lacks a robust, general benchmark to evaluate AI models for learning.

Key findings

In evaluations with 189 educators and 206 pedagogy experts, experts preferred Gemini 2.5 Pro in 73.2% of match-ups over other leading models.
Gemini 2.5 Pro demonstrated markedly higher performance across key principles of good pedagogy compared to other evaluated models.
Gemini 2.5 Pro ranked first overall among evaluated models in supporting realistic learning goals.

Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.

What is the application?

Teaching – Assessment and Feedback,

Learning – Student Support

Who is the user?

Who age?