Unveiling The Learning Mind Of Language Models: A Cognitive Framework And Empirical Study

Zhengyu Hu, Hong Kong University of Science and Technology

Jianxun Lian, Microsoft (United States)

Zheyuan Xiao, The Hong Kong University of Science and Technology (Guangzhou)

Seraphina Zhang, University of Cambridge

Tianfu Wang, Hong Kong University of Science and Technology

Nicholas Jing Yuan, Microsoft (United States)

Xing Xie, Microsoft Research Asia (China)

Hui Xiong, The Hong Kong University of Science and Technology (Guangzhou)

Date

12/2025

AI Overview

Why it matters

Effective learning is essential for AI agents to adapt to dynamic environments and acquire new knowledge. However, systematic research investigating how well large language models acquire and generalize new knowledge across tasks remains scarce.

Study at a glance

Question: How effectively do large language models acquire and generalize knowledge from instructions, abstract concepts, and accumulated experiences?
Setting and data: Benchmark evaluations across mathematics, logic, planning, and multi-agent games, using datasets such as MagpieMath, GSM8K, and TextArena environments.
Method: Model evaluation across experimental instructional, conceptual, and experiential tasks, alongside benchmark testing on a multi-agent game platform (LearnArena).
AI tools: GPT-4o, GPT-4o-mini, Qwen2.5-7B, Qwen2.5-32B, Llama3.1-8B, Mistral-7B

Key findings

Learners trained with interactive clarification consistently outperform those receiving passive solutions across eight mathematical benchmarks, though gains diminish when instructors are too small.
Injecting conceptual rules aids larger models but degrades smaller models' performance, showing that conceptual understanding is scale-emergent.
Models perform well with a few in-context experience examples but degrade sharply when examples exceed 900, showing difficulties with many-shot learning.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Large language models (LLMs) have shown impressive capabilities across tasks such as mathematics, coding, and reasoning, yet their learning ability, which is crucial for adapting to dynamic environments and acquiring new knowledge, remains underexplored. In this work, we address this gap by introducing a framework inspired by cognitive psychology and education. Specifically, we decompose general learning ability into three distinct, complementary dimensions: Learning from Instructor (acquiring knowledge via explicit guidance), Learning from Concept (internalizing abstract structures and generalizing to new contexts), and Learning from Experience (adapting through accumulated exploration and feedback). We conduct a comprehensive empirical study across the three learning dimensions and identify several insightful findings, such as (i) interaction improves learning; (ii) conceptual understanding is scale-emergent and benefits larger models; and (iii) LLMs are effective few-shot learners but not many-shot learners. Based on our framework and empirical findings, we introduce a benchmark that provides a unified and realistic evaluation of LLMs' general learning abilities across three learning cognition dimensions. It enables diagnostic insights and supports evaluation and development of more adaptive and human-like models.