Classifying German Language Proficiency Levels Using Large Language Models

Elias-Leander Ahlers, University of Münster

Witold Brunsmann, University of Münster

Malte Schilling, University of Münster

Date

12/2025

AI Overview

Why it matters

Assessing language proficiency is essential for instruction tailored to learners' needs. However, traditional classification approaches rely on labor-intensive, hand-crafted features that have limited scalability across the full proficiency scale.

Study at a glance

Question: Can large language models accurately classify German texts into Common European Framework of Reference proficiency levels?
Setting and data: 1,567 German texts (A1–C2 levels) compiled from existing learner corpora and synthetic text generation.
Method: Model evaluation comparing prompt engineering, probing classifiers, and supervised fine-tuning (LoRA).
AI tools: LLaMA-3-8B-Instruct, Claude 3.5 Sonnet, LLaMA-3-8B, Meta-LLaMA-3-70B-Instruct, Qwen/Qwen2-72B-Instruct, Mixtral-8x22B-Instruct-v0.1

Key findings

Fine-tuning LLaMA-3-8B-Instruct achieved an accuracy of 76.7%, a weighted F1 score of 0.769, and 100% group accuracy across adjacent levels.
Training a probing neural network on the model's internal representations yielded 65.83% accuracy and 99.2% group accuracy, outperforming purely prompt-based methods.
For prompt-based classification, German few-shot prompts raised classification accuracy to 59.3%, compared to 33.3% for German zero-shot and 23.3% for English base prompts.

Summary drafted with AI from the full paper and may contain errors. See the full paper for details.

Assessing language proficiency is essential for education, as it enables instruction tailored to learners needs. This paper investigates the use of Large Language Models (LLMs) for automatically classifying German texts according to the Common European Framework of Reference for Languages (CEFR) into different proficiency levels. To support robust training and evaluation, we construct a diverse dataset by combining multiple existing CEFR-annotated corpora with synthetic data. We then evaluate prompt-engineering strategies, fine-tuning of a LLaMA-3-8B-Instruct model and a probing-based approach that utilizes the internal neural state of the LLM for classification. Our results show a consistent performance improvement over prior methods, highlighting the potential of LLMs for reliable and scalable CEFR classification.