Date
Publisher
arXiv
As Competency-Based Education (CBE) is gaining traction around the world, the shift from marks-based assessment to qualitative competency mapping is a manual challenge for educators. This paper tackles the bottleneck issue by suggesting a "Human-in-the-Loop" benchmarking framework to assess the effectiveness of multiple LLMs in automating secondary-level mathematics assessment. Based on the Grade 10 Optional Mathematics curriculum in Nepal, we created a multi-dimensional rubric for four topics and four cross-cutting competencies: Comprehension, Knowledge, Operational Fluency, and Behavior and Correlation.
The multi-provider ensemble, consisted of open-weight models -- Eagle (Llama 3.1-8B) and Orion (Llama 3.3-70B) -- and proprietary frontier models Nova (Gemini 2.5 Flash) and Lyra (Gemini 3 Pro), was benchmarked against a ground truth defined by two senior mathematics faculty members (kappa_w = 0.8652). The findings show a marked "Architecture-compatibility gap". Although the Gemini-based Mixture-of-Experts (Sparse MoE) models achieved "Fair Agreement" (kappa_w ~ 0.38), the larger Orion (70B) model exhibited "No Agreement" (kappa_w = -0.0261), suggesting that architectural compliance with instruction constraints outweighs the scale of raw parameters in rubric-constrained tasks. We conclude that while LLMs are not yet suitable for autonomous certification, they provide high-value assistive support for preliminary evidence extraction within a "Human-in-the-Loop" framework.
Study at a glance
- Question: How accurately can different large language models assess secondary school students' mathematics competencies compared to expert teacher evaluations?
- Setting and data: Occidental Public School, Nepal; 33 Grade 10 students; 16 handwritten mathematics assessment responses across four topics.
- Method: Model or benchmark evaluation comparing model predictions against expert teacher double-blind evaluations using weighted Cohen's kappa.
Why it matters
Shifted qualitative competency mapping in mathematics creates a manual challenge for educators, yet comparisons between language models and expert teachers remain rare at the rubric level, especially in secondary schools across the Global South.
Key findings
- Gemini 2.5 Flash achieved the highest agreement with human teachers (quadratic weighted kappa of about 0.38), reaching fair agreement.
- Llama 3.3-70B exhibited no agreement with human teachers (quadratic weighted kappa of -0.0261), indicating larger model size did not improve rubric compliance.
- Agreement between different models was highest between Gemini 2.5 Flash and Gemini 3 Pro (kappa of 0.5597), but only slight between Gemini and Llama models.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
