Date
Publisher
arXiv
As online education platforms continue to expand, there is a growing need for
assessment methods that not only measure answer accuracy but also capture the
depth of students' cognitive processes in alignment with curriculum objectives.
This study proposes and evaluates a rubric-based assessment framework powered
by a large language model (LLM) for measuring algebraic competence,
real-world-context block coding tasks. The problem set, designed by mathematics
education experts, aligns each problem segment with five predefined rubric
dimensions, enabling the LLM to assess both correctness and quality of
students' problem-solving processes. The system was implemented on an online
platform that records all intermediate responses and employs the LLM for
rubric-aligned achievement evaluation. To examine the practical effectiveness
of the proposed framework, we conducted a field study involving 42 middle
school students engaged in multi-stage quadratic equation tasks with block
coding. The study integrated learner self-assessments and expert ratings to
benchmark the system's outputs. The LLM-based rubric evaluation showed strong
agreement with expert judgments and consistently produced rubric-aligned,
process-oriented feedback. These results demonstrate both the validity and
scalability of incorporating LLM-driven rubric assessment into online
mathematics and STEM education platforms.
Study at a glance
- Question: Can large language models reliably score students' algebraic competence across multi-stage block coding tasks using expert rubrics?
- Setting and data: Two middle schools in South Korea, November–December 2024, 42 middle school students.
- Method: System prototype evaluation and field study comparing model-assigned rubric scores with expert judgments.
Why it matters
Online learning platforms often evaluate only answer correctness, failing to capture students' multi-step mathematical reasoning. Standardized benchmarks and scalable tools for rubric-based assessment across multi-stage math and coding tasks remain rare.
Key findings
- Language-model rubric scores strongly agreed with expert evaluations across five dimensions (r = 0.79, rho = 0.76), showing an average upward bias of 5.1 points.
- Middle school students scored highest on conceptual understanding and equation solving (means near 80) but struggled most with proactively applying computational tools (mean near 50).
- Students successfully completing key applied programming tasks achieved significantly higher block-coding rubric scores (85.0) than peers who did not solve both items (55.3).
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
