Date
AI Overview
Why it matters
Critical thinking is an essential human skill in an AI-saturated world, yet educational data mining lacks scalable, reliable ways to rapidly assess its complex underlying subskills in authentic student work.
Study at a glance
- Question: Can large language models accurately assess critical thinking subskills in student argumentative essays?
- Setting and data: 500 human-coded argumentative essays from grades 6–12 in the United States, drawn from the PERSUADE 2.0 corpus.
- Method: Model evaluation comparing zero-shot prompting, few-shot prompting, and supervised fine-tuning against human rubric ratings.
- AI tools: GPT-5, Llama-3.1 8B Instruct, ModernBERT-base, GPT-4.1
Key findings
- Fine-tuning Llama 3.1 8B performed best, achieving an accuracy of .604 and a reliability score of .606 that matched human coder agreement thresholds.
- Models performed best on subskills with balanced and distinct proficiency levels, but struggled when label distributions were heavily skewed or required subtle distinctions.
- Adapting models to completely unseen subskills degraded performance, though including detailed scoring rubrics and step-by-step justifications helped cross-subskill generalization.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
As the world becomes increasingly saturated with AI-generated content, disinformation, and algorithmic persuasion, critical thinking - the capacity to evaluate evidence, detect unreliable claims, and exercise independent judgment - is becoming a defining human skill. Developing critical thinking skills through timely assessment and feedback is crucial; however, there has not been extensive work in educational data mining on defining, measuring, and supporting critical thinking. In this paper, we investigate the feasibility of measuring "subskills" that underlie critical thinking. We ground our work in an authentic task where students operationalize critical thinking by writing argumentative essays. We developed a coding rubric based on an established skills progression and completed human coding for a corpus of student essays. We then evaluated three distinct approaches to automated scoring: zero-shot prompting, few-shot prompting, and supervised fine-tuning, implemented across three large language models (GPT-5, Llama 3.1 8B, and ModernBERT). Fine-tuning Llama 3.1 8B achieved the best results and demonstrated particular strength on subskills with highly separable proficiency levels with balanced labels across levels, while lower performance was observed for subskills that required detection of subtle distinctions between proficiency levels or imbalanced labels. Our exploratory work represents an initial step toward scalable assessment of critical thinking skills across authentic educational contexts. Future research should continue to combine automated critical thinking assessment with human validation to more accurately detect and measure dynamic, higher-order thinking skills.
Research Categories
What is the application?
Who age?
Why use AI?
Study design
AI tools
Datasets
Publication Source
Publisher
arXiv
