Date
Publisher
arXiv
This paper presents the Personalized Thinking Model (PTM), a hierarchical and interpretable learner representation designed for AI supported education. PTM organizes evidence from learner journals into a five-layer structure covering behavioral instances, behavioral patterns, cognitive routines, metacognitive tendencies, and self-system values. PTM is grounded in Marzano's New Taxonomy of Educational Objectives and tries to clone learner's thinking model and build cognitive twin. It was constructed using a pipeline that combines large language model inference (Gemini 2.5 Pro), sentence embeddings, dimensionality reduction, and consensus clustering. This paper evaluates PTM fidelity through three methods applied to 40 participants in a seven-week study. First, automatic evaluation using atomic information point matching yielded an overall F1 score of 74.57% before human-in-the-loop (HITL) refinement and 75.48% after refinement. Second, user evaluation using a Likert scale produced mean ratings of 4.26 and 4.30 on a five-point scale for pre and post-HITL conditions respectively. Third, semantic alignment verification showed that topic coherence increased from 0.436 at the behavioral layer to 0.626 at the core value layer, while lexical overlap with journal vocabulary decreased from 0.114 to 0.007 across those same layers. These results suggest that the PTM produces outputs with acceptable fidelity, was generally perceived by users as reflecting their thinking, and showed a pattern consistent with semantic abstraction across layers.
Study at a glance
- Question: Can AI accurately construct an interpretable, multi-layer model of a student's thinking from reflective journals?
- Setting and data: Universitas Negeri Yogyakarta, Indonesia; 40 undergraduate Information Technology students aged 18 to 24; reflective journals collected over a seven-week field study.
- Method: System development and evaluation combining automated atomic point matching, user Likert surveys, semantic analysis, and follow-up qualitative interviews.
Why it matters
Personalization systems often rely on surface-level activity logs that cannot explain why learners respond differently, creating a black-box problem. Inspectable cognitive structures are needed to represent reasoning processes and support tailored learning companions.
Key findings
- The model achieved an overall accuracy F1 score of 74.57% before human refinement and 75.48% after human refinement, exceeding the predefined 70% threshold.
- Participating students rated the model an average of 4.26 out of 5 before refinement and 4.30 after refinement, indicating strong agreement that it reflected their thinking.
- Higher layers showed increased topic coherence alongside lower vocabulary overlap with journals, suggesting successful semantic abstraction from concrete behaviors to core values.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
