Date
AI Overview
Why it matters
Most classroom AI activities focus on prompting pre-trained models, leaving model training opaque. Tools that reveal how data curation and training shape model behavior are scarce.
Study at a glance
- Question: How does hands-on experience training an in-browser language model shift students' explanations of why models produce strange outputs?
- Setting and data: First-year Introduction to Computing course at a Finnish university, 162 students with complete pre- and post-test data.
- Method: Pre- and post-intervention open-ended testing with qualitative content analysis and a two-proportion z-test.
- AI tools: Little Language Machine, nanoGPT
Key findings
- Attributions of incorrect model outputs to training data significantly increased from 13 percent in the pre-test to 38 percent in the post-test.
- Student misconceptions, such as believing language models search the web or mistranslate from English to Finnish, dropped from 37 to 10 mentions.
- Students shifted away from viewing models as autonomous or intelligent agents toward using discipline-specific explanations centered on training data size, quality, and duration.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Most classroom engagements with generative AI focus on prompting pre-trained models, leaving the role of training data and model mechanics opaque. We developed a browser-based tool that allows students to train a small transformer language model entirely on their own device, making the training process visible. In a CS1 course, 162 students completed pre- and post-test explanations of why language models sometimes produce incorrect or strange output. After a brief hands-on training activity, students' explanations shifted significantly from anthropomorphic and misconceived accounts toward data- and model-based reasoning. The results suggest that enabling learners to directly observe training can support conceptual understanding of the data-driven nature of language models and model training, even within a short intervention. For K-12 AI literacy and AI education research, the study findings suggest that enabling students to train - and not only prompt - language models can shift how they think about AI.
Research Categories
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
AI tools
Publication Source
Publisher
arXiv
