Date
Publisher
arXiv
The rapid integration of generative artificial intelligence (GenAI) technology into education necessitates precise measurement of GenAI literacy to ensure that learners and educators possess the skills to engage with and critically evaluate this transformative technology effectively. Existing instruments often rely on self-reports, which may be biased. In this study, we present the GenAI Literacy Assessment Test (GLAT), a 20-item multiple-choice instrument developed following established procedures in psychological and educational measurement. Structural validity and reliability were confirmed with responses from 355 higher education students using classical test theory and item response theory, resulting in a reliable 2-parameter logistic (2PL) model (Cronbach's alpha = 0.80; omega total = 0.81) with a robust factor structure (RMSEA = 0.03; CFI = 0.97). Critically, GLAT scores were found to be significant predictors of learners' performance in GenAI-supported tasks, outperforming self-reported measures such as perceived ChatGPT proficiency and demonstrating external validity. These results suggest that GLAT offers a reliable and valid method for assessing GenAI literacy, with the potential to inform educational practices and policy decisions that aim to enhance learners' and educators' GenAI literacy, ultimately equipping them to navigate an AI-enhanced future.
Study at a glance
- Question: Can a multiple-choice test reliably and accurately measure students' generative AI literacy?
- Setting and data: Responses from 355 higher education students.
- Method: Test development and validation (classical test theory and item response theory).
Why it matters
Rapid integration of generative AI into education requires precise measurement of GenAI literacy to ensure learners and educators can critically evaluate the technology. However, existing assessment instruments often rely on self-reports, which may be biased.
Key findings
- Among 355 higher education students, scores on a 20-item generative AI literacy test significantly predicted their performance on AI-supported tasks.
- The objective assessment predicted task performance more effectively than self-reported measures like perceived ChatGPT proficiency.
- Results suggest the 20-item multiple-choice assessment provides a reliable and valid measure of generative AI literacy.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
