Date
Publisher
arXiv
The widespread adoption of Artificial Intelligence (AI) in K-12 education highlights the need for psychometrically-tested measures of teachers' AI literacy. Existing work has primarily relied on either self-report (SR) or objective-based (OB) assessments, with few studies aligning the two within a shared framework to compare perceived versus demonstrated competencies or examine how prior AI literacy experience shapes this relationship. This gap limits the scalability of learning analytics and the development of learner profile-driven instructional design. In this study, we developed and evaluated SR and OB measures of teacher AI literacy within the established framework of Concept, Use, Evaluate, and Ethics. Confirmatory factor analyses support construct validity with good reliability and acceptable fit. Results reveal a low correlation between SR and OB factors. Latent profile analysis identified six distinct profiles, including overestimation (SR > OB), underestimation (SR < OB), alignment (SR close to OB), and a unique low-SR/low-OB profile among teachers without AI literacy experience. Theoretically, this work extends existing AI literacy frameworks by validating SR and OB measures on shared dimensions. Practically, the instruments function as diagnostic tools for professional development, supporting AI-informed decisions (e.g., growth monitoring, needs profiling) and enabling scalable learning analytics interventions tailored to teacher subgroups.
Study at a glance
- Question: How well do teachers' self-reported AI literacy ratings match their objectively measured AI knowledge and skills?
- Setting and data: Taiwan, February 2025, survey and assessment data from 288 pre-service and in-service K–12 teachers.
- Method: Survey assessment analyzed with Rasch modeling, factor analyses (EFA and CFA), and latent profile analysis.
Why it matters
Few studies examine whether teachers' perceived AI literacy aligns with their demonstrated knowledge. This gap limits the scalability of learning analytics and the development of tailored teacher professional development.
Key findings
- Teachers' self-reported AI literacy and objective performance showed consistently weak correlations ranging from 0.07 to 0.24 across dimensions.
- Latent profile analysis revealed six distinct groups of teachers displaying patterns of overestimation, underestimation, or alignment between perceived and tested AI competencies.
- Overestimating AI competence appeared only among teachers with prior AI literacy experience, whereas a profile with low self-ratings and low test scores was unique to inexperienced teachers.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
