Date
Publisher
arXiv
As Artificial Intelligence (AI) becomes more pervasive in various aspects of life, AI literacy is becoming a fundamental competency that enables individuals to move safely and competently in an AI-pervaded world. There is a growing need to measure this competency, e.g., to develop targeted educational interventions. Although several measurement tools already exist, many have limitations regarding subjective data collection methods, target group differentiation, validity, and integration of current developments such as Generative AI Literacy. This study develops and validates the AI Competency Objective Scale (AICOS) for measuring AI literacy objectively. The presented scale addresses weaknesses and offers a robust measurement approach that considers established competency and measurement models, captures central sub-competencies of AI literacy, and integrates the dimension of Generative AI Literacy. The AICOS provides a sound and comprehensive measure of AI literacy, and initial analyses show potential for a modular structure. Furthermore, a first edition of a short version of the AICOS is developed. Due to its methodological foundation, extensive validation, and integration of recent developments, the test represents a valuable resource for scientific research and practice in educational institutions and professional contexts. The AICOS significantly contributes to the development of standardized measurement instruments and enables the targeted assessment and development of AI skills in different target groups.
Study at a glance
- Question: Can an objective, knowledge-based test reliably measure multiple dimensions of AI literacy across diverse adult populations?
- Setting and data: Online survey of 514 German native-speaking adults recruited via Prolific, residing in Germany.
- Method: Multi-stage test development, item response theory (3PL model), confirmatory factor analysis, and correlational validity analyses.
Why it matters
Most existing AI literacy tests rely on subjective self-assessments that reflect perceived ability rather than actual knowledge. Objective, validated tools are needed to systematically assess competencies, including generative AI skills, and inform targeted educational programs.
Key findings
- The 51-item objective AI test demonstrated strong psychometric quality, high accuracy across ability levels, and internal consistency (alpha = .83, Composite Reliability = .90).
- Test scores correlated with an existing objective AI knowledge test (r = .58) and practical quiz scores (r = .42), but not with subjective self-assessments (r = .04).
- Participants with academic degrees, technical professional backgrounds, and males scored significantly higher on the AI competency test, whereas performance did not differ significantly across age groups.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
