Date
Publisher
arXiv
This paper presents the design and evaluation of a community-based artificial intelligence (AI) workflow developed for the Kaiapuni Assessment of Educational Outcomes (KĀ'EO) program, the only native language assessment used for federal accountability in the United States. The project explored whether document-grounded language models could ethically and effectively augment human analysis of item performance while preserving the cultural and linguistic integrity of the Hawaiian language. Operating under the KĀ'EO AI Policy Framework, the workflow used NotebookLM for cross-document synthesis of psychometric data and Claude 3.5 Sonnet for developer-facing interpretation, with human oversight at every stage. Fifty-eight flagged items across Hawaiian Language Arts, Mathematics, and Science were reviewed during Round 2 of the AI Lab, producing six interpretive briefs that identified systemic design issues such as linguistic ambiguity, Depth-of-Knowledge (DOK) misalignment, and structural overload. The findings demonstrate that AI can serve as an ethically bounded amplifier of human expertise, accelerating analysis while simultaneously prioritizing fairness, human expertise, and cultural authority. This work offers a replicable model for responsible AI integration in Indigenous-language educational measurement.
Study at a glance
- Question: Can document-grounded AI tools effectively and ethically assist educators in analyzing Hawaiian-language assessment items without sacrificing linguistic integrity?
- Setting and data: Hawaiian-medium KĀʻEO assessment program, October 2025, 58 flagged items across Hawaiian Language Arts, Mathematics, and Science with high-performing comparators.
- Method: Design-based research (iterative refinement of an AI-augmented review workflow)
Why it matters
Psychometric analysts rarely speak Hawaiian, and translating secure native-language tests burdens small teams. Assessment programs need ways to bridge technical analysis and content development while protecting Indigenous linguistic sovereignty and community ownership.
Key findings
- Across Hawaiian Language Arts, Mathematics, and Science, document-grounded AI produced six interpretive briefs that all passed human review without fundamental revisions.
- The AI workflow effectively diagnosed technical and structural patterns, such as mathematical multi-step cognitive overload and fine-grained grammatical traps that confused high-performing students.
- The AI missed cultural and figurative nuances of Hawaiian, incorrectly interpreting valued language constructions as grammatical errors.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
