Date
Publisher
arXiv
The rapid integration of Artificial Intelligence (AI) into educational
technology promises to revolutionize content creation and assessment. However,
the quality and pedagogical alignment of AI-generated content remain critical
challenges. This paper investigates the impact of lightweight prompt
engineering strategies on the cognitive alignment of AI-generated questions
within OneClickQuiz, a Moodle plugin leveraging generative AI. We evaluate
three prompt variants-a detailed baseline, a simpler version, and a
persona-based approach-across Knowledge, Application, and Analysis levels of
Bloom's Taxonomy. Utilizing an automated classification model (from prior work)
and human review, our findings demonstrate that explicit, detailed prompts are
crucial for precise cognitive alignment. While simpler and persona-based
prompts yield clear and relevant questions, they frequently misalign with
intended Bloom's levels, generating outputs that are either too complex or
deviate from the desired cognitive objective. This study underscores the
importance of strategic prompt engineering in fostering pedagogically sound
AI-driven educational solutions and advises on optimizing AI for quality
content generation in learning analytics and smart learning environments.
Study at a glance
- Question: How do different prompt engineering approaches impact the cognitive alignment and quality of AI-generated quiz questions?
- Setting and data: 135 computer science quiz questions generated across 5 concepts and 3 Bloom's Taxonomy levels; subset of 45 questions evaluated by human reviewers.
- Method: Mixed methods combining automated DistilBERT classification of generated questions with qualitative human expert ratings on a 5-point scale.
Why it matters
Generative AI can automate quiz creation, but producing pedagogically sound questions aligned to intended cognitive depths remains a challenge. Without strategic prompt guidance, AI tools risk generating generic or misaligned educational content.
Key findings
- Detailed prompts with explicit level descriptions and action verbs achieved a 96% match rate to intended Bloom's levels, significantly outperforming simpler and persona-based prompts.
- Simpler prompts and persona-based prompts achieved match rates of only 60% and 40%, respectively, frequently overshooting intended levels or generating higher cognitive complexity than requested.
- All three prompt types consistently scored a perfect 5.0 out of 5 for question clarity and relevance, demonstrating that basic generation quality did not guarantee cognitive alignment.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
