Date
Publisher
arXiv
This study evaluates the integration of Bloom's Taxonomy into OneClickQuiz, an Artificial Intelligence (AI) driven plugin for automating Multiple-Choice Question (MCQ) generation in Moodle. Bloom's Taxonomy provides a structured framework for categorizing educational objectives into hierarchical cognitive levels. Our research investigates whether incorporating this taxonomy can improve the alignment of AI-generated questions with specific cognitive objectives. We developed a dataset of 3691 questions categorized according to Bloom's levels and employed various classification models-Multinomial Logistic Regression, Naive Bayes, Linear Support Vector Classification (SVC), and a Transformer-based model (DistilBERT)-to evaluate their effectiveness in categorizing questions. Our results indicate that higher Bloom's levels generally correlate with increased question length, Flesch-Kincaid Grade Level (FKGL), and Lexical Density (LD), reflecting the increased complexity of higher cognitive demands. Multinomial Logistic Regression showed varying accuracy across Bloom's levels, performing best for "Knowledge" and less accurately for higher-order levels. Merging higher-level categories improved accuracy for complex cognitive tasks. Naive Bayes and Linear SVC also demonstrated effective classification for lower levels but struggled with higher-order tasks. DistilBERT achieved the highest performance, significantly improving classification of both lower and higher-order cognitive levels, achieving an overall validation accuracy of 91%. This study highlights the potential of integrating Bloom's Taxonomy into AI-driven assessment tools and underscores the advantages of advanced models like DistilBERT for enhancing educational content generation.
Study at a glance
- Question: How well do artificial intelligence models generate and classify multiple-choice questions across different cognitive levels in Bloom's Taxonomy?
- Setting and data: 3,691 computer science questions generated using Google Vertex AI, evaluated across six Bloom's Taxonomy cognitive levels.
- Method: Model or benchmark evaluation comparing traditional machine learning algorithms and a fine-tuned transformer model (DistilBERT).
Why it matters
While artificial intelligence tools can generate varied quiz questions, their adherence to hierarchical cognitive frameworks like Bloom's Taxonomy remains under-explored, raising questions about whether automated assessments are pedagogically sound.
Key findings
- DistilBERT achieved the highest performance across all cognitive levels, reaching an overall validation accuracy of 91%.
- Traditional classifiers like Naive Bayes and linear support vector classification accurately identified lower-order recall questions but struggled to classify higher-order cognitive tasks.
- Higher cognitive levels were associated with longer question length, higher reading grade levels, and greater lexical density, with question length strongly correlating with grade level.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who is the user?
Who age?
Why use AI?
Study design
