Date
AI Overview
Why it matters
Project-based learning assessments often fail to capture higher-order thinking skills when transferring knowledge to new contexts. Although scenario-based design problems can evaluate this transfer, creating them manually demands substantial instructor time and effort.
Study at a glance
- Question: Can AI models generate high-quality design problems that effectively assess students' higher-order thinking skills in project-based computing courses?
- Setting and data: Survey of 31 computing instructors; 80 generated problems evaluated by experts; 78 undergraduate computing students across three courses.
- Method: Mixed methods combining an instructor survey, expert rubric evaluation of generated problems, and classroom deployment with student assessments and keystroke logging.
- AI tools: GPT-4o, Gemini 2.5 Pro, ChatGPT
Key findings
- Large language models generated high-quality design problems, with the reasoning model significantly outperforming the standard model on scenario realism and feasibility.
- Student performance on design problems showed negligible correlation with traditional project grades, suggesting these problems capture aspects of higher-order thinking not reflected in project deliverables.
- Students spent roughly two minutes planning and actively revised responses during computer-based exams, behavior consistent with engaging in higher-order evaluative thinking.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Project-based learning (PjBL) is common in computing education, but traditional assessments of PjBL often fail to capture higher-order thinking (HOT), especially in transfer contexts. This study introduces "design problems" (DPs): concise, scenario-based prompts that require applying project concepts in new situations, to address this gap. We examined instructor perceptions, the ability of large language models (LLMs) to generate DPs, and student experiences. Surveys of 31 instructors, evaluation of 80 LLM-generated DPs, and student performance data showed that while instructors value DPs, creation effort is a barrier. LLMs helped by producing high-quality prompts with strong expert agreement. Students rated DPs from different LLMs similarly, and their performance on DP tasks showed negligible correlation with traditional project grades, suggesting DPs may capture distinct aspects of HOT. Keystroke data also suggested deeper cognitive engagement of students through planning and revision behaviors. Overall, DPs appear to be a useful complement to traditional assessments, especially in situations where AI use or collaboration may undermine individual learning.
Research Categories
What is the application?
Who age?
Why use AI?
AI tools
Publication Source
Publisher
arXiv
