Date
Publisher
arXiv
The proliferation of generative AI tools has rendered traditional modular assessments in computing and data-centric education increasingly ineffective, creating a disconnect between academic evaluation and authentic skill measurement. This paper presents a theoretically grounded framework for designing AI-resilient assessments, supported by formal analysis and empirical validation.
We make three primary contributions. First, we establish two formal propositions. (1) Assessments composed of interconnected problems, in which outputs serve as inputs to subsequent tasks, are inherently more AI-resilient than modular assessments due to their reliance on multi-step reasoning and sustained context. (2) Semi-structured problems with deterministic success criteria provide more reliable measures of student competency than fully open-ended projects, which allow AI systems to default to familiar solution templates. These results challenge widely cited recommendations in recent institutional and policy guidance that promote open-ended assessments as inherently more robust to AI assistance.
Second, we validate these propositions through empirical analysis of three university data science courses (N = 117). We observe a substantial AI inflation effect: students achieve near-perfect scores on AI-assisted modular homework, while performance drops by approximately 30 percentage points on proctored exams (Cohen d = 1.51). In contrast, interconnected projects remain strongly aligned with modular assessments (r = 0.954, p < 0.001) while maintaining AI resistance, whereas proctored exams show weaker alignment (r = 0.726, p < 0.001).
Third, we translate these findings into a practical assessment design procedure that enables educators to construct evaluations that promote deeper engagement, reflect industry practice, and resist trivial AI delegation.
Study at a glance
- Question: How can educators design assignments and tests that generative AI cannot easily solve?
- Setting and data: Three university data science courses with 117 students.
- Method: Compares student performance across different assessment types, including homework, projects, and proctored exams.
Why it matters
Generative AI tools have made traditional modular assessments increasingly ineffective in computing and data-centric education. This creates a disconnect between academic evaluation and authentic skill measurement.
Key findings
- In three university data science courses with 117 students, scores dropped by about 30 percentage points from AI-assisted modular homework to proctored exams.
- Interconnected tasks where outputs feed into subsequent problems resisted AI better than standalone questions by relying on multi-step reasoning and sustained context.
- Semi-structured problems with clear success criteria measured student competence more reliably than open-ended projects, where AI can default to familiar solution templates.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
