Date
Publisher
arXiv
Recent advances in Generative Artificial Intelligence (GenAI) have substantially improved the ability of large language models (LLMs) to generate and explain source code. However, their performance on authentic object-oriented programming (OOP) assessments remains insufficiently understood. This study evaluates five widely used GenAI systems, ChatGPT-5.2, DeepSeek-V3, Gemini 2.5 Flash, Claude Sonnet 4.5, and M365 Copilot, using programming tests and examination tasks from an introductory university OOP course. The generated solutions were assessed using the same grading criteria applied to students and compared with historical student results from the same course, as well as findings from the previous year. Common errors were also analyzed to identify recurring limitations across models. All evaluated GenAI systems achieved higher scores than the average student cohort and frequently obtained full marks on longer programming tasks. Nevertheless, they occasionally produced non-compiling code and continued to struggle with advanced OOP concepts, particularly interfaces, abstract classes, and certain inheritance-related tasks. Performance was also limited on graphics-related questions involving image interpretation. Compared with the previous year, the evaluated systems demonstrated noticeable improvements across most assessments while exhibiting several recurring error patterns. The findings provide an updated evaluation of the capabilities and limitations of contemporary GenAI systems on authentic introductory OOP assessments. They also offer evidence that can inform the design of programming assessments, the responsible integration of GenAI tools into software engineering education, and future studies evaluating the evolution of AI-assisted programming.
Study at a glance
- Question: How do contemporary generative AI systems perform and make errors on authentic introductory object-oriented programming assessments?
- Setting and data: University of Tartu, Estonia; 285 historical students and 5 AI systems tested on introductory Java OOP course assessments in Estonian.
- Method: Benchmark evaluation comparing AI outputs graded with course rubrics against historical student performance and prior-year evaluations.
Why it matters
Little research evaluates AI performance on authentic object-oriented programming assessments, especially for non-English prompts. Rapid model development also requires longitudinal evaluations using consistent assessments to identify evolving capabilities and remaining conceptual challenges.
Key findings
- All five AI systems achieved higher average scores than the student cohort on most assessments, with ChatGPT-5.2 scoring 31.01 and Claude reaching 30.95 out of 33 on the exam.
- Compared to evaluations from one year prior, evaluated systems improved considerably across assessments, successfully resolving earlier recurring mistakes in queue handling and class hierarchies.
- Models continued to struggle with conceptual questions involving interfaces, abstract classes, and visual graphics recognition, and they occasionally produced non-compiling code.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
Who age?
Why use AI?
Study design
