Date
Publisher
arXiv
This research full paper presents an enhancement pipeline for large language models (LLMs) in assessing homework for an undergraduate circuit analysis course, aiming to improve LLMs' capacity to provide personalized support to electrical engineering students. Existing evaluations have demonstrated that GPT-4o possesses promising capabilities in assessing student homework in this domain. Building on these findings, we enhance GPT-4o's performance through multi-step prompting, contextual data augmentation, and the incorporation of targeted hints. These strategies effectively address common errors observed in GPT-4o's responses when using simple prompts, leading to a substantial improvement in assessment accuracy. Specifically, the correct response rate for GPT-4o increases from 74.71% to 97.70% after applying the enhanced prompting and augmented data on entry-level circuit analysis topics. This work lays a foundation for the effective integration of LLMs into circuit analysis instruction and, more broadly, into engineering education.
Study at a glance
- Question: Can multi-step prompting, context data augmentation, and targeted hints improve GPT-4o's accuracy in grading circuit analysis homework?
- Setting and data: Undergraduate circuit analysis course at a research-intensive university in the Southeastern US, 283 handwritten student solutions from 37 students across 119 textbook problems.
- Method: Model evaluation comparing baseline and enhanced prompting strategies across student homework submissions
Why it matters
Students need personalized learning support, which places heavy workloads on instructors. However, large language models can produce recurring errors and hallucinations that may mislead engineering students during homework grading.
Key findings
- Applying multi-step prompting and augmented data increased GPT-4o's correct response rate from 74.71% to 97.70% on 87 entry-level circuit analysis student solutions.
- The enhancement pipeline correctly resolved 11 of 13 specific homework assessment errors previously made by GPT-4o.
- In initial benchmarking across six circuit topics, baseline GPT-4o achieved 88.55% average accuracy, outperforming Llama 3 70B (85.08%) and GPT-3.5 Turbo (51.66%).
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Why use AI?
Study design
