Date
Publisher
arXiv
The rapid integration of large language models (LLMs) into undergraduate education presents an urgent challenge for engineering instructors. Despite widespread student adoption, there remains a critical lack of domain-specific empirical evidence to guide pedagogical policies and classroom interventions. This manuscript presents a descriptive study design and preliminary findings from an undergraduate engineering mechanics course conducted in Spring 2026. We detail a reproducible survey instrument used to capture student AI usage patterns, attitudes, and verification practices, which are subsequently linked to academic performance metrics. Additionally, we document a deployable sequence of nine structured, instructor-led AI demonstrations designed to model strategic LLM delegation and evaluation. While our preliminary data highlight shifting student behaviors and complex relationships between AI reliance and course outcomes, the primary contribution of this work is the provision of an open-access methodological framework. By making our complete study design, survey tools, and demonstration materials publicly available, we urge other engineering educators to collect and share similar empirical data. Navigating this unprecedented technological shift will require a collaborative, evidence-based approach to fully understand its long-term impacts on student learning.
Study at a glance
- Question: How do undergraduate engineering students use large language models, and how does use relate to course performance?
- Setting and data: Undergraduate engineering mechanics course at Boston University, Spring 2026, 105 consenting students across three lecture sections.
- Method: Descriptive analysis combining pre- and post-semester surveys with midterm grades, alongside qualitative responses coded using three language models.
Why it matters
Despite widespread student adoption of large language models, instructors lack domain-specific empirical evidence to guide classroom policies and pedagogical interventions in engineering courses.
Key findings
- Undergraduate use of artificial intelligence for engineering mechanics rose from 37% weekly use at baseline to 51% by the end of the semester.
- Baseline mechanics-specific tool use was inversely related to performance, with 10 of 24 top-quartile students reporting never using the tools compared to 3 of 22 bottom-quartile students.
- While students primarily used tools for concept clarification, independent verification remained common, although descriptions of minimal or conditional checking more than doubled from 5 to 12.
Summary drafted with AI from the full paper and may contain errors. See the full paper for details.
What is the application?
Who age?
Study design
