Date
Publisher
arXiv
This study evaluates the performance of OpenAI's o1-preview model in higher-order cognitive domains, including critical thinking, systematic thinking, computational thinking, data literacy, creative thinking, logical reasoning, and scientific reasoning. Using established benchmarks, we compared the o1-preview models's performance to human participants from diverse educational levels. o1-preview achieved a mean score of 24.33 on the Ennis-Weir Critical Thinking Essay Test (EWCTET), surpassing undergraduate (13.8) and postgraduate (18.39) participants (z = 1.60 and 0.90, respectively). In systematic thinking, it scored 46.1, SD = 4.12 on the Lake Urmia Vignette, significantly outperforming the human mean (20.08, SD = 8.13, z = 3.20). For data literacy, o1-preview scored 8.60, SD = 0.70 on Merk et al.'s "Use Data" dimension, compared to the human post-test mean of 4.17, SD = 2.02 (z = 2.19). On creative thinking tasks, the model achieved originality scores of 2.98, SD = 0.73, higher than the human mean of 1.74 (z = 0.71). In logical reasoning (LogiQA), it outperformed humans with average 90%, SD = 10% accuracy versus 86%, SD = 6.5% (z = 0.62). For scientific reasoning, it achieved near-perfect performance (mean = 0.99, SD = 0.12) on the TOSLS,, exceeding the highest human scores of 0.85, SD = 0.13 (z = 1.78). While o1-preview excelled in structured tasks, it showed limitations in problem-solving and adaptive reasoning. These results demonstrate the potential of AI to complement education in structured assessments but highlight the need for ethical oversight and refinement for broader applications.
Study at a glance
- Question: Can OpenAI's o1 model outperform humans in complex thinking and reasoning tasks?
- Setting and data: OpenAI's o1-preview model and human participants from diverse educational levels, including undergraduate and postgraduate students.
- Method: Compares the AI model's performance with human scores across multiple higher-order thinking tests.
Why it matters
Whether AI models like OpenAI o1 can outperform humans in higher-order cognitive thinking remains an open question. Assessing AI against human participants from diverse educational levels tests abilities like critical, creative, and scientific reasoning.
Key findings
- Against undergraduate and postgraduate students, the AI scored higher in critical thinking, achieving 24.33 compared to 13.8 for undergraduates and 18.39 for postgraduates.
- The AI surpassed humans across benchmark tests of scientific reasoning, data literacy, systematic thinking, and creative originality.
- Despite excelling at structured tasks, the model showed limitations in problem-solving and adaptive reasoning.
Summary drafted with AI from the paper’s abstract and may contain errors. See the full paper for details.
Who age?
Why use AI?
Study design
