The performance of ChatGPT on short-answer questions in a psychiatry examination: A pilot study
Loading...
Supplementary material
Other Title
Authors
Hsu, Che-Wei
Author ORCID Profiles (clickable)
Degree
Grantor
Date
2024
Supervisors
Type
Journal Article
Ngā Upoko Tukutuku (Māori subject headings)
Keyword
artificial intelligence
clinical skill assessment
examination
psychiatric education
clinical skill assessment
examination
psychiatric education
ANZSRC Field of Research Code (2020)
Citation
Lin, C-C., du Plooy, K., Gray, A., Brown, D., Hobbs, L., Patterson, T., Tan, V., Fridberg, D., & Hsu, C-W. The performance of ChatGPT on short-answer questions in a psychiatry examination: A pilot study. Taiwanese Journal of Psychiatry, 38(2), 94-98.https://doi.org/10.4103/TPSY.TPSY_19_24
Abstract
Objectives: We compared ChatGPT’s performance to medical students’ on psychiatry examinations and investigated whether raters could distinguish answers between them.
Methods: We used a copy of short-answer questions from a psychiatry examination to compare the performance of three randomized groups – ChatGPT, student, and hybrid (student-modified ChatGPT responses). Furthermore, we investigated raters’ ability to identify response origins.
Results: ChatGPT-assisted answers, both ChatGPT alone (p < 0.001) and hybrid (p < 0.001), had significantly better examination performance than did independent students work. Raters showed high accuracy in identifying the origin of the responses, correctly identifying 92% of both students’ and ChatGPT-assisted responses. But raters were only 61% accurate in making the distinction between ChatGPT and hybrid answers.
Conclusion: ChatGPT showed superior performance in a psychiatry examination compared to students’ work, but raters showed high accuracy in distinguishing them. Further investigation is warranted to optimize the advantages and mitigate the drawbacks of incorporating such technology into psychiatric education and health care.
Publisher
Wolters Kluwer Medknow Publications
Permanent link
Link to ePress publication
DOI
https://doi.org/10.4103/TPSY.TPSY_19_24
Copyright holder
Authors.
Copyright notice
CC BY-NC-SA V4.0