About this Physics and Astronomy article
Large Language Models Encode Clinical Knowledge by Karan Singhal; Shekoofeh Azizi; Tao Tu; S. Sara Mahdavi; Jason Wei; Hyung Won Chung; Nathan Scales; Ajay Tanwani; Heather Cole-Lewis; Stephen Pfohl; Perry Payne; Martin Seneviratne; Paul Gamble; Chris Kelly; Abubakr Babiker; Nathanael Schärli; Aakanksha Chowdhery; Philip Mansfield; Dina Demner-Fushman; Blaise Agüera y Arcas; Dale Webster; Greg S. Corrado; Yossi Matias; Katherine Chou; Juraj Gottweis; Nenad Tomasev; Yun Liu; Alvin Rajkomar; Joelle Barral; Christopher Semturs; Alan Karthikesalingam; Vivek Natarajan is a Physics and Astronomy article available to read on EtoBox.
Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model^1^ (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM^2^ on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA^3^, MedMCQA^4^, PubMedQA^5^ and Measuring Massive Multitask Language Understanding (MMLU) clinical topics^6^), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However,
It is typically read by researchers, students, and practitioners in Physics and Astronomy.
- Author
- Karan Singhal; Shekoofeh Azizi; Tao Tu; S. Sara Mahdavi; Jason Wei; Hyung Won Chung; Nathan Scales; Ajay Tanwani; Heather Cole-Lewis; Stephen Pfohl; Perry Payne; Martin Seneviratne; Paul Gamble; Chris Kelly; Abubakr Babiker; Nathanael Schärli; Aakanksha Chowdhery; Philip Mansfield; Dina Demner-Fushman; Blaise Agüera y Arcas; Dale Webster; Greg S. Corrado; Yossi Matias; Katherine Chou; Juraj Gottweis; Nenad Tomasev; Yun Liu; Alvin Rajkomar; Joelle Barral; Christopher Semturs; Alan Karthikesalingam; Vivek Natarajan
- Publisher
- Nature Publishing Group UK
- Published
- 2023
- Language
- EN
- Field
- Physics and Astronomy (Physical Sciences)