- Scholarly Contributions -
Highlighting the scholarly work of tomorrow’s clinical informatics leaders
March 3, 2025
Physician Perspectives on Ambient AI Scribes
Shreya J Shah 1 2, Trevor Crowell 2, Yejin Jeong 2, Anna Devon-Sand 2, Margaret Smith 2, Betsy Yang 1 2 3, Stephen P Ma 1, April S Liang 1, Clarissa Delahaie 4, Caroline Hsia 4, Tait Shanafelt 1 5, Michael A Pfeffer 1 4, Christopher Sharp 1, Steven Lin 1 2, Patricia Garcia 1
Affiliations
1Department of Medicine, Stanford University School of Medicine, Stanford, California.
2Stanford Healthcare AI Applied Research Team, Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, California.
3Geriatric Research Education and Clinical Center, Veterans Administration Healthcare System, Palo Alto, California.
4Technology and Digital Solutions, Stanford Medicine, Stanford, California.
5WellMD Center, Stanford University School of Medicine, Stanford, California.
Abstract
Importance: Limited qualitative studies exist evaluating ambient artificial intelligence (AI) scribe tools. Such studies can provide deeper insights into ambient AI implementations by capturing lived experiences.
Objective: To evaluate physician perspectives on ambient AI scribes.
Design, setting, and participants: A qualitative study using semistructured interviews guided by the Reach, Efficacy, Adoption, Implementation, Maintenance/Practical, Robust Implementation, and Sustainability Model (RE-AIM/PRISM) framework, with thematic analysis using both inductive and deductive approaches. Physicians participating in an AI scribe pilot that included community and faculty practices, across primary care and ambulatory specialties, were invited to participate in interviews. This ambient AI scribe pilot at a health care organization in California was conducted from November 2023 to January 2024.
Main outcome and measures: Facilitators and barriers to adoption, practical effectiveness, and suggestions for improvement to enhance sustainability.
Results: Twenty-two semistructured interviews were conducted with AI pilot physicians from primary care (13 [59%]) and ambulatory specialties (9 [41%]), including physicians from community practices (12 [55%]) and faculty practices (10 [45%]). Facilitators to adoption included ease of use, ease of editing, and generally positive perspectives of tool quality. Physicians expressed positive sentiments about the impact of the ambient AI scribe tool on cognitive demand (16 of 16 comments [100%]), temporal demand (28 comments [62%]), work-life integration (10 of 11 comments [91%]), and overall workload (8 of 9 comments [89%]). Physician perspectives of the impact of the ambient AI scribe tool on their engagement with patients were mostly positive (38 of 56 comments [68%]). Barriers to adoption included limited functionality with non-English speaking patients and lack of access for physicians without a specific device. Physician perspectives on accuracy and style were largely negative, particularly regarding note length and editing requirements. Several specific suggestions for tool improvement were identified, and physicians were optimistic regarding the potential for long-term use of ambient AI scribes.
Conclusion and relevance: In this qualitative study, ambient AI scribes were found to positively impact physician workload, work-life integration, and patient engagement. Key facilitators and barriers to adoption were identified, along with specific suggestions for tool improvement. These findings suggest the potential for ambient AI scribes to reduce clinician burden, with user-centered recommendations offering practical guidance on ways to improve future iterations and improve adoption.
Read the full article here:
January 19, 2025
Clinical entity augmented retrieval for clinical information extraction
Ivan Lopez # 1 2, Akshay Swaminathan # 3 4, Karthik Vedula 5, Sanjana Narayanan 6, Fateme Nateghi Haredasht 6, Stephen P Ma 7, April S Liang 8, Steven Tate 9, Manoj Maddali 4 10, Robert Joseph Gallo 11 12, Nigam H Shah 6 13 14, Jonathan H Chen 4 6 7 14 15
Affiliations
1Stanford University School of Medicine, Stanford, CA, USA. ivlopez@stanford.edu.
2Department of Biomedical Data Science, Stanford, CA, USA. ivlopez@stanford.edu.
3Stanford University School of Medicine, Stanford, CA, USA.
4Department of Biomedical Data Science, Stanford, CA, USA.
5Poolesville High School, Poolesville, MD, USA.
6Stanford Center for Biomedical Informatics Research, Stanford, CA, USA.
7Division of Hospital Medicine, Stanford University School of Medicine, Stanford, CA, USA.
8Division of Clinical Informatics, Stanford University School of Medicine, Stanford, CA, USA.
9Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA.
10Division of Pulmonary, Allergy, and Critical Care Medicine, Stanford University School of Medicine, Stanford, CA, USA.
11Center for Innovation to Implementation, VA Palo Alto Healthcare System, Menlo Park, CA, USA.
12Department of Health Policy, Stanford University, Stanford, CA, USA.
13Technology and Digital Solutions, Stanford Healthcare, Palo Alto, USA.
14Clinical Excellence Research Center, Stanford University School of Medicine, Stanford, CA, USA.
15Department of Medicine, Stanford, CA, USA.
#Contributed equally.
Large language models (LLMs) with retrieval-augmented generation (RAG) have improved information extraction over previous methods, yet their reliance on embeddings often leads to inefficient retrieval. We introduce CLinical Entity Augmented Retrieval (CLEAR), a RAG pipeline that retrieves information using entities. We compared CLEAR to embedding RAG and full-note approaches for extracting 18 variables using six LLMs across 20,000 clinical notes. Average F1 scores were 0.90, 0.86, and 0.79; inference times were 4.95, 17.41, and 20.08 s per note; average model queries were 1.68, 4.94, and 4.18 per note; and average input tokens were 1.1k, 3.8k, and 6.1k per note for CLEAR, embedding RAG, and full-note approaches, respectively. In conclusion, CLEAR utilizes clinical entities for information retrieval and achieves >70% reduction in token usage and inference time with improved performance compared to modern methods.
Read the full article here:
February 25, 2025
Ambient artificial intelligence scribes: utilization and impact on documentation time
Stephen P Ma 1, April S Liang 1, Shreya J Shah 1 2, Margaret Smith 2, Yejin Jeong 2, Anna Devon-Sand 2, Trevor Crowell 2, Clarissa Delahaie 3, Caroline Hsia 3, Steven Lin 1 2, Tait Shanafelt 1 4, Michael A Pfeffer 1 3, Christopher Sharp 1, Patricia Garcia 1
Affiliations
1Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305, United States.
2Stanford Healthcare AI Applied Research Team, Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, CA 94305, United States.
3Technology and Digital Solutions, Stanford Medicine, Stanford, CA 94305, United States.
4WellMD Center, Stanford University School of Medicine, Stanford, CA 94305, United States.
Abstract
Objectives: To quantify utilization and impact on documentation time of a large language model-powered ambient artificial intelligence (AI) scribe.
Materials and methods: This prospective quality improvement study was conducted at a large academic medical center with 45 physicians from 8 ambulatory disciplines over 3 months. Utilization and documentation times were derived from electronic health record (EHR) use measures.
Results: The ambient AI scribe was utilized in 9629 of 17 428 encounters (55.25%) with significant interuser heterogeneity. Compared to baseline, median time per note reduced significantly by 0.57 minutes. Median daily documentation, afterhours, and total EHR time also decreased significantly by 6.89, 5.17, and 19.95 minutes/day, respectively.
Discussion: An early pilot of an ambient AI scribe demonstrated robust utilization and reduced time spent on documentation and in the EHR. There was notable individual-level heterogeneity.
Conclusion: Large language model-powered ambient AI scribes may reduce documentation burden. Further studies are needed to identify which users benefit most from current technology and how future iterations can support a broader audience.
Keywords: ambient intelligence; ambient scribes; artificial intelligence; documentation; informatics.
Read the full article here:
February 1, 2025
Ambient artificial intelligence scribes: physician burnout and perspectives on usability and documentation burden
Shreya J Shah 1 2, Anna Devon-Sand 2, Stephen P Ma 1, Yejin Jeong 2, Trevor Crowell 2, Margaret Smith 2, April S Liang 1, Clarissa Delahaie 3, Caroline Hsia 3, Tait Shanafelt 1 4, Michael A Pfeffer 1 3, Christopher Sharp 1, Steven Lin 1 2, Patricia Garcia 1
Affiliations
1Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305, United States.
2Stanford Healthcare AI Applied Research Team, Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, CA 94305, United States.
3Technology and Digital Solutions, Stanford Medicine, Stanford, CA 94305, United States.
4WellMD Center, Stanford University School of Medicine, Stanford, CA 94305, United States.
Abstract
Objective: This study evaluates the pilot implementation of ambient AI scribe technology to assess physician perspectives on usability and the impact on physician burden and burnout.
Materials and methods: This prospective quality improvement study was conducted at Stanford Health Care with 48 physicians over a 3-month period. Outcome measures included burden, burnout, usability, and perceived time savings.
Results: Paired survey analysis (n = 38) revealed large statistically significant reductions in task load (-24.42, p <.001) and burnout (-1.94, p <.001), and moderate statistically significant improvements in usability scores (+10.9, p <.001). Post-survey responses (n = 46) indicated favorable utility with improved perceptions of efficiency, documentation quality, and ease of use.
Discussion: In one of the first pilot implementations of ambient AI scribe technology, improvements in physician task load, burnout, and usability were demonstrated.
Conclusion: Ambient AI scribes like DAX Copilot may enhance clinical workflows. Further research is needed to optimize widespread implementation and evaluate long-term impacts.
Keywords: ambient intelligence; ambient scribes; artificial intelligence; documentation; informatics.