How Should You Evaluate an AI Medical Scribe?

Do AI medical scribes make mistakes in clinical notes?

Yes. Published evaluations have found errors in AI-generated notes, and a clean-sounding draft is not proof that it accurately represents the visit. The evaluation question is not simply whether the note reads well; it is whether a clinician can verify the clinically important details against the encounter.

A controlled instrument-validation study published in the Journal of Medical Internet Research tested two commercial ambient scribe products using 11 outpatient encounter scripts. Researchers identified 127 errors across 44 draft notes; 31 notes, or 70%, contained at least one error. Omissions were the most frequent error type.

Omissions deserve particular attention because they are easy to miss. The study authors note that finding an omitted detail requires recalling what happened during the encounter; a reviewer cannot always spot what is absent by reading the draft alone. A useful evaluation process therefore compares notes with encounter details and explicitly checks for missing information, incorrect attribution between speakers, and details that were not actually stated.

The controlled setting also limits what can be inferred: scripted encounters may not represent the variability of routine clinical work. A separate primary-care evaluation of six AI scribes found that none consistently produced fully accurate, error-free transcripts or notes. Longer and more complex encounters, as well as multiple speakers, affected performance. Together, these findings make review capability and error reporting core product requirements, not finishing touches.

How well do AI scribes work with Epic or other EHRs?

Some AI scribes can be launched through common platforms or within an EHR, but that does not mean their data exchange or documentation workflow will fit your organization. Test the complete path from recording through draft generation, clinician edits, and final entry in the chart.

In the six-scribe primary-care study, five products generated a SOAP-format note about one minute after recording stopped in a standardized 15-minute encounter. Researchers found that four of six could be accessed through common platforms and launched within common EHRs, but data-exchange capabilities were limited. Speed at producing a draft and ease of integrating that draft into clinical documentation are different measures.

Before a pilot, check whether the tool works with the EHR configuration clinicians actually use, whether it supports the required note structure and templates, and how edits reach the final record. Test ordinary visits as well as longer, more complex encounters and visits with multiple speakers. Record where copy-and-paste, reformatting, or manual correction is required; those steps can determine whether a fast draft saves time in the real workflow.

Time savings should be measured rather than assumed. A three-month quality-improvement study of 45 physicians across eight ambulatory disciplines found that an ambient scribe was used in 9,629 of 17,428 encounters (55.25%). Median time per note decreased from 4.86 minutes before the pilot to 3.64 minutes during it, while utilization and results varied among users. These findings describe that study's setting, not a guaranteed result for another product or practice.

Do patients need to consent to an AI scribe recording a visit?

Consent requirements depend on the applicable law and organizational policy, so practices should settle the rules before using a scribe with patients. Patient communication is necessary even where a particular legal framework does not require explicit consent for individual care.

NHS England guidance advises informing patients at the start of a session. In England, under the common-law approach described in that guidance for individual care, explicit consent is not required; patients may dissent, and their wishes should be respected. The guidance also advises clinicians to check that the product has been approved by their organization and to follow local policies and legal requirements. These England-specific points should not be treated as a universal rule.

The American Psychological Association recommends that practices develop an informed-consent process and documentation policy and check applicable state recording and privacy laws. It notes that some states, including Illinois, require specific written informed consent before an AI tool records a clinical visit. The APA also recommends reviewing vendor privacy terms and business associate agreements when protected health information is involved.

A practical policy should explain when the tool is used, how a patient can decline, and what staff do if a patient objects. It should also establish who approves the product and how recording, privacy, and documentation questions are handled. Local legal and organizational requirements determine the particulars; a vendor's default settings do not answer them.