AI scribes generated notes full of hallucinations and errors, Ontario auditor general report finds

Ontario’s auditor general has put out a report on Tuesday on artificial intelligence adoption in the government, finding that AI scribes are making mistakes at a high rate while putting patient health data at risk.As described in the report, evaluators ran two transcription tests for all 20 of the approved AI scribe systems in the province and found that 12 of these scribes (60%) created inaccurate information, capturing a different drug than what was prescribed by the doctor.In addition, nine of the 20 (45%) scribes had hallucinations, sometimes making suggestions to patient treatment plans that were never discussed such as referring the patient to therapy or ordering blood tests.“Evaluators also noted hallucinations that could impact patients’ health. For example, the submitted notes included statements that there were ‘no masses found’ or that there was presence of anxiety in the patient, although this information was not discussed in the recordings,” the report says.On top of that, the vast majority of the scribes (85%) also missed key details about the patients’ mental health issues, even though this was mentioned in the simulated recordings.“Inaccuracies in medical notes generated by AI Scribe systems could potentially result in inadequate or harmful treatment plans that may potentially impact patient health outcomes. It is important that the AI Scribe systems are tested to provide assurances as to the quality of their generated notes and to minimize inaccuracies,” the report said.In addition, the auditor general’s office found that out of the approved vendors, 11 failed to submit third-party security assessment reports, and the province relied on the vendors’ self-attestation of compliance to privacy requirements “without requiring supporting evidence or independent verification.”“This approach allowed vendors to potentially overstate their compliance with security and privacy requirements,” the auditor general’s office said. The report also found that the province “did not perform a comprehensive evaluation of vendors” to mitigate the risk of bias. In the province’s selection criteria for AI scribe vendors, venders were given a score based on how well their product performed in several criteria, with each criteria having a different weighting. The auditor general’s office took issue with the fact that the weighting for the “bias controls” criteria only counted for 2% of the score, while the category for third-party security reports was only 4%. The “accuracy” criteria was also only 4% of the score, but on the other hand, whether the company had a presence in Ontario counted for 30% of the score.The vendors were also not required to provide live demos of their AI scribe systems to their evaluators, creating a “potential risk that vendors could process the recordings multiple times or alter the system-generated notes, compromising the integrity of the evaluation process.”

AI scribes generated notes full of hallucinations and errors, Ontario auditor general report finds

Register for free access or login.

Related Topics