AI Wrote the Note. Can AI Check It? Auditing AI-Generated Psychology Session Notes

Many psychologists now use AI note-takers that record a session, transcribe it and summarise it into a clinical note. When the summary gets something wrong, the error lands in the note, and from there can flow into reports, assessments and letters built on it. Australian medico-legal indemnity insurers advise practitioners to check every note against its transcript, a time cost that works against the main reason for adopting these tools.

This project tests whether a large language model can carry out a first-pass check. Three commercial AI note-taking vendors produced 40 notes from 14 counselling transcripts in the open-source AnnoMI Motivational Interviewing dataset (Wu et al., 2022). Each note and its transcript were given to Claude Sonnet 5.5 five times, in independent API calls, with a prompt to identify wording in the note that the transcript could not confirm and to explain the discrepancy in plain language. Only issues flagged in at least three of the five runs were kept. Omissions were deliberately out of scope.

The process produced 131 stable flags. 39 of the 40 notes had at least one; 35 had at least one flag where the note altered the meaning of what was said, and 30 had at least one flag for content not present in the session. A manual review of a random sample of about 15 notes found the flags held up against the transcripts. The output is a practitioner-facing report that marks each flagged passage and links it to the verbatim transcript lines, so the clinician, not the model, decides what to correct.
These are preliminary findings. They assume an accurate transcript, and the manual check was a single-reviewer sample rather than a formal validation.

Reference:
Wu, Z., Balloccu, S., Kumar, V., Helaoui, R., Reiter, E., Reforgiato Recupero, D., & Riboni, D. (2022). Anno-MI: A dataset of expert-annotated counselling dialogues. In ICASSP 2022 – 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6177–6181). IEEE. https://doi.org/10.1109/ICASSP43922.2022.9746035

Project Team and Collaborators:
Patrick Carroll, Honours student, Psychology, James Cook University