As doctors, we’re fond of telling patients not to eat junk food. Fast food is tempting because it’s fast and easy, but it’s really unhealthy. Doctors are increasingly facing similar decisions and temptations to “eat the AI fast food” in their clinical practice. AI has already and will continue to rapidly transform healthcare; General Practitioners (GP’s) are increasingly turning to AI-driven tools to save time, enhance documentation, and support clinical decision-making. However, the integration of AI into your patient care, especially in high-stakes environments, demands a thoughtful, ethical approach that prioritises patient safety and upholds fundamental rights. “Human in The Loop” is an important guideline, but what does this really mean in best practice? What should you, and should you NOT do?
The EU AI Act and Human In The Loop
Let’s start with laws and legal guidelines, the EU AI Act (implemented August 2024), sets clear legal standards for the development and deployment of AI systems within the EU and for EU citizens[1][2]. For high-risk AI applications, like AI-driven medical devices and clinical decision support systems, the Act mandates that these technologies must be designed so that natural persons (i.e., human professionals) can effectively oversee their functioning throughout their use[3][4][5]. This “human in the loop” requirement is not just a technicality; it is a critical safeguard to prevent or minimize risks to health, safety, and fundamental rights. But what does that really mean?
Under Article 14 of the AI Act, high-risk AI systems must:
The Act also emphasises the importance of training and authorising human overseers, ensuring they have the competence and authority to intervene when necessary[5]. While the Act provides a strong foundation for ethical AI use, it leaves some ambiguity regarding the specifics of human oversight responsibilities, highlighting the ongoing need for clear, practical guidance in clinical settings[5]. Our aim is to define some easy-to-follow best-practice guidelines.
The Risks of AI-Generated Doctor-Patient Interaction Summaries
One area of particular concern is the use of AI to automate the creation of doctor-patient interaction summaries. While these tools promise to reduce administrative burden and improve documentation efficiency, they introduce significant risks that are not always immediately apparent.
Hallucinations and Factual Errors
AI models, particularly large language models (LLMs), are prone to “hallucinations” i.e. generating factually incorrect statements or fabricating details that were never discussed during a consultation[6][7][8]. Recent studies have shown that AI-generated medical summaries frequently contain inaccuracies, including incorrect patient information, misrepresented symptoms, and erroneous medication instructions[6][8]. In one real-world example, an AI transcription tool introduced exaggerated details into a patient’s medical record, which were only discovered upon careful review by the patient herself[7].
Errors of Omission
Perhaps even more concerning is the risk of errors of omission. AI systems may fail to detect or record important statements made during a consultation, leading to incomplete or misleading medical records[9]. Because these omissions are not obvious—especially in long, detail-driven summaries—they are especially difficult for doctors to detect during post-call reviews. This can have serious clinical consequences, as vital information may be overlooked, potentially affecting diagnosis, treatment, and patient safety[9]. Because this technology is so new
Clinical and Legal Implications
The variability and unpredictability of AI-generated summaries mean that even with identical inputs, the output can differ in content, emphasis, and organisation[9]. This variability can subtly influence clinical decision-making, nudging doctors toward different diagnostic or treatment paths based on what the AI chooses to include or exclude. Moreover, the legal implications are significant: if AI-generated inaccuracies or omissions go unnoticed, they may only come to light when medical decisions are challenged or when records are used as evidence in legal proceedings[7].
The Way Forward: Ethical AI in General Practice
To use AI ethically in general practice, it is essential to:
Conclusion - Real-time Human-AI teaming.
AI has the potential to revolutionise general practice, but its use must be guided by strong ethical principles and regulatory frameworks. The persistent challenges of AI hallucination and omissions in AI-generated summaries underscore the need for real-time human doctor in the loop confirmation when creating medical notes and patient charts. Doctors are at significant professional risk when using AI to summarise and recommend care plans especially after long consultations.
