High-risk AI in healthcare: what EU and US frameworks actually require

Healthcare AI is operating in a pre-enforcement window right now. No headline fine has landed yet for a clinical AI failure in the way that financial services AI has attracted multi-million dollar penalties. But the legal infrastructure is in place, and regulators on both sides of the Atlantic have signalled that scrutiny is coming. The organisations that prepare now will be the ones whose deployments survive the first wave of enforcement.

Why healthcare AI is explicitly high-risk under the EU AI Act

The EU AI Act's high-risk classification covers AI systems used as safety components in medical devices, AI used to triage patients or make clinical decisions, and AI tools used to assess risk in healthcare settings. This is not a borderline case — the Act names medical and healthcare AI directly in Annex III. If your organisation deploys AI that contributes to clinical decisions about patients, it is almost certainly a high-risk system under the Act.

High-risk designation triggers the full set of obligations: technical documentation, data governance, human oversight requirements, accuracy and robustness testing, post-market monitoring, and registration in the EU AI database before deployment. For a healthcare AI system, this means your training data must be documented for potential regulatory inspection, your model must be tested against the specific populations it will be used on, and a human must be able to understand and override every output it generates.

What "human oversight" means for a diagnostic AI

The phrase "human oversight" appears throughout the EU AI Act, but it is frequently misunderstood. Having a clinician who can technically override a recommendation is not sufficient if the override rate is effectively zero because the interface makes the model's output the path of least resistance. Regulators and auditors look at whether oversight is meaningful in practice, not just nominally available.

For a diagnostic or triage AI, meaningful oversight requires:

• The clinician seeing the model's output can also see the factors driving it — not just a confidence score, but the specific indicators the model weighted.
• The interface design does not present the model's recommendation in a way that anchors the clinician's judgment before they have formed their own.
• There is a logged mechanism for the clinician to record disagreement with the model output, and those logs are reviewed to detect systematic override patterns.
• Training covers not just how to use the tool but when to question it — and that training is documented.

The US regulatory picture

In the US, healthcare AI sits at the intersection of FDA regulatory authority over Software as a Medical Device (SaMD), OCR enforcement under HIPAA (which covers the data used to train and operate these models), and emerging state-level AI legislation. Texas's TRAIGA law (effective January 2026 — verify the current date and scope against the enacted statute before relying on this) includes healthcare-specific provisions that many teams operating in Texas have not yet mapped. The overlap between TRAIGA's AI obligations and HIPAA's data protection requirements creates compliance obligations that do not cleanly belong to either framework — and that gap is where enforcement risk accumulates.

The FDA's predetermined change control plan guidance for AI/ML-based SaMD adds a further layer: if your clinical AI updates itself based on new data (as most modern diagnostic AI does), the FDA requires a documented plan covering what changes are permitted, what testing they require, and how changes are monitored. This is not optional for devices in FDA jurisdiction.

What an audit looks at for healthcare AI

An iDharma audit of a healthcare AI deployment checks against the EU AI Act's Annex III obligations, ISO/IEC 42001 AI management system requirements, and — where applicable — FDA SaMD guidance and HIPAA data governance obligations. The specific questions we answer:

• Training data documentation: Is the provenance of your training data documented? Were protected characteristics handled consistently with non-discrimination requirements? Were validation data sets drawn from populations representative of your intended patient population?
• Accuracy and robustness: Has the model been tested against the specific clinical contexts it will encounter? Are accuracy metrics reported at the sub-group level, not just in aggregate?
• Human oversight mechanisms: Are override pathways meaningful and documented? Are override patterns monitored?
• Post-deployment monitoring: Is there a live monitoring process that would detect model drift or performance degradation before it reaches patients?
• Incident response: If the model produces a clinically harmful output, is there a documented response plan? Has it been tested?

The pre-enforcement window is short

Healthcare is at the same point financial services AI was in 2023, before the enforcement actions that clarified what the rules meant in practice. The organisations that audited and fixed their models before the first fines landed are now in a structurally better position — both legally and competitively — than those that waited. The window to get ahead of enforcement in healthcare AI is open. It will not stay open indefinitely. Request an audit or use the free Risk Snapshot to see where your exposure sits.