The complete security guide for AI clinical decision support — FDA SaMD regulatory classification, adversarial clinical data attacks, EHR injection vulnerabilities, patient safety governance, and the specific AIZA-HexTyx controls required for safe clinical AI deployment.
Clinical decision support AI operates in an environment where the consequences of a successful adversarial attack are not financial loss or data exposure — they are patient harm. An AI system that influences a clinician's decision about medication dosage, diagnostic prioritisation, or treatment selection is operating in the highest-stakes decision environment in enterprise AI. The FDA's recognition of this reality — expressed through its Software as a Medical Device (SaMD) regulatory framework — reflects a fundamental principle: when AI influences clinical decisions, it must be held to medical device standards, not software application standards.
The security implications are specific and demanding. An adversarial attack against a clinical AI system doesn't just need to bypass a content filter — it needs to influence a clinician's decision in a direction that harms a patient. This is a harder attack to execute but also a harder attack to detect, because a subtly wrong recommendation from a clinical AI may look indistinguishable from a correct recommendation to everyone who sees it, including the clinician acting on it. The healthcare AI security foundation: Healthcare AI Security: HIPAA-Compliant LLM Deployment →
The FDA's Software as a Medical Device regulatory pathway applies to AI that meets the definition: software intended to be used for medical purposes that performs these purposes without being part of a hardware medical device. Clinical AI systems that fall within this definition include: AI that diagnoses or identifies disease from imaging, AI that recommends or prioritises diagnostic tests, AI that suggests medication dosing or selection, and AI that predicts clinical deterioration or adverse events.
Risk classification under FDA SaMD framework: The FDA classifies SaMD using a two-axis risk matrix: the significance of the information provided by the SaMD to the healthcare decision (critical, serious, or non-serious) and the state of the healthcare situation or condition (critical, serious, or non-serious). AI that provides information for immediate treatment decisions in critical conditions sits in the highest-risk quadrant and is subject to the most rigorous regulatory requirements, including premarket approval (PMA) rather than 510(k) clearance.
For security purposes, FDA SaMD risk classification determines the depth of adversarial testing required before deployment:
| SaMD Classification | Healthcare Situation | Decision Type | Required Security Gate |
|---|---|---|---|
| Class III (highest) | Critical (life-threatening) | Treat, diagnose, drive | Risk score < 15, bypass < 5%, zero confirmed leaks, independent red team |
| Class II (moderate) | Serious (long-term impact) | Aid diagnosis, treatment | Risk score < 25, bypass < 10%, zero confirmed leaks |
| Class I (lower) | Non-serious (limited impact) | Inform, administrative | Risk score < 40, bypass < 25% |
FDA's 2023 action plan for AI/ML-based SaMD specifically requires "predetermined change control plans" — documentation of what changes to the AI system will require new regulatory review. For security purposes, this means: any change to the model, system prompt, or training data that affects the system's clinical outputs must be assessed against the predetermined change control plan before deployment. A system prompt change that increases the bypass rate by more than 5 percentage points triggers a change control review.
Clinical AI systems face adversarial attack vectors that don't exist in general enterprise AI deployments. The attackers are not script kiddies running known jailbreak templates — they are sophisticated actors (nation-states, ransomware groups, or insiders) who understand clinical workflows and can craft attacks that are clinically plausible while being adversarially effective.
Adversarial EHR data manipulation: Electronic Health Record systems are the primary data source for clinical AI. If an attacker can modify EHR data — through a direct EHR system compromise, through a healthcare supply chain attack targeting an EHR vendor, or through social engineering that causes a clinician to enter false data — they can influence any downstream AI that uses that EHR data. The attack is invisible at the AI layer: the AI is processing legitimate EHR records; it's the underlying data that has been manipulated.
This is the clinical equivalent of training data poisoning: the attack happens at the data source, and its effects propagate to every AI decision that uses that data source. Defence requires data integrity controls at the EHR layer (audit logs, access controls, anomaly detection on EHR record modification patterns) combined with clinical AI output monitoring that detects when the AI's recommendations diverge from expected clinical patterns for a given patient profile.
Adversarial medical imaging inputs: For AI systems that process medical imaging (radiology AI, pathology AI, dermatology AI), adversarial image attacks — imperceptible perturbations to image pixel values that cause the AI to misclassify the image — are a specific and well-documented threat. An adversarial perturbation that causes a radiology AI to miss a lung nodule, or a pathology AI to classify a malignant cell as benign, requires no EHR access and no network penetration — only access to the imaging data pipeline at any point between acquisition and AI processing.
AIZA-HexTyx's multimodal_injection advanced module tests this attack class for AI systems that process medical imaging. For clinical imaging AI, this test must be run not just with standard adversarial perturbations but with clinically-calibrated perturbations designed to produce specific misclassifications — the test scenario that matches the real adversarial intent.
Indirect injection through clinical notes: Clinical AI systems that process free-text clinical notes — nursing notes, physician progress notes, discharge summaries — face indirect injection through those notes. A clinician whose account has been compromised, or a social engineering attack that causes a clinician to enter manipulated text, can plant adversarial instructions in clinical notes that the AI subsequently processes. For AI systems that use clinical notes as RAG source content, Aegis CP2 must validate every clinical note chunk before it enters the model context — treating clinical notes as untrusted content even though they appear to come from internal systems. The complete HIPAA technical safeguards guide: HIPAA Technical Safeguards for AI Systems: PHI in LLM Prompts →
The FDA's post-market surveillance requirements for SaMD AI require that manufacturers monitor deployed AI systems for safety signals — evidence that the AI's performance in real-world use is diverging from its validated performance in controlled testing. For security purposes, this requirement maps directly to runtime behavioral monitoring: the same Aegis signals that detect adversarial attacks also detect clinical performance drift that may constitute a patient safety signal.
Clinical safety signals from Aegis monitoring:
Recommendation anomaly detection: Configure MultiTurnTracker's semantic drift detection to measure drift in the clinical recommendation domain — not just in the adversarial attack domain. A clinical AI whose recommendation distribution shifts significantly over time (more aggressive treatment recommendations, more frequent "urgent" priority assignments, systematic changes in diagnostic confidence) may be exhibiting drift that constitutes a patient safety signal requiring clinical review, not just a security review.
Retrieval anomaly monitoring: For RAG-based clinical AI that retrieves from clinical knowledge bases, Aegis CP2's block rate monitoring serves dual purpose: high block rates from clinical knowledge sources indicate either adversarial corpus poisoning (security incident) or significant changes in the clinical knowledge base content (potential patient safety issue if the content changes are unvalidated). Both require investigation.
Output confidence calibration monitoring: Clinical AI systems should be monitored for changes in output confidence calibration — whether the AI's stated confidence levels correlate correctly with its actual accuracy. A security-degraded model (one that has been subjected to successful adversarial inputs) often shows miscalibration as one of its first observable effects: the model becomes confidently wrong. Monitor confidence calibration as a patient safety metric alongside accuracy metrics.
Clinical AI governance committee requirements: FDA SaMD requirements, combined with The Joint Commission's standards for clinical decision support, require that clinical AI be governed by a multidisciplinary committee including clinical, technical, and quality representatives. This committee must review: AI system changes (predetermined change control plan review), post-market surveillance signals (safety and performance monitoring), adverse events involving AI (incident reporting), and the results of periodic adversarial testing (security validation).
The risk classification framework that maps FDA SaMD classes to AI security tiers: AI Risk Classification: The Complete Enterprise Governance Guide →
FDA's SaMD framework and The Joint Commission's clinical decision support standards both require that clinical AI operates as a decision support tool — augmenting clinician judgment, not replacing it. From a security perspective, this human oversight requirement is also a blast-radius limitation: a successful adversarial attack against a clinical AI system with mandatory human review produces a wrong recommendation that a clinician can catch; the same attack against a fully autonomous clinical AI system produces a wrong action that may harm a patient before any human intervenes.
The AIZA-HexTyx architecture supports human oversight through two mechanisms: the Dead Man's Switch (DMS) as the autonomous execution kill-switch for any clinical AI workflow that involves automated action, and Aegis audit logging as the immutable record of every AI interaction that supports retrospective clinical review. For clinical AI, the DMS should be configured with the shortest reasonable timeout — 60 seconds for systems with real-time clinical oversight, 120 seconds for systems with near-real-time oversight — because the consequence of a DMS timeout in a clinical context (AI system stops operating) is far preferable to the consequence of an undetected adversarial attack in a clinical context (patient harm).
The agentic runtime governance guide covering DMS architecture and human oversight controls: AI Autonomous Agentic Runtime Governance: Complete Enterprise Control Guide →
FDA SaMD compliance:
Adversarial clinical data protection:
Patient safety runtime monitoring:
The complete pre-deployment audit that produces FDA SaMD development documentation: How to Audit an AI Model Before Deployment: Complete Enterprise Guide →
Run a HexTyx scan and generate the complete clinical AI security evidence — adversarial validation report, MITRE ATT&CK layer, behavioral monitoring setup, and the predetermined change control plan template.
Generate Clinical AI Evidence Package →