AI Compliance Checklist for Healthcare Startups in 2026
Six things to settle before you deploy, including the two rules that took effect in 2025 and are still missing from most compliance checklists you will find on this subject.
Healthcare AI compliance is usually described as a hurdle at the end of a project. It is better understood as a set of constraints on the architecture, because almost every requirement below is cheap if you know it on day one and expensive if you learn it in month nine. The most costly example is the simplest: whether protected health information crosses your trust boundary at inference time. Decide that early and everything else follows. Decide it late and you are rebuilding the data layer.
This checklist covers the six areas we work through before writing code. It is written for the engineering lead or technical founder who has to build the thing, not for counsel, and it is not legal advice.
Step 1: Map the data flows, and mark the boundary#
Before any safeguard, draw where PHI actually goes. The useful version of this diagram is not a list of components, it is a line around the systems you control, with every crossing of that line marked and justified.
Inside your trust boundary
Sources
EHR via FHIR, patient uploads, device feeds, claims
Processing
De-identification, chunking, embedding, retrieval
Storage
Database, vector index, object store, backups, logs
Every arrow below crosses the boundary
Third parties, each needing a BAA
Model provider
Inference calls carrying clinical context
Cloud and infrastructure
Compute, storage, managed services
Observability
Prompt tracing and error reporting, the one teams forget
The observability box is the one that catches people. Prompt tracing and error monitoring tools are added by engineers for good reasons, often after the compliance review, and they quietly ship full prompt contents, which in a clinical application means PHI, to a vendor nobody put in scope. Scrub payloads at the SDK boundary or self host the collector.
Also decide explicitly whether you need PHI at inference at all. Where a use case can run on data de-identified under the HIPAA Safe Harbor, meaning the eighteen identifier categories removed, or under a documented expert determination, a large part of this checklist stops applying to that path. It is worth an afternoon to find out.
Step 2: HIPAA technical safeguards, read for AI systems#
The Security Rule predates all of this and maps onto it cleanly once you know where the new copies of PHI live. They are mostly in places that did not exist in a traditional application.
| Safeguard | Requirement | Where AI adds a new copy of PHI |
|---|---|---|
| Encryption at rest | AES-256 across all PHI storage. | The vector index, any embedding cache, and model weights or adapters fine tuned on PHI. All three are PHI stores and are routinely left out of the inventory. |
| Encryption in transit | TLS 1.2 or better on every connection. | Calls to model providers, and traffic to any evaluation or tracing service. |
| Access control | Role based, least privilege. | Permissions must filter retrieval inside the query. Filtering after the model has read a document is not access control. |
| Audit logging | Every PHI access recorded. | Each query, the retrieved record identifiers, the model and version, the output, and whether a clinician accepted it. |
| Integrity and retention | PHI not improperly altered or destroyed. | Prompt logs are PHI and inherit your retention and deletion policy, including when a patient exercises a right. |
Step 3: Settle the medical device question before you build#
This determines whether you are shipping software or seeking clearance, and it is the one decision that can invalidate a roadmap. The 21st Century Cures Act carved certain clinical decision support out of the device definition, and the FDA set out how it reads that exemption in its final clinical decision support guidance in September 2022. The carve out is narrower than most teams assume.
Not analyzing a signal or image
No device data, no medical image, no pattern from a signal acquisition system
Displays or analyzes medical information
Of a kind normally communicated between clinicians
Supports a recommendation
Rather than providing a specific directive output
Basis is independently reviewable
The clinician can see and evaluate why, and need not rely on it
Step 4: Bias testing, now with a paper trail requirement#
Test across age, sex, race and ethnicity, language, disability status and payer or socioeconomic proxies, measuring not just overall accuracy but the error rates that matter clinically within each group. A model with equal accuracy everywhere can still have very different false negative rates by group, and in a diagnostic path the false negative is the harm.
Two practical notes. Watch for proxies: healthcare cost is a well known proxy for need that encodes historical access inequity, so a model using spend to predict risk can systematically under-serve the people who historically received less care. And keep the documentation, because under Section 1557 the obligation is to make reasonable efforts to identify and mitigate the risk, and an undocumented effort is indistinguishable from none. Fairlearn and Aequitas are both fine for the measurement.
Step 5: Vendor due diligence, with the questions that get real answers#
- A signed BAA from every vendor touching PHIIncluding the model provider, the cloud platform, and any tracing, analytics or support tool in the path. Availability differs by product tier and by how you access a model, so confirm it for the specific offering you are integrating rather than for the company name.
- Training opt out in writing, not in a blog postConfirm in the contract that your inputs and outputs are not used to train or improve their models, and get the retention period for prompt logs in the same document.
- Zero or short retention on the inference pathMany providers hold request data for abuse monitoring by default. Ask what the configured retention is for your account, and what deletion on request actually means operationally.
- SOC 2 Type II, read rather than collectedCheck the scope section covers the service you are using and look at the exceptions. A report with the relevant system out of scope is a filing, not an assurance.
- Subprocessor list and breach notification timingYou carry the HIPAA notification clock. A vendor who will tell you within seventy two hours is workable; one who commits to nothing specific is not.
- A written exit pathHow you get your data out and what is deleted on termination, agreed before you are dependent on them.
Step 6: Monitoring, because the model is not the thing that drifts first#
Model weights are static. What moves underneath them is everything else: a coding practice changes, an EHR upgrade alters how a field is populated, a new patient population arrives, a clinical guideline is revised. The system degrades without anything in it changing, which is why monitoring inputs matters at least as much as monitoring outputs.
- Input distribution monitoringAlert on shifts in the data reaching the model, not only on its predictions. This is the earliest signal you will get.
- Clinician override rateThe most honest performance measure available, and free. A rising override rate is the system telling you it has drifted before any metric does.
- Real world outcome trackingCompare predictions against what actually happened, segmented by the same demographic groups you tested at launch.
- Scheduled revalidationQuarterly performance and bias retest against a held out set, on the calendar rather than triggered by an incident.
- An incident path that includes rollbackDecide in advance who can turn the system off, how clinicians are told, and what the fallback workflow is. Write it before you need it.
None of this is exotic engineering. Nearly all of it is work that is trivial while the architecture is still on a whiteboard and painful once there is a production data layer assuming otherwise. The sequence that saves the most money is also the shortest: decide whether PHI crosses the boundary, settle the device question, and make every output cite its sources. The rest of the checklist is much easier once those three are answered.
Building clinical AI that has to survive a compliance review?
Book a free 30 minute call. We will work through the data flow boundary and the device question with you, which are the two decisions that most often determine whether the rest of the build is straightforward or a rewrite.
Book a Strategy Call →