Back to Blog

AI Compliance Checklist for Healthcare Startups in 2026

Six things to settle before you deploy, including the two rules that took effect in 2025 and are still missing from most compliance checklists you will find on this subject.

Andriy Markiv February 15, 2026 9 min read
Share LinkedIn X Facebook

 

Healthcare AI compliance is usually described as a hurdle at the end of a project. It is better understood as a set of constraints on the architecture, because almost every requirement below is cheap if you know it on day one and expensive if you learn it in month nine. The most costly example is the simplest: whether protected health information crosses your trust boundary at inference time. Decide that early and everything else follows. Decide it late and you are rebuilding the data layer.

This checklist covers the six areas we work through before writing code. It is written for the engineering lead or technical founder who has to build the thing, not for counsel, and it is not legal advice.

Step 1: Map the data flows, and mark the boundary#

Before any safeguard, draw where PHI actually goes. The useful version of this diagram is not a list of components, it is a line around the systems you control, with every crossing of that line marked and justified.

Inside your trust boundary

Sources

EHR via FHIR, patient uploads, device feeds, claims

Processing

De-identification, chunking, embedding, retrieval

Storage

Database, vector index, object store, backups, logs

Every arrow below crosses the boundary

Third parties, each needing a BAA

Model provider

Inference calls carrying clinical context

Cloud and infrastructure

Compute, storage, managed services

Observability

Prompt tracing and error reporting, the one teams forget

Figure 1. The only question this diagram has to answer: at which arrow does PHI leave your control, and what contract covers it there. Every crossing needs a Business Associate Agreement or a reason it is not PHI any more.

The observability box is the one that catches people. Prompt tracing and error monitoring tools are added by engineers for good reasons, often after the compliance review, and they quietly ship full prompt contents, which in a clinical application means PHI, to a vendor nobody put in scope. Scrub payloads at the SDK boundary or self host the collector.

Also decide explicitly whether you need PHI at inference at all. Where a use case can run on data de-identified under the HIPAA Safe Harbor, meaning the eighteen identifier categories removed, or under a documented expert determination, a large part of this checklist stops applying to that path. It is worth an afternoon to find out.

Step 2: HIPAA technical safeguards, read for AI systems#

The Security Rule predates all of this and maps onto it cleanly once you know where the new copies of PHI live. They are mostly in places that did not exist in a traditional application.

SafeguardRequirementWhere AI adds a new copy of PHI
Encryption at restAES-256 across all PHI storage.The vector index, any embedding cache, and model weights or adapters fine tuned on PHI. All three are PHI stores and are routinely left out of the inventory.
Encryption in transitTLS 1.2 or better on every connection.Calls to model providers, and traffic to any evaluation or tracing service.
Access controlRole based, least privilege.Permissions must filter retrieval inside the query. Filtering after the model has read a document is not access control.
Audit loggingEvery PHI access recorded.Each query, the retrieved record identifiers, the model and version, the output, and whether a clinician accepted it.
Integrity and retentionPHI not improperly altered or destroyed.Prompt logs are PHI and inherit your retention and deletion policy, including when a patient exercises a right.

Step 3: Settle the medical device question before you build#

This determines whether you are shipping software or seeking clearance, and it is the one decision that can invalidate a roadmap. The 21st Century Cures Act carved certain clinical decision support out of the device definition, and the FDA set out how it reads that exemption in its final clinical decision support guidance in September 2022. The carve out is narrower than most teams assume.

  1. Not analyzing a signal or image

    No device data, no medical image, no pattern from a signal acquisition system

  2. Displays or analyzes medical information

    Of a kind normally communicated between clinicians

  3. Supports a recommendation

    Rather than providing a specific directive output

  4. Basis is independently reviewable

    The clinician can see and evaluate why, and need not rely on it

Figure 2. The exemption is conjunctive. Software has to satisfy all four to sit outside the device definition, and the fourth is where generative systems most often fail, because a fluent summary is not the same as showing the basis.

Step 4: Bias testing, now with a paper trail requirement#

Test across age, sex, race and ethnicity, language, disability status and payer or socioeconomic proxies, measuring not just overall accuracy but the error rates that matter clinically within each group. A model with equal accuracy everywhere can still have very different false negative rates by group, and in a diagnostic path the false negative is the harm.

Two practical notes. Watch for proxies: healthcare cost is a well known proxy for need that encodes historical access inequity, so a model using spend to predict risk can systematically under-serve the people who historically received less care. And keep the documentation, because under Section 1557 the obligation is to make reasonable efforts to identify and mitigate the risk, and an undocumented effort is indistinguishable from none. Fairlearn and Aequitas are both fine for the measurement.

Step 5: Vendor due diligence, with the questions that get real answers#

  • A signed BAA from every vendor touching PHIIncluding the model provider, the cloud platform, and any tracing, analytics or support tool in the path. Availability differs by product tier and by how you access a model, so confirm it for the specific offering you are integrating rather than for the company name.
  • Training opt out in writing, not in a blog postConfirm in the contract that your inputs and outputs are not used to train or improve their models, and get the retention period for prompt logs in the same document.
  • Zero or short retention on the inference pathMany providers hold request data for abuse monitoring by default. Ask what the configured retention is for your account, and what deletion on request actually means operationally.
  • SOC 2 Type II, read rather than collectedCheck the scope section covers the service you are using and look at the exceptions. A report with the relevant system out of scope is a filing, not an assurance.
  • Subprocessor list and breach notification timingYou carry the HIPAA notification clock. A vendor who will tell you within seventy two hours is workable; one who commits to nothing specific is not.
  • A written exit pathHow you get your data out and what is deleted on termination, agreed before you are dependent on them.

Step 6: Monitoring, because the model is not the thing that drifts first#

Model weights are static. What moves underneath them is everything else: a coding practice changes, an EHR upgrade alters how a field is populated, a new patient population arrives, a clinical guideline is revised. The system degrades without anything in it changing, which is why monitoring inputs matters at least as much as monitoring outputs.

  • Input distribution monitoringAlert on shifts in the data reaching the model, not only on its predictions. This is the earliest signal you will get.
  • Clinician override rateThe most honest performance measure available, and free. A rising override rate is the system telling you it has drifted before any metric does.
  • Real world outcome trackingCompare predictions against what actually happened, segmented by the same demographic groups you tested at launch.
  • Scheduled revalidationQuarterly performance and bias retest against a held out set, on the calendar rather than triggered by an incident.
  • An incident path that includes rollbackDecide in advance who can turn the system off, how clinicians are told, and what the fallback workflow is. Write it before you need it.

None of this is exotic engineering. Nearly all of it is work that is trivial while the architecture is still on a whiteboard and painful once there is a production data layer assuming otherwise. The sequence that saves the most money is also the shortest: decide whether PHI crosses the boundary, settle the device question, and make every output cite its sources. The rest of the checklist is much easier once those three are answered.

Building clinical AI that has to survive a compliance review?

Book a free 30 minute call. We will work through the data flow boundary and the device question with you, which are the two decisions that most often determine whether the rest of the build is straightforward or a rewrite.

Book a Strategy Call →