← Yoonbo Cho

HIPAA and LLMs in late 2026: what a BAA doesn't cover

Nov 2, 2026

An engineer's reading of where HIPAA stands for LLM systems in November 2026. Not legal advice.

The sentence I hear most often from teams building health AI is "the vendor signed a BAA, so we're fine." I believed something close to it before I built Arbiter, my clinical-document RAG prototype (synthetic records only). Reading the vendor terms properly changed how I'd design the whole thing.

Here is what I found, with sources. Regulations and vendor pages change often, so check the dates.

There is still no AI-specific HIPAA rule

HIPAA doesn't have a chapter on models. The same Privacy and Security Rules apply to a prompt as to a database row. What has changed is the pressure around them.

HHS proposed a major Security Rule update in January 2025: a written technology asset inventory and network map, encryption of ePHI at rest and in transit, MFA, vulnerability scans every six months, and a yearly penetration test. As of now it is still a proposal. The regulatory agenda reportedly pushed the final rule to 2027, and a large coalition of provider groups has asked HHS to withdraw it. I'd still build to it, because the enforcement trend points the same way. OCR's risk analysis initiative kept producing settlements through 2026, including $700,000 from Ambry Genetics in September after a phishing attack.

A BAA covers features, not companies

This was the part that surprised me. Every major provider that signs a BAA limits it to specific products and configurations.

So the real question isn't "do they sign a BAA". It's "is every call my code makes going to a covered feature". A batch job that someone adds later to save money can quietly fall outside the agreement. In Arbiter I'd treat the list of covered endpoints as configuration the code checks, the same way it refuses to call a model before a document is de-identified.

Zero data retention is not a BAA

Zero data retention (ZDR) means the provider doesn't keep your prompts. It doesn't make the provider your business associate, and it isn't available for every model. Anthropic notes that some models need 30-day retention and can't run under ZDR. You need both, and they're set in different places.

I ran into the same gap from the other side in Everblue, my journaling app. Whether a provider retains prompts is account configuration, not code, so the code has a flag for ZDR routing that stays off until someone verifies it in the dashboard. The app's privacy wording reads that flag. If you can't point to the setting, don't make the claim.

You probably have more ePHI than you think

Teams protect the documents and forget what the pipeline creates from them:

I'd treat all of these as ePHI. They belong in the risk analysis and the asset inventory, and a missing or shallow risk analysis is the most common theme in OCR settlements. In Arbiter the map back to real identifiers is encrypted per tenant and never enters the vector index, so the index holds only scrubbed text.

Removing the 18 identifiers is half of Safe Harbor

Safe Harbor has two conditions. You remove the 18 listed identifiers, and you have no actual knowledge that what's left could identify the person. Free-text clinical notes make the second condition harder than it sounds. A rare diagnosis plus a job title plus a town can identify someone without any of the 18 present.

A February 2026 preprint from NYU (not yet peer reviewed) ran a linkage attack on about 223,000 scrubbed notes and re-identified a small share of patients uniquely, up to 0.34%. That sounds small until you multiply it by a hospital's volume. For notes, Expert Determination, where a qualified expert certifies the risk is very small, is often the better route.

My own testing showed the first condition is hard too. Off-the-shelf detection missed ZIP codes in 14 of 20 synthetic documents while my recall metric read 0.967, because it matched whole strings.

HIPAA isn't the only rule that applies

Even where HIPAA doesn't reach, state law might:

And under Section 1557, covered entities have to make reasonable efforts to find decision-support tools that use race, sex, age or disability as inputs and reduce the risk of discrimination.

Takeaways

  1. Treat a BAA as a list of covered features. Make your code check that every model call goes to one.
  2. Get the BAA and zero data retention separately, and check that your model supports ZDR at all.
  3. Inventory derived data: embeddings, logs, caches, indexes and eval sets.
  4. For free-text notes, plan for Expert Determination, or at least test for re-identification beyond the 18 identifiers.
  5. Check state law even when HIPAA doesn't apply.

Sources: HHS de-identification guidance, Security Rule NPRM, Ambry Genetics settlement, Anthropic BAA scope, OpenAI BAA guide, AWS HIPAA-eligible services, Google Cloud HIPAA, NYU preprint, Section 1557 final rule.

I'm looking for an AI engineering role (remote, US). If your team puts health data near LLMs, I'd love to talk.