HIPAA in the Age of AI: What Healthcare & Health Tech Companies Need to Know

At a Glance

• • • AI doesn’t create an exception to HIPAA. The regulation still focuses on who has the personal data, where the data is located, and the reason it’s being used.
    • While BAAs are often needed, a BAA doesn’t address training, retention, secondary use, model-provider, or product-improvement questions.
    • The modern HIPAA risk is the full AI stack. There needs to be visibility into embedded AI features, upstream model providers, downstream subprocessors, and exactly what AI systems can do.

A product is about to go live. Or it’s getting close. Security review is underway. Everyone knows the business team wants to move quickly. In some cases, it isn’t a new AI product being licensed, it’s software the company is already using that has launched a new AI feature.

Suddenly, a host of different things can happen to patient data when AI is introduced into the stack. Can the vendor now use protected health information to improve and train its model? Does the related business associate agreement reflect how the tool actually works?

AI assistants are already a feature in almost all patient portals. Even on the back end, AI assistance is used to create documentation, for scheduling, coding, intake, and systems that act across an array of tools.

So, it’s a  seemingly simple question to ask: What exactly is the AI tool allowed to do with the patient data?

In July 2026, the U.S. Department of Health and Human Services updated its business associate guidance. It specifically says that a third-party AI chatbot on a provider’s patient portal is a clear example of a business associate when it is providing services involving patient PHI. 

This tells us that the expectations of HIPAA and other regulatory bodies are catching up to technology that’s implemented and already collecting data.

When is HIPAA Actually Triggered?

The HIPAA Privacy Rule protects certain health data when a regulated entity holds or transmits it. In short, HIPAA turns on three things: the entity’s role, the type of information involved, and why it’s used or disclosed.

Covered entities. HIPAA applies to health plans, healthcare clearinghouses, and healthcare providers that transfer patient health data electronically, involving certain specified transactions.

Business associates. A company is likely to be subject to HIPAA if it performs certain functions or services for a covered entity. Specifically if it creates, receives, maintains, or transmits PHI for that company. Subcontractors that handle PHI for a business associate can also be considered business associates.

PHI. Protected Health Information is health information that is both individually identifiable and held or transmitted by a covered entity or business associate. It can be electronic, paper, or verbal. The Security Rule, however, places specific focus on electronic PHI, or ePHI.

The HIPAA boundary. HIPAA doesn’t cover every health tech company, app, or piece of health data.  AI is making this boundary particularly significant. We’ll come back to that.

AI is Already in the Stack. Can you Tell Where?

I’m sure legal teams have noticed that the client agreement review with AI provisions that turns out to be the hardest sometimes starts with a vendor that was approved years ago.

A business may have completed procurement and security review steps, and finalized contracts long before the vendor added a copilot, generative search function, automated summary creator, a new model provider, or an AI agent. The commercial relationship already looks familiar. The data flow probably isn’t.

Before asking whether an AI feature is “HIPAA compliant,” here are four more useful questions to ask:

1. What PHI or ePHI is accessible to the system?
2. Where does that data go after it leaves your environment?
3. Which vendors, sub-processors, or human reviewers can access the data?
4. What logs, embeddings, outputs, or other artifacts are still available once the transaction between the parties is finished?

For HIPAA, what truly matters is the data flow, not what the product is called or marketed for.

You Need the BAA. But it’s not the entire answer.

When the  BAA gets signed, everyone thinks the HIPAA issue is covered. That’s a false sense of security.

A BAA does require controls and defines permitted uses of PHI. But HHS (the U.S. Department of Health and Human Services) is clear on its limits. A BAA generally can’t let a business associate use or disclose PHI in ways that would break the Privacy Rule if the covered entity did it directly. And a BAA often can’t answer the commercial and technical questions that matter most when you deploy AI.

We still need to know:

• • Does the vendor use prompts, outputs, or PHI to improve the model?
  • Can the vendor train on data for safety or quality assurance?
  • Can information be combined across customers?
  • Which model providers and sub-processors drive the service?

That’s why AI and technology contracts and data processing agreements should always be read with the BAA, product documentation, security terms, and actual architecture.

It Isn’t Enough To Just Say “We Don’t Use Your Data to Train”

It sounds reassuring. It also leaves a lot of unanswered questions.

When a vendor says it doesn’t use customer data to train, ask what is excluded from that “training”, and what it still allows itself to do.

Sometimes the vendor might use  the data only to generate a response in that session. Other times it tunes or configures customer-specific models, or keeps prompts and outputs to evaluate the model, catch abuse, or run quality checks. In some cases, the data feeds a model or service shared across many customers. And occasionally the vendor de-identifies, aggregates, or derives data for its own product development.

Many different uses. But they don’t all create the same legal or commercial risk. A business that’s evaluating AI training rights should isolate each use, and not rely on a single promise just because it sounds good in a sales deck.

It’s less about whether a third party actually uses the data to train. The key question is, “What are you allowed to do with our data, for whose benefit, and under what controls?”

Follow the Route that the PHI Takes Through the Full AI Stack

The company that you’ve entered a contract with may be just one layer in a bigger system.

An AI product usually relies on cloud hosts, foundation-model providers, vector databases, logging or observability tools, and human review. If any of those parties receive PHI, the HIPAA analysis doesn’t stop at the primary vendor. It follows the data downstream.

HHS’s business associate guidance is very clear on this point. If you’re a business associate, you need BAAs with your subcontractors before you disclose any PHI to them. And the disclosure has to be only for work you’re doing for a covered entity.

See how vendor mapping becomes more than a procurement exercise? The map is a key part of understanding where the PHI actually goes.

The Office for Civil Rights made exactly the same operational point in an enforcement action of July this year. It told regulated entities to pinpoint where ePHI is located, and how it enters, flows through, and leaves the entity’s systems. The settlement doesn’t just highlight that you should know the data flow. It also reinforced that an accurate and thorough risk analysis is a requirement.

When it comes to AI, that’s your data map. And it should include the systems behind the system too.

More Context Doesn’t Mean More PHI

As a general rule, AI systems often perform better with more context. However, HIPAA doesn’t give them an unlimited right to receive it.

The “Minimum Necessary” Rule asks for something straightforward. It requires covered entities to take steps to limit how they use, disclose, and request PHI to what they need for the job. There are exceptions, notably certain treatment-related uses and disclosures.

If an appointment tool needs information for scheduling purposes, does it really also need the clinical note?

If a coding tool needs a specific part of the record, should it have ongoing access to everything else?

If an AI feature can work within structured fields, do you need to send an entire narrative account?

The usual product question is, “What can we provide to the model?” The better governance question is, “What does this use case actually need?”

The Modern HIPAA Problem.

HIPAA recognizes two ways to de-identify PHI: Safe Harbor and Expert Determination. Safe Harbor says that specific identifiers need to be removed and that there is no knowledge that the remaining information could identify the person. Expert Determination necessitates a qualified expert to determine the risk of identification to be very small and to document the analysis.

This is huge for AI because the most valuable inputs are usually unstructured. Think about clinical notes, patient messages, call transcripts, intake forms, and voice recordings. They can all contain identifying details even after a name is redacted.

Just because a vendor uses  words like “anonymous,” “aggregated,” or “de-identified” in their commercial contract, it doesn’t mean that this should automatically be treated as the HIPAA de-identification standard.

Ask which standard applies, when de-identification happens, and whether raw PHI reaches the AI system before that step occurs.

AI Agents Raise the Stakes

The next phase of healthcare AI is technology that can instantly pull information, connect other tools, update records, generate and start workflows, and perform across multiple platforms.

That makes the HIPAA conversation quite different.

From asking what this model can see, we need to move to asking, “What can this system access, combine, retain, and do?”

For an agentic system, look at:

• Which systems it can reach and what permissions it has.
• Whether it can write back to a system or only read from it.
• Which actions require human approval.
• What activity is logged and monitored.
• Whether credentials and access are limited to the task the system actually performs.

That approach fits neatly with the HIPAA Security Rule’s focus on access controls, safeguards, and ongoing risk analysis. It also fits the broader risk-management approach that NIST sets out in its Generative AI Risk Management Profile.

Your HIPAA Risk Analysis Can’t Stay Frozen

If you did your risk analysis before a major AI deployment, it might not correctly describe the environment you now operate in.

HHS risk analysis guidance considers risk analysis an ongoing process. It notes that organizations should specifically reassess risk when new technology is introduced, or when business operations change.

Now, this is especially important.

HHS has also proposed major updates to the HIPAA Security Rule. The goal of the updates is to make the security requirements more specific, including regular review, testing, and updating.

Don’t build AI governance around what you think a future rule might require. Your AI governance should be built around a data and access model you can actually explain.

HIPAA Isn’t the Whole Health-Data Analysis

We see this catch health tech companies all the time. HIPAA isn’t a catch-all federal privacy law that applies to every company that collects health data.

A B2C app that handles sensitive health information may not be subject to HIPAA if that app isn’t a covered entity or business associate. HHS guidance on health apps and APIs notes that if an individual sends health data to an app that isn’t a covered entity or a business associate, that data may no longer be protected by HIPAA.

That doesn’t give businesses a zone that’s privacy-free. The FTC’s Health Breach Notification Rule applies to certain health and health-related apps and technologies that aren’t covered by HIPAA. State consumer health and privacy laws add another layer. That’s precisely why state privacy and AI laws should be discussed together when a product sits outside traditional HIPAA coverage.

Q             

 “What health data do we have, what’s the data flow, and what are we doing with that data?”

The Contract Review Mistake We See Most Often

In any organization, different teams are working from different viewpoints.

Legal reads the BAA, while security reviews the architecture. The product team understands the AI features. Procurement negotiates the vendor terms. Privacy asks about the notice and data map.

In our work with healthcare and technology companies, we’ve found that when we read the BAA, master agreement, DPA, AI terms, security exhibit, and product configuration as one system, all teams see the whole picture and everything moves faster. A contract review for a healthcare enterprise isn’t very meaningful if the technology operates differently from the contract.

What Should Healthcare Companies Do Now?

1. Detail the specific features of the AI tools you use. Include embedded vendor features, model providers, copilots, agents, and added features. Your AI tracking list has to identify more than tools purchased as AI.
2. Map the data flow within the platforms. Detail the PHI that enters each system, where it goes, what comes back, and what gets retained.
3. Check on the HIPAA role of your vendors and subprocessors. This is where you list which vendors act as a business associate or subcontractor, and which ones don’t trigger HIPAA compliance at all.
4. Confirm legal looked at the agreements as a whole. That means the BAA, master agreement, data processing agreement, AI terms, privacy terms, security exhibit, and subprocessor commitments as one set, not stand-alone documents.
5. Separate the use case from secondary uses. Have a repository outlining which use cases are part of model training, product improvement, de-identification, retention, or use across different customers, and which are internal.
6. Determine what AI agents can access and do. For agentic systems, outline what the AI can reach, change, send, or trigger, and where a human in the loop is needed.
7. Create a review process for evolving AI features. A new model, retention practice, or new agent capability needs a fresh review. Legal should be brought in to see if updates to the tool change the risk profile.

How to Be in the Strongest Position

AI is now part of the healthcare industry’s infrastructure. The companies in the strongest position will be the ones that know whether the contracts and governance match what the technology does.

Gouchev Law advises healthcare, digital health, technology, and AI companies on the agreements and governance behind AI in regulated healthcare environments. We represent both customers adopting AI and providers bringing AI-enabled products to market. From BAAs and technology vendor agreements to data privacy and internal governance. Reach out to our technology lawyers who focus on the intersection of technology and healthcare.

This article is for general informational purposes and doesn’t constitute legal advice or create an attorney-client relationship.

About the Author

More Resources For You