Hospital AI Committees Need Architecture-Based Privacy Reviews

The wrong starting question

Hospitals establishing AI committees to evaluate new models typically begin their privacy review with a single question: Was protected health information used to train this model? According to Peter Grantcharov, chief technology officer of Aimbient, and David Knobel, a partner at Hopp & Partners, that approach collapses critical architectural distinctions and can simultaneously under-govern high-risk systems while blocking low-risk ones.

Consider two models both trained on PHI: a large language model fine-tuned on clinical notes that accepts arbitrary prompts and generates open-ended responses, versus a computer-vision model trained to return only a blur mask or binary label. Under current review practices, both might receive identical scrutiny despite radically different exposure surfaces.

Why it matters

Undifferentiated AI governance creates false confidence around generative models while imposing unnecessary barriers on narrow classification tools that could actually reduce PHI exposure by automating redaction workflows. The result is worse patient protection, not better—committees miss the architectural features that create real risk while delaying models that would decrease the number of humans handling identifiable data.

Architecture determines exposure pathways

The fundamental issue is that "trained on PHI" describes a development process, not a deployed model's capacity to expose information. When a narrow model is fine-tuned, training data influences its mathematical parameters to improve accuracy on a defined task. That does not automatically convert the model into a searchable database of patient records.

For a closed, non-generative model with no prompt interface, no exposed weights or embeddings, and outputs limited to labels, timestamps, or bounding boxes, there may be no realistic pathway for a user to retrieve underlying patient information. The data influenced the model's weights, but that influence does not make it meaningfully accessible.

HIPAA itself reflects this practical approach. Under the Expert Determination method, information can be considered de-identified when a qualified expert determines the identification risk is "very small" in context—not mathematically impossible. Once properly de-identified, it is no longer PHI. Hospital AI governance should apply the same discipline when assessing residual exposure from deployed models.

The false equivalence problem

Generic checklists fail in both directions. For generative models, a committee may confirm that a business associate agreement exists, data is encrypted, and access is logged—yet never examine whether users can enter arbitrary prompts, query the model adaptively, or whether it retrieves information from live clinical records. These are the features that create meaningful exposure pathways.

The same checklist can over-govern narrow models. A face-detection model must learn from images containing faces; an out-of-body detector for minimally invasive video must see the identifiable frames it is designed to flag. Blocking controlled training on PHI can reduce redaction accuracy and preserve manual workflows where more people view and handle identifiable data for longer periods.

Five architectural questions

A technically serious privacy review should examine:

• What can the model accept and produce? Arbitrary clinical text and open-ended answers create different exposure surfaces than labels or coordinates.
• Can a user probe it? Promptable models supporting repeated, adaptive queries differ fundamentally from fixed-function models with no exploratory interface.
• What is actually accessible? Final outputs are not the same as access to confidence scores, embeddings, model weights, or unrestricted APIs.
• What is the credible path to a patient? Reviewers should describe the specific access, steps, and resources required to move from the deployed model to identifiable information.
• What risk is created by saying no? Blocking controlled PHI use for training may reduce de-identification accuracy or preserve workflows that create more exposure opportunities.

The final question is routinely neglected. Committees scrutinize approval risks without weighing them against clinical benefits or the additional privacy exposure from preserving manual, PHI-intensive processes.

These details were first reported by Fierce Healthcare in an industry voices piece by Grantcharov and Knobel.