How I Validate AI Clinical Tools for FDA Compliance

I have watched intelligent teams make the same mistake over and over: they test whether the model sounds good, then assume they have validated the product. That is not validation. That is a weekend demo. The moment an AI clinical tool touches triage, documentation, coding, message routing, or a clinician-facing recommendation, the question changes. You are no longer judging a model in isolation. You are judging a clinical system that includes prompts, UI, integrations, fallback behavior, human review, audit logging, and post-release change control.

When buyers call us about Medexa or any other clinical AI workflow, I tell them the same thing: start with the decision the system influences, then work backward to the evidence you need. If you get the intended use wrong, everything else becomes theater. We have seen teams spend weeks building beautiful test packs for features that were never the compliance risk. We have also seen the opposite: teams ship a narrow, low-risk assistant and then quietly expand it into decision support without re-running their validation stack. That is how you end up in trouble.

For compliance-focused buyers, the core mistake is believing FDA readiness is mostly about the algorithm. It is not. The algorithm matters, but the regulatory question is about the total system: intended use, risk classification, human factors, cybersecurity, transparency, and operations after launch. A very ordinary model can become a regulatory headache if it is wired into a workflow in a way that nudges clinical behavior without guardrails. I have seen a modest rules-plus-LLM assistant create more risk than a more sophisticated model, simply because the lightweight tool lacked obvious provenance, versioning, and supervision.

What I validate first at AST

At AST, we do not start with benchmark scores. We start with the claim the product makes. Then we test whether the software can prove that claim under realistic operating conditions. That sounds obvious until you look at most vendor decks. They jump straight to accuracy charts and call it a day. I do not care how impressive the model looks if I cannot trace the output back to the input, the version, the rule set, and the reviewer who accepted it.

That is why I structure validation around four questions:

• What is the intended use? Is this documentation support, coding support, triage support, or clinical decision support? The answer changes the risk profile.
• What claims does the vendor actually make? Marketing language is not harmless. If the tool suggests diagnoses, orders, or risk stratification, I assume higher scrutiny.
• How does the human stay in the loop? I want to know whether the clinician reviews, approves, edits, or overrides every output before it affects care.
• How will the tool change over time? If the model, prompt logic, payer rules, or inference service can shift without a release gate, validation evaporates fast.

That last point is where teams get surprised. The first system I helped validate looked solid until we discovered that a vendor-side prompt tweak changed output behavior with no formal release note. The product team called it a minor optimization. The compliance team called it an uncontrolled change. They were both right in their own language, and only one language matters when you are under audit.

FDA validation is not one document

If you think validation means producing one thick PDF and filing it away, you are already behind. Real validation is a chain of artifacts. Each one answers a different question, and they have to line up.

1. Define intended use and boundaries Write the exact task the AI performs, who uses it, what system it sits inside, and what it does not do. Do not leave room for creative interpretation.
2. Map risk by workflow, not by feature A summarization feature can be low risk in one context and high risk in another. Risk comes from decision impact, not from the label on the button.
3. Build traceability from requirement to test Every requirement should map to one or more tests, and every test should map back to a requirement. If you cannot trace it, you cannot defend it.
4. Test with real workflow edge cases Include missing data, contradictory inputs, partial histories, copy-forward notes, interrupted sessions, and degraded integrations. Happy-path testing is not clinical validation.
5. Lock change control Document what triggers re-validation: model updates, prompt changes, rule engine edits, new payer policies, new integrations, and major UI changes.
6. Define human review and fallback A compliant system must say what happens when the model is uncertain, when a source feed is missing, or when the clinician rejects the suggestion.

That sequence matters because it prevents a very common failure mode: teams test too late. If you wait until after implementation to decide what evidence you need, you usually end up retrofitting validation around whatever was easiest to build. That is backwards. I have seen teams build stunning user experiences that were impossible to validate cleanly because the product had no stable boundary. The software kept “helping” in ways nobody could measure.

At AST, we build validation around the operational truth of the system. For AI documentation and claims workflows, that means every generated element needs provenance. If a note summary, code suggestion, or prior auth draft is produced, I want to know what source context was available, what deterministic rule fired, and what human accepted it. That is the same mindset we use in our clinical AI work: the workflow has to survive scrutiny, not just admiration.

A practical validation checklist for buyers

If I were sitting on your procurement or compliance review this week, this is the checklist I would use before I signed anything. It is intentionally blunt.

• Intended use is written in plain language. If the description sounds like marketing copy, push back.
• Clinical responsibility is explicit. The tool should state what clinicians must review manually.
• Dataset provenance is documented. You need to know what data trained, tuned, or evaluated the system.
• Test cases include your edge conditions. Specialty, payer mix, note style, and local workflow matter.
• Audit logs are exportable. You should be able to reconstruct who saw what, when, and why.
• Model and rule versions are pinned. Validation without version control is not validation.
• Cybersecurity is part of release sign-off. AI features still live inside real infrastructure.
• Rollback is defined. If the system misbehaves, there must be a documented way to disable or revert it quickly.

There is one more thing I insist on: test the negative space. Most vendors love to show what works. I care about what fails. What happens when the encounter note is incomplete? What happens when the patient says one thing and the EMR says another? What happens when a payer rule changes and the prior auth draft is now stale? If a product only works when the inputs are clean and the context is perfect, it is not clinically ready.

The table is simple because the decision should be simple. If a vendor cannot answer those four rows cleanly, I do not care how modern the UI looks. Pretty software is cheap. Controlled software is hard.

How I would evaluate a vendor in one week

You do not need a six-month scientific program to spot serious problems. You do need a disciplined review. Here is the sequence I recommend when the buying committee wants to move quickly without becoming reckless.

1. Ask for the exact intended use statement Do not accept a vague product label. Get the sentence the vendor wants in the contract or risk file.
2. Request the validation package You want test scope, acceptance criteria, versioning, and failure handling. If the package is thin, that tells you something.
3. Interview the implementation team Ask how the tool is deployed, updated, monitored, and rolled back in production.
4. Run your own edge-case review Use real local scenarios: specialty notes, unusual abbreviations, missing data, and workflow interruptions.
5. Check governance ownership Someone on your side must own ongoing review after go-live. If nobody owns it, the risk spreads everywhere.

That sequence also exposes a subtle problem: some vendors have a strong lab story but a weak production story. Their offline testing looks professional, but their deployment model is fuzzy. That gap matters because clinical AI fails in production, not in slides. The system has to hold up when nurses multitask, physicians edit under time pressure, and integration feeds arrive late.

One reason we built Medexa the way we did is that the approval model is explicit. The model can draft, surface, and organize, but human review is always part of the path before anything reaches a payer. That same discipline applies on the clinical side. If a tool nudges care or documentation, your validation should prove who owns the final decision and how that decision is auditable.

What regulators and auditors actually care about

I do not pretend that every AI clinical tool will be reviewed in the same way. The regulated universe is broader and messier than that. But the themes that come up again and again are predictable: intended use, risk management, traceability, transparency, and change control. If you can show those cleanly, you are in a much better place than the teams that try to win with vague claims about model intelligence.

Auditors also notice operational sloppiness. If your policy says reviewers approve every output but your logs show unattended changes, that mismatch is a problem. If your validation packet says the model is pinned but the production vendor can silently modify prompts, that mismatch is a problem. If your SOP says clinicians can override suggestions but the UI buries the override behind extra clicks, that mismatch is a problem. Compliance breaks when declared process and real process diverge.

That is the part people underestimate. A lot of compliance work is not about inventing controls. It is about making sure the control you claim actually exists in the product. I have spent enough time in real deployments to know that the ugly edge case is usually the truth. The pretty workflow diagram is what failed to show up on the floor.

My bottom line is simple: do not buy AI clinical tools on promise. Buy them on evidence, boundaries, and control. If a vendor can show me how the product behaves under stress, how it records its own decisions, and how it changes without breaking the compliance story, I pay attention. If they cannot, I move on fast.