AI in clinical software: showing uncertainty and keeping the clinician in the loop

30 September 2026

Patient portal & EHR Trust, Ethics & Safety Clinical vs Patient Interfaces

The FDA's clinical decision support guidance gives designers a simple test: can a clinician review the basis of an AI output and still decide for themselves? This article turns that and other stable sources into screen-level questions.

Not clinical, regulatory or legal advice This article is design education. It describes what published guidance asks of AI outputs on a screen and what a designer might ask. It is not clinical advice, not a compliance verdict on any product, and not a view on whether any tool should be used. Guidance changes; check current versions.

The FDA's Clinical Decision Support Software guidance, issued on 29 January 2026, describes software that lets a clinician independently review the basis of a recommendation, so that the clinician does not rely mainly on it. That idea is a useful design test for any screen that shows an AI or machine-learning output, whether or not the product is a regulated device.

The guidance is non-binding, and this article does not say what any product must do. It borrows the questions. What is this output based on? How sure is it? Can a person check it and disagree? Those questions sit inside Trust, Ethics & Safety and apply to both clinician and patient screens.

Show what the output rests on

For clinician-facing software, the FDA guidance lists what helps a clinician review a recommendation. It names the intended use, user and patient population. It names the inputs and how they were obtained. It asks for a plain language description of how the method was developed and validated, and of the data it relied on, so a clinician can judge whether it resembles their own patients. It also asks the output to show relevant patient-specific information and anything missing, corrupted or unexpected in the inputs.

Each of these is a design task. A recommendation with no visible inputs is hard to check. An output that hides a missing lab value looks as confident as one that has every value. The same guidance says this information should be prioritised and layered, so the most decision-relevant part comes first and detail is available on request, to avoid overload.

Say how sure it is, and where it stops working

The joint FDA, Health Canada and MHRA document "Transparency for machine learning-enabled medical devices: guiding principles" (updated 13 June 2024) frames transparency as who needs information, why, what, where, when and how. Its list of relevant information includes intended use, workflow fit, performance, known limitations, bias and data gaps. It says information should be placed in the software interface itself, and that targeted information can appear at high-risk steps in a workflow.

Its companion, the ten Good Machine Learning Practice guiding principles (October 2021), makes two points a designer can use. Principle 7 puts the focus on the performance of the human and AI team, not the model alone. Principle 9 says users should get clear, essential information: intended use, performance across subgroups, limitations and how to report a concern.

Show a range or a plain-language confidence statement, not a bare score. The Humanising the number card shows why a number needs context.
Show the limits next to the output, for example the groups the tool was not tested on, not only in a manual.
Show when inputs are missing or old. A silent gap reads as confidence.
Say plainly whether the output is a suggestion, a flag or a decision, and who made it.

Guard against automation bias

The FDA guidance defines automation bias as a tendency to over-rely on a suggestion from an automated system, leading either to following wrong advice or to not acting because nothing prompted action. It adds that the risk rises when a decision is urgent. It does not treat software meant for critical, time-sensitive decisions as one a clinician has time to review.

Goddard, Roudsari and Wyatt reviewed the evidence in the Journal of the American Medical Informatics Association. They screened 13,821 papers, kept 74 and concluded that the effect is consistent across fields. Pooling four healthcare studies, they found erroneous advice was more likely to be followed in groups with decision support than in control groups, with a risk ratio of 1.26. The paper carries a 2011 publication year and predates today's AI tools, so treat it as the basis for a question, not a measure of any current product.

The same review reports design factors that shifted reliance. Prominent wrong advice was more likely to be followed. Showing a confidence level that updates with each piece of advice improved appropriate reliance, compared with one fixed level for the whole system. Making the reasoning visible increased appropriate reliance. Offering supportive information rather than commands reduced the bias.

Check that an AI output is not the largest or first thing on a screen a clinician must verify.
Show the reasoning or the supporting data before, or beside, the suggestion.
Avoid wording that reads as an instruction. Prefer "suggested" or "flagged for review".
Watch alert volume: a stream of low-value prompts wears down attention. See the Alert fatigue card.

Keep a person able to check and override

Checking is only possible if the path is short. Show the source record beside the output. Let the clinician change the value and see what changes. Record an override without a hurdle, and never make the AI choice a default that takes extra steps to reverse. In the Medication reconciliation card the person resolves conflicts between sources; the same principle applies when software proposes the resolution.

When a recommendation is time-critical, a review-the-basis screen may not be realistic. For a designer, that means asking early: how much time does the person have, and does the interface assume more?

Say what the tool is, and is not

Patient-facing screens need the same honesty in simpler words. State that an output is generated by software, what it used, and that it is not a diagnosis. Do not use a clinical tone that implies a person reviewed it if none did. Keep the route to a human visible. The WHO's 2021 guidance "Ethics and governance of artificial intelligence for health" lists six principles, among them protecting human autonomy and ensuring transparency, explainability and intelligibility. The WHO's news summary says the information should be documented before design or deployment and easy to access. These principles concern governance, not screens, but they give the reason behind each choice above.

For the wider trade-offs between clinician and patient views, see Clinical vs patient interfaces and RPM UX patterns.

Questions to take to a design review

What is this output based on, and can the person see it in two taps?
What does it not know, and where is that stated on the screen?
How does a person disagree, and how many steps does it take?
Is the output presented as a suggestion or as an answer?
Was the screen tested with the people who will use it, under realistic time pressure?