A customer receives an answer they did not expect. A trade credit limit lower than last year’s, a refund request sent down a longer route, a delivery slot that somehow never seems to be offered. They contact the person they usually deal with and ask why. The answer, more often than it should be, is some version of “that’s what the system says”.
At that point the conversation tends to end. The customer either accepts it or quietly takes their business elsewhere. The person on the other end has no way to reopen the outcome and often does not know whether an AI tool played any part in it. If the outcome was wrong, nobody inside the company finds out.
This is the third part of a series on practical AI accountability for mid-sized companies. The first part sorted AI uses by what they decide; the second looked at designing the human review. This one starts from the other side: the person the outcome lands on. It adds two columns to the register, how that person finds out a system was involved, and who they can turn to if they want it looked at again.
I think this is the part companies skip most easily, because it feels like customer service rather than governance. In my view it is closer to the opposite: for some errors, it is the only way they ever become visible.
The person affected sees what nobody inside can
Inside a company, errors are caught by people who look at outputs: reviewers, analysts, whoever checks a sample. They can only judge what reaches them, and they judge it with the same information the system had.
The affected person has something they lack: they know their own facts. They know about the payment that was made, the address that changed, the large order that was a one-off. When a system gets something wrong because its data was incomplete or out of date, the person on the receiving end is frequently the first and sometimes the only one able to tell.
That matters most for outcomes that never reach a reviewer at all: the option that was not offered, the request that was routed away, the case set aside without anyone looking. Inside the company, those rarely register as decisions. To the person affected, they are the decision.
A company that gives the affected person no route back is discarding one of its best error detectors.
How they find out
The first column is about awareness. A line buried in the terms of service may technically disclose that automated tools are used, but it tells nobody anything at the moment it matters.
What tends to work is telling people at the point where the outcome reaches them, in plain words, and only for the uses that warrant it. The sort from the first part of the series does the filtering here: uses that act directly on a person need this; uses that only inform someone inside the company usually don’t.
The message does not need to explain how the system works, and it should not expose logic the company has good reason to keep private. It needs to say that a system was part of the outcome and that the person can ask for a second look. Where it helps, it can add, in broad terms, what kind of information the outcome was based on.
In the register, this column records where and how the person is told: the screen, the email, the script the account manager follows. If the honest answer is “they aren’t”, that is worth writing down too.
Who they turn to
The second column is the route. A generic contact address is not a route; it is a place where requests wait. The person needs a named way in, a role, a team, a specific option on the page and whoever receives the request needs a way to get the outcome looked at by someone who can actually change it.
The most common failure here is the loop. The request is received, passed back through the same system, and the same result comes out with an apology attached. From the outside it looks like a second look. It isn’t, because nothing about the second pass could have produced a different answer.
It also helps to decide in advance what the person will hear back, including when the outcome stands. “We checked, and this is what it was based on” is a reasonable answer. Silence is not, and it teaches people not to bother asking next time.
What the requests tell the company
Every request is information. If requests are recorded, which use, what the person said, what the second look found, patterns start to appear: one use that attracts far more requests than the rest, one kind of customer who keeps being affected, one data field that turns out to be stale. Each of those is a reason to revisit how that use was sorted, or how its review was designed.
A lack of requests deserves a careful reading. It can mean the use works well. It can also mean the first column is weak, and people simply don’t know they can ask.
My recommendation is to write two short lines for each use that acts on a person: where they are told, and where they go. If either line is hard to write, that is the finding. The next part of the series moves earlier in the chain, to a moment that usually comes before any of this: what to check before company information is uploaded to an AI provider.
Rhisco Group’s Risk, Models & AI Governance practice builds risk models, control artefacts and AI governance architecture so that deploying AI leads to positive outcomes. If you are working out how the people affected by your AI can ask for a second look, we can help: www.rhisco.com/services
This article was co-created with the assistance of artificial intelligence under strict supervision, editing, and verification of our team.