Once a company has sorted its uses of AI, the ones at the top of the list usually receive the same safeguard: a person reviews the output before it takes effect. It is the right instinct. It is also one of the easiest controls to declare and one of the hardest to make real.
The phrase “a human reviews it” tends to end the conversation, because it sounds like an answer. In practice it leaves the important questions open. Where in the process does that person look? Who is it, and can they say no? What do they have in front of them when they decide? When nobody answers those questions on purpose, they get answered by default, usually by whatever the screen happens to show and however much time the reviewer happens to have.
The result is a familiar pattern. The reviewer sees a finished output, has no easy way to check it, is under time pressure, and approves. People tend to trust a system’s suggestion more than they should when checking it is hard, and a review designed without that in mind turns into a signature. The control exists on paper; the decision is still being made by the system.
This is the second piece in a practical series on AI accountability for mid-sized companies. The first covered how to sort AI uses by what they decide. This one takes the uses at the top of that list and looks at how to design the point where a person reviews what the system produced. In my view, three design choices do most of the work: where the review sits, who does it, and what they see.
Where the review sits
A review only protects anyone if it happens while the outcome can still be changed. That sounds obvious, but it is common for the review to sit at the point where it is convenient rather than where it matters. A weekly report of what the system did is useful for learning. It is not a review of a decision that already reached a customer on Monday.
The practical test is to follow one output from the moment it is produced to the moment it affects someone and mark the last point where a person could still stop it. That is where the review belongs. For a tool that suggests discounts for key accounts, it is before the quote leaves, not when the margin report arrives at month end. For a feature that drafts letters declining a claim, it is before the letter is sent, not when the customer complains.
There is a second, less obvious placement problem. Some systems only show the reviewer what they let through. A screening feature that filters applications presents the shortlist for approval; the applications it discarded never reach anyone. Reviewing the shortlist feels like oversight, but the decisions most worth checking are the ones that were never shown. For uses like these, the review must include a look at what was excluded, even if only a small selection of it.
Who reviews, and whether they can say no
The reviewer should be a named role, not “the team” or “whoever is available”. Shared responsibility for a review usually means that each person assumes someone else looked closely. A name, or at least a specific position, makes it clear who is answerable for the approval.
Naming someone is not enough on its own. The person needs three things for the review to mean anything. They need enough knowledge of the work to recognise a wrong answer, which usually means someone who does that work, not someone who supervises the tool. They need the time to do it, which means the review is counted as part of their job rather than squeezed between other tasks. And they need the authority to reject or change the output without asking permission and without having to justify themselves more than they would for an approval.
That last point is where many reviews quietly fail. If rejecting an output means escalating, filling in a form or explaining a delay to someone senior, while approving takes one click, the design is telling the reviewer what to do. I would also be cautious about making the reviewer someone whose own targets depend on the output going through. That does not make them dishonest; it makes the review harder for them than it needs to be.
What they see before approving
A reviewer who sees only the final output can check whether it looks reasonable. They cannot check whether it is right. To do that, they need at least some of what produced it: the information the system used, the main factors behind the result when the tool can provide them, and enough context about the case to notice when something does not fit.
Consider a proposed price for a named customer. Shown alone, it is a number that is either plausible or not. Shown next to the customer’s recent history and the main reasons the system gave for it, it becomes something a person can agree or disagree with. The second version takes slightly longer to review. It is also the only one that makes the review worth the time.
What the reviewer sees should include a simple way to disagree and to record why. If the only button is “approve”, the design has already answered the question. A short field for the reason behind a rejection or a change costs little, and over time it becomes the most useful record the company has about where the system gets things wrong.
How to tell the review has stopped working
Even a well-designed review can drift into a formality. The signs are usually visible without any measurement system: reviewers who cannot say what they check, approvals that take a few seconds regardless of the case, and long periods without a single rejection or change. None of these proves the review has failed, but each one is a reason to sit with the reviewer and look at a few recent cases together.
My recommendation is to write the review down, use by use, in three lines: where it sits, who does it, and what that person sees before approving. Those are the three columns this piece adds to the register the series is building, next to the two answers from the sorting exercise. If a use at the top of the list cannot fill those three lines, it has a review in name only. The next piece looks at the other side of the same decision: how a person affected by the output finds out, and who they can turn to when they want it looked at again.
Rhisco designs risk models, control artefacts and AI governance architecture to ensure positive outcomes when companies deploy AI. If the uses at the top of your list have a review nobody designed, let’s talk: www.rhisco.com/services
This article was co-created with the assistance of artificial intelligence under strict supervision, editing, and verification of our team.