Your automation will be wrong sometimes. Design the exception path first.

A while back I reviewed an invoice processing automation at a company that was, on paper, delighted with it. The model classified 92 percent of incoming invoices correctly. Everyone had signed off on that number. It sounded excellent.

Then I asked what happened to the other 8 percent. There was a pause, and then someone said: “They go to the shared mailbox.” Nobody in the room could tell me who checked that mailbox, how often, or how long an invoice typically sat there. It turned out the answer was roughly eleven days, and that two people were quietly re-doing work the system had already half-processed.

The automation was not broken. The exception path was, because nobody had designed one.

The 8 percent is where your business case lives or dies

Here is the arithmetic almost nobody does before approving a project.

Say you process 500 invoices a month. Manually, each one takes 4 minutes, so about 33 hours a month. The automation handles 92 percent of them, which leaves 40 invoices for a human. If those 40 took the same 4 minutes each, you would still be saving over 30 hours. Great project.

But exceptions never take 4 minutes. An exception first has to be noticed, which takes time on its own. Then someone has to work out what the system did, why it was uncertain, and what the correct answer is. In practice I see exception handling take three to four times longer than the original manual step. At 15 minutes each, those 40 invoices cost 10 hours a month. Your 33-hour saving is now a 23-hour saving, which is still fine.

Now change one number. Make the accuracy 85 percent instead of 92, which is entirely normal for messier document types. That is 75 exceptions at 15 minutes, or nearly 19 hours. Add the time someone spends monitoring the queue and chasing what got stuck, and you are close to breaking even on a project you paid €18.000 for.

The success rate is the number everyone quotes. The cost per exception is the number that decides whether the project pays back.

The two kinds of wrong

Not all errors are equal, and the difference matters more than the error rate itself.

The first kind is the visible failure. The system knows it is uncertain, stops, and asks. This is the good kind. It is annoying, it costs handling time, but it is contained and you can see it.

The second kind is the silent failure. The system is confident and wrong. It books an invoice to the wrong cost centre, reads 1.450 as 1.45, matches a payment to the wrong customer. Nothing flags. Nothing stops. The number flows into your reporting and is discovered three months later by an accountant, or never.

A system with 5 percent visible failures is usually safer than a system with 1 percent silent ones. When you evaluate an automation, ask the builder not “how accurate is it” but “how does it behave when it is wrong”. If the honest answer is “it just gives an answer”, you are buying the second kind.

Five things to decide before you build

These are the questions I now put on the table during the design phase, before a line of code exists.

1. What accuracy is good enough for this specific process?

There is no universal threshold. For routing internal support tickets, 85 percent is fine, because a misrouted ticket costs someone thirty seconds to forward. For anything touching payments, customer contracts, or regulatory reporting, 85 percent is unacceptable at any price. Set the number per process, in writing, before you start.

2. How does the system say “I am not sure”?

Most models can produce a confidence score alongside their answer. Use it. Set a threshold under which the system does not decide, it escalates. This is what converts silent failures into visible ones, and it is usually a small amount of work at build time that pays for itself many times over.

3. Who owns the exception queue, by name?

Not a department. Not a shared mailbox. A person, with a named backup for holidays. If you cannot answer this question during design, the exceptions will land somewhere by accident, and that somewhere is almost always the most conscientious person on the team, who will absorb it silently until they leave.

4. How long may an exception sit there?

Give the queue a service level, even an informal one. Two working days, say. Then make ageing visible, so an item that passes the limit is obvious to someone other than the person handling it. Exception queues fail quietly, which is exactly why they need a visible clock.

5. How do exceptions make the system better?

This is the step most companies skip entirely. Every resolved exception is a labelled example of something the system got wrong. If nobody reviews them as a batch, say monthly, you keep the same error rate forever. If someone does, you usually find that half the exceptions come from two or three recurring causes, and those are often fixable in a day. That is how an 85 percent automation becomes a 94 percent one in the second quarter, without a new project.

What this looks like when it works

At one client, the exception queue was reviewed every Friday morning for twenty minutes. Within two months they had found that most exceptions came from a single supplier whose layout had changed, and from documents scanned at an angle by one particular office. Both were solved without touching the model. The queue dropped from around 60 items a week to under 15.

That twenty-minute meeting was worth more than any model tuning, and it existed only because the exception path had an owner, a clock, and a review moment from day one.

The real test of an automation

A well-designed exception path is the difference between an automation your team trusts and one they quietly work around. The second is more common than vendors admit. People keep the old spreadsheet running next to the new system, because they were burned once and cannot tell when the system is wrong. At that point you are paying for both.

So when you next see a proposal promising 90-something percent accuracy, spend most of the meeting on the rest. Ask what happens to it, who sees it, how fast, and how it feeds back. If the proposal has no answer, it is not finished.

Mapping where automation actually pays back, exception handling included, is a large part of what the AI value scan does. If you want an honest view before you commit budget, the AI value scan is a good place to start.