The self-check
Six weeks after you deployed it, can you answer these four questions?
Not a maturity model and not a certification. Four questions about a system you already have in production. They take about ten minutes to answer honestly and they are hard on purpose — if they were easy, nobody would need to ask them.
Nothing here is collected. There is no form, no score emailed to you, and no report gated behind an address. Read the four, answer them on paper, and keep the answers.
The four
- 01
What did it actually do last week — step by step, not by design?
A real answerA trace you can read: the inputs it received, the decisions it took, the actions it performed, in order, for a real period. Reconstructed from logs, not from the architecture diagram.
Not an answer"It classifies tickets and routes them." That is what it was built to do. The question is what it did.
- 02
How often was it wrong — against what task set, defined by whom?
A real answerA number with a denominator you can inspect, measured against a task set someone specific chose, with the choosing written down. Ideally the set includes cases you expected it to fail.
Not an answerA vendor benchmark, or an accuracy figure whose evaluation set you have never seen. Both measure a different system than yours.
- 03
What would have happened without it?
A real answerA comparison. A held-back group, a period before deployment measured the same way, or a sample re-done by hand. Anything that answers "compared to what?"
Not an answerTime saved, tickets closed, or adoption. Those are measurements of the system running, not of the system helping. This is the one almost nobody has.
- 04
How would you know if it degraded?
A real answerA named signal, a threshold, and a person who sees it. Something that fires without anyone remembering to look.
Not an answer"We would notice." Silence is not health — a system that quietly got worse produces no complaints until the damage is downstream and expensive.
How to read your answers
Count only the ones you could show someone.
Not the ones you could argue for — the ones where you could open a document, a dashboard or a log and let a sceptical person check. Answering in your head does not count, and that restriction is the entire exercise.
Which is extremely common and not a moral failing. It does mean nobody can currently tell whether the system is helping, including you.
You know what it does and roughly how often it errs. The gap is almost always question three — the counterfactual — and it is the one that decides whether the thing is worth keeping.
Genuinely rare. If you can show all four, you can defend the system to a board, a regulator or a sceptic — and you should be writing about it, not reading this.
Why asking the vendor does not close the gap
They can answer two of these. Never the last two.
A supplier can tell you what the system was built to do and how it performed on their evaluation set. They cannot tell you what would have happened in your organisation without it, and they cannot tell you when it has drifted on your data — those answers only exist where the system actually runs. This is not a claim about vendor honesty. It is a claim about where the evidence lives.
If the answer is "no" to most of them
That is the gap, not a failure.
Every one of these questions is answerable. None requires a platform, a certification or a new budget line — they require deciding that "we think it is working" is not the same as knowing, and then doing the boring measurement that closes the difference. That measurement is what this laboratory publishes, method first, including the parts that fail.