A Clean Result Is Not the Same as a Clean System

An audit can complete, report nothing, and have examined nothing.

Imagine a report that comes back clean.

Every stage of the process ran. Nothing errored. The summary says zero issues found. And underneath, the internal accounting reconciles perfectly: the number of items examined matches the number recorded, which matches the number reported. The arithmetic is flawless.

Now suppose the scan examined no files at all.

Both of those things can be true at the same time, of the same report. I want to show you exactly how, because the mechanism is simple enough to follow without any technical background, and once you have seen it you will read clean reports differently for the rest of your career.

Why a clean report is so persuasive

A clean result arrives with two things attached that make it feel settled.

The first is completion. Nothing crashed, nothing timed out, no stage reported an error. Completion feels like evidence, and it is evidence of something, but what it is evidence of is that the process finished. That is not the same as the process having looked.

The second is internal consistency. A serious assurance tool checks itself. It counts what it examined, counts what it recorded, counts what it reported, and confirms the three numbers agree. That kind of double-entry check exists precisely to catch findings being lost between stages, and it is a good control. It is also the part I want you to be most careful about.

The mechanism

Every scan works from a list: the set of files that are in scope for this examination. Findings are checked against that list, and anything falling outside it is discarded as out of scope. That is ordinary and correct behaviour. You do not want a report full of findings about files nobody asked you to look at.

Now consider what happens if that list arrives empty.

The check does not ask “was this file excluded?” It asks “is this file in the examined set?” Against an empty set, the answer is no. For every file. So every finding is out of scope, and every finding is discarded.

The scan does not fail. It has nothing to fail on. Each stage runs, processes what it was given, discards everything on a rule that is working exactly as designed, and finishes. The report says zero.

Passing no list does not mean examine everything. It means discard everything. And the difference between those two readings is invisible in the output.

The arithmetic that would not have saved anyone

Here is where the self-check comes in, and here is why I said to be careful about it.

The double-entry control compares three numbers: examined, recorded, reported. In this scenario all three are zero. Zero equals zero equals zero. The check passes. It passes honestly. It is doing precisely the job it was built to do, which is to detect findings going missing between stages, and no findings went missing between stages, because there were never any findings.

That is the part worth pausing on. The control was not broken, bypassed, or misconfigured. It was working, and it was silent, because a balanced set of books tells you the bookkeeping is sound and tells you nothing whatsoever about whether anyone did any work.

An empty result is consistent with everything.

How it was actually caught, and it was not by a test

I want to be exact about this, because the honest version of this story is both weaker and better than the version I could tell.

This never happened. No audit of ours passed on nothing. No report of this kind was ever produced, sent, or relied on.

It was found before any of that, by a person reading the code in advance of running it, who stopped on the line that builds the scope list and asked what the next line would do if that list were empty. Then they traced it: empty set, every file out of scope, every finding discarded, zero reported, books balanced. Then they stopped and reported it rather than proceeding.

A guard was added, and it is a small one. Before the examination begins, confirm that the list of files is not empty, and that the set of paths to be scanned is not empty. If either is empty, stop. It is perhaps two lines of logic, and it exists only because someone declined to assume.

There is a detail here I find genuinely uncomfortable, and it belongs in the record. The document authorising that work had a section in which we argued against ourselves and wrote down, in advance, what we thought could go wrong. It named this exact risk. It also named the double-entry check as the thing that would catch it. That prediction was wrong. The double-entry check would have sat there balancing at zero and reported nothing amiss. We identified the right hazard and named the wrong safeguard, and only reading the code found that out.

When the tool was later run properly against a real application, the guard was live and the numbers were not vacuous: a hundred and fifty-five files considered, a hundred and twenty-seven examined, twenty-eight excluded, fifty-eight findings. Those are numbers you can interrogate. Zero is not.

The same problem wearing a different coat

There is a milder and far more common version of this, and we have measured it.

We examined one AI-generated application twice, against the identical code, changing nothing but the depth of the look. The first pass ran the way most fast checks run, without installing the project’s dependencies, and returned a hundred and forty-two findings and an impression of a slightly untidy but basically sound application. The second pass installed the dependencies so the analysis could reason about the code properly. It returned seven hundred and thirty-seven.

Five hundred and ninety-five additional findings, all in a single class, and the first pass had not missed them through error. It could not see that class at all. It was blind to it structurally, by virtue of how it had been run.

Nothing about the code changed between those two numbers. Only the depth of the look changed.

That is one application, chosen because the contrast in it is unusually clean, and it demonstrates a mechanism rather than estimating a population. The additional findings are candidates indicating lost protection, not a count of guaranteed failures. But the shape is the same as the empty-list case: a number that looks like a verdict and is actually a report on how far somebody looked.

The question that separates the two

A low finding count means one of two opposite things. Either the code is genuinely clean in that dimension, or that dimension was never exercised. From the number alone, you cannot tell which, and the number will never volunteer the difference.

So the question to put to any clean result is not “are you sure?” That invites a confident yes you have no way to evaluate. The question is: what was in scope, and who verified the scope?

That question has an answer that either exists or does not. A report that can state, for each class of risk, whether it was fully examined, partially examined, limited by the environment, or not examined at all, has given you something you can reason about. A report that returns a count with no such map has given you a number wearing the costume of a verdict.

A clean result tells you what was examined only if you already know what was examined.

Of the last clean report you accepted: what was in scope, and who checked the scope itself?

Where this comes from, and its limits

I sell software review services. That is what I do for a living, and it is the reason I have opinions about this.

Findings referenced here are anonymised. The evidence has stated coverage limits, and they are in the body above rather than in a footnote: the empty-scope defect was caught by inspection and never occurred in any run, and the two-pass comparison is a single application demonstrating a mechanism, not a population estimate. Security findings are candidates, not confirmed exploits. No secret values are shown. Code examples, where used, are illustrative composites and not audited source, and this edition contains none.