Before You Ship: A Working Demo Is Not an Evidenced Milestone

The Claim

A visible demonstration shows that something ran. It does not, by itself, establish that the identified software build supports the milestone being claimed

The twenty minutes that decide the money

An investor watches a demo. The founder clicks through a booking flow: an account is created, a slot is selected, a payment succeeds, a confirmation appears. Twenty minutes, nothing breaks, and the tranche request on the table says the platform is feature-complete and production-ready.

Everyone in the room now believes something slightly different, and nobody says which.

The founder believes the product works, and they are not lying — they have used it. The investor believes they have seen evidence, and they have seen something. What was actually established is narrower than either party will say out loud: on one machine, on one path, on one occasion, a sequence of operations completed without visible failure.

That is not nothing. It is genuinely not nothing. It is just not the milestone.

What a demo is structurally good at

A demonstration is optimised, by everyone involved and usually without deceit, for the path that works. The founder walks the route they have walked fifty times. The data is the data that has always been there. Nobody enters a malformed date, cancels mid-payment, or opens two tabs and books the same slot twice.

That is a real form of evidence, and it should be treated as one. A demo establishes visible function: this build, on this path, did the thing. It is the fastest and cheapest way to answer that question, and no other instrument answers it as directly.

AI-assisted development has made that path faster to build than at any point in the history of the trade — and it has made it faster to build than the paths around it. Assembling a smooth interface and a working happy path is precisely what these tools do best. They also produce output that is articulate and finished-looking, which raises a reader’s confidence faster than it raises the assurance underneath. The gap between how good the demo looks and how much has been established has widened, and it widened quietly.

The gap has been measured, and not only by us

Independent research has been pointing at this for two years. A Stanford study (Perry et al., 2023) found that developers using AI assistants produced more security vulnerabilities while reporting higher confidence in the safety of what they had written. A Purdue analysis (Kabir et al., 2024) found that a majority of AI programming answers contained functional or logical errors, and that users accepted a substantial share of them anyway — the well-formed, articulate answer being harder to doubt than a scruffy one. Veracode’s 2025 analysis reported that a large fraction of AI-generated code introduced known vulnerability classes.

Those are three different research groups measuring three different things, and none of them is measuring your build. Cite them for the shape of the problem, not the size of yours.

What we found when we looked twice at the same code

We examine AI-built applications and produce a bounded evidence record. Across five applications examined read-only, we located 296 findings.

Some of the tendencies people describe anecdotally are, in that record, counted rather than asserted. Of those 296: 168 were duplicated implementations of the same thing, 37 were dead or unused branches, and 91 were dependency advisories. The duplication figure carries a caveat we would rather state than have someone else find — a large share of it is generated interface boilerplate copied into place, which makes that code copy-paste-heavy, not broken. Weighted honestly, it says something narrower than it first looks.

The more interesting number is what happened when we examined a single application twice.

One substantial application — 149 files, a real backend — returned 142 findings on a read-only pass. The same application, unchanged, examined with a deeper type-aware pass, returned 737. The additional 595 were unsafe type-use signals: places where the code has quietly stopped being able to tell you what it is handling. They are not compiler failures — that build type-checks — which is exactly why nothing in the demo would have shown them.

Same code. Same commit. Roughly eighty per cent of what we could eventually locate was invisible to the shallower look.

We publish that number against ourselves, because it is the whole point. If the depth of examination changes the finding count by a factor of five, then any statement of the form “we checked it” is meaningless until it says how far it looked. A demo is the shallowest look available.

Why the usual instruments do not close the gap

The reasonable objection is that this is what scanners and test suites are for.

They are, and they are good at what they do. But a static analyser reads what is present. It is very effective at finding a dangerous pattern in code that exists, and structurally incapable of noticing an obligation that was never implemented at all. There is no line for it to flag. Multi-tenant isolation that nobody wrote, an unhappy path nobody handled, a rate limit nobody added — these are absences, and absences do not appear in a report of what is there.

Automated tests inherit a version of the same problem. When tests are generated from the code they are testing, they encode the assumptions already in it. They confirm that the thing does what it was written to do. They cannot tell you whether what it was written to do is what the milestone requires.

This is not a claim that we detect more than those tools. We do not make that claim and it is not our argument. Snyk, Sonar and a competent penetration tester will find things we will not. Our claim is different in kind: we produce a record aimed at a decision, and we state what it does not cover.

What the gap costs, and who carries it

The gap is not an abstraction. It shows up as ordinary operational trouble.

For a founder, it usually arrives in this order: a launch date committed on the strength of the demo; customers arriving at paths nobody walked; remediation interrupting the roadmap that the funding was raised to build; an agency or a senior hire spending their first weeks reverse-engineering generated logic before they can add to it. The most expensive item is the quietest one — being unable to say clearly, when asked, what is still uncertain.

A funder carries a narrower version of the same thing. They are not asking for a guarantee that the software is perfect; no one can supply that. They need to know what was actually examined, what remains unverified, what could block the next release or capital decision, and whether the next tranche funds growth or funds hidden remediation.

What would make a milestone evidenced

A milestone claim is a claim about obligations — the things the software must do for the milestone to be real. Payment captured only after authorisation succeeds. A failed payment leaving no confirmed booking. A customer seeing only their own organisation’s data.

An evidenced milestone means those obligations were stated by the people making the claim, examined against an identified build pinned by content hash, and reported with each obligation’s evidence state named — found, partial, absent, conflicting, or unassessed — against a declared scope.

To make the shape concrete: the illustrations that follow are hypothetical, not findings about any software we have examined. A payment path that visibly succeeds while nothing establishes what happens on a retry or a rollback. A login that works while tenant isolation is unverified. A polished interface shipping over dependencies nobody has examined. Each of those demos beautifully. None of them is settled by the demo.

A record that answers this has to carry, at minimum: which software was examined and at which build; where each finding is located; how severe it is and how confident we are; what the examination covered and what it explicitly excluded; what can and cannot be relied on as a result; what should be remediated first; and which decision the record actually supports. Those parts are described here, not demonstrated — the full specimen record is a separate piece of work and is not yet available to show.

That last clause carries the weight. The point of naming what was not examined is that the reader can see the shape of what they still do not know. Absence of a finding is not evidence of absence. Areas outside the stated scope are marked unverified — never passed.

What this does not tell you

This essay is about one narrow gap and does not close the others.

An evidence record cannot tell you whether the market wants the product, whether the founder can build a company, or whether the valuation is sane. It does not predict success. It has nothing to say about behaviour under load or at production data volume unless someone paid for a running system to be examined, which is a different engagement.

And our own evidence has limits we will state before anyone asks. Five applications is a small number. The findings are located and reproducible, not confirmed exploits. Security findings are candidates carrying coverage caveats, not proven breaches. We have not been independently benchmarked against specialist tooling and we do not claim to be better than it.

The decision

If you are releasing capital against a technical milestone, the question is not did the demo work? You saw that it did.

The question is: what does the supplied software evidence establish about the milestone being claimed — and how far did anyone actually look?

A demonstration answers the first half of that question and is silent on the second. The silence is the part that costs money.

Answering it does not force a single outcome, and it is worth being explicit that the answer is rarely stop. A record of this kind supports proceeding; proceeding on stated conditions; fixing named blockers first; gathering more evidence in a specific area; holding a release; or concluding that the evidence is insufficient to decide either way. The examination is not designed to fail the application. It is designed to prevent unsupported certainty.

So before treating a working demonstration as an achieved milestone, ask five questions and expect answers to all of them. Which paths were actually tested? Which obligations were explicitly checked? What remains unverified? What evidence supports the release or funding decision? And what must be fixed next?

If those five have answers, you are deciding on evidence. If they do not, you are deciding on a demo — and you should at least know which one you are doing.

Have a working AI-built application but no bounded evidence record?
Request a private review to understand what has been examined, what remains unverified, and what the next release, funding or handover decision requires. Here

Mohan Iyer is the founder of Trust Before Ship and is developing the Vibe Code Forensic Audit.

Trust Before Ship produces a read-only, bounded evidence record for AI-generated software — so founders, technical teams and decision-makers can distinguish a working demonstration from a defensible milestone.

Findings are presented in aggregated or de-identified form, with identifying details withheld. Evidence has stated coverage limits. Security findings are candidates, not confirmed exploits. No secret values are shown. Code examples, where used, are illustrative composites and not audited source.

Sources

Stanford study (Perry et al., 2023)

Purdue analysis (Kabir et al., 2024)

Veracode’s 2025 analysis