What Builders Told Me About the Day “It Works” Stopped Being Enough
Mohan
14 October 2026
Why a series. The public argument about AI is being held
Why a series. The public argument about AI is being held by a handful of industry leaders, and it is mostly about speed: go faster, or slow down. Underneath it, something quieter is happening. In comment threads, small teams and one-person companies, practitioners are working out their own answers to a different question: how do we control what AI is allowed to do, and how do we prove what it built? Nobody is coordinating it. It is spreading from the ground up. This series is a report from that level.
My son built an app.
He used one of the AI app builders, and in a few days he had something that looked like a real product. Then he asked his dad to help install it and connect the database.
So I walked through it. I read the code – just a few selected examples. And what I found was that only the happy paths had been built. As long as the user did exactly what was expected, it was lovely. The moment something went pear-shaped, it fell over.
Nothing had told him. The tool that wrote it was the only thing that had ever looked at it.
I would like to say I was surprised. I was not. I have built six apps with AI myself, and every one of them looked finished before it was.
That gap, between looking finished and being finished, is what I have spent the last several weeks asking other builders about.
“Vibe coding” is less than two years old as a phrase. Andrej Karpathy coined it in February 2025, describing a way of working where you “fully give in to the vibes” and “forget that the code even exists”. [5] It is now how a great many real products get built.
I am for it. For most founders the alternative to AI-built software is not perfect hand-built software. It is no software.
So I did not go looking for an argument. Whenever a builder posted about shipping with AI tools, I asked the same thing underneath: what actually went wrong, and how did you find out?
Here is what they told me.
Uri Laish built a production control system with AI assistance, and made the point I have repeated most often since. The quality of the code had not dropped. The time had been compressed.
If two years of development arrives in one month, two years of ordinary software problems arrive in one month too. Count the defects without counting the development behind them and you will badly misread what happened.
It is a fair answer to the “AI code is full of bugs” headlines. It also leaves a harder question on the table. When a month of output has to be trusted in production, who sorts the ordinary problems from the serious ones?
Manzurul Haque runs several AI models like an engineering team: one builds, another reviews, another audits. On a project he calls the best work of his life, everything worked locally. No errors. No failing tests.
Then, in staging, after one particular sequence of actions, a strange bug appeared. It took him three days to find the cause, and the cause was simple. During a performance audit, the second model had silently changed an npm package. The app kept working, and several more commits went in before anything showed.
By then, in his words, “the cause and the symptom were separated by multiple commits.” The models kept trying to fix the visible problem. They were debugging the symptom.
What saved him was not a cleverer prompt. He stopped asking the AI to “fix the bug” and had it build a diagnostic script that compared the two environments: installed packages, dependency versions, lockfiles, build configuration, environment variables. In other words: what is objectively different between the system that works and the one that does not?
That comparison exposed the mismatch. His own verdict is better than mine: “The AI didn’t save me from the problem. My understanding of the system helped me use AI to find it.”
I love that story, because the failure was never “the AI wrote bad code”. The speed had compressed everything, including the dependency changes and the handoffs between agents, and nothing was keeping a record.
Mahdy Hasan wrote about an AI-built app that reached about 200 concurrent users and choked.
It had not looked unfinished. It had simply never been asked what it would do under load.
This is my son’s app at a larger scale. A founder does not have to become an architect. But before real people arrive, somebody has to know what was tested, where the limits are, and what is still unknown. Otherwise the users are the test suite, and they did not sign up for that.
Jeffrey Kesselman reminded everyone of the oldest joke in the trade, which he calls the 80/80 rule: the first 80 per cent of the project takes 80 per cent of the time, and the last 20 per cent also takes 80 per cent of the time. That last 20 per cent is the polish, the deep testing, the debugging.
AI makes the first 80 per cent arrive fast and look wonderful. That is the trap. Polish is visible. Deep testing is not. The person receiving the software cannot tell whether that final stretch happened.
Antonio Šajatović describes the moment every fast builder dreads with a picture I cannot improve on. You are the safety operator of a self-driving car. The trip is smooth, so you stop watching the road. Then the car turns down an unmarked track and halts, with the dashboard full of error codes.
“You are now stranded,” he writes. “Because you relied on the AI blindly, you have zero situational awareness.”
He has a name for what built up along the way: epistemic debt. You did not lose the code. You lost your map of it.
I asked him what was usually missing when this happened, and how he got out. His answer was not a better AI. It was detailed tracing: better evidence of what the system was actually doing.
Three public records say the same thing from different angles.
We are poor judges of our own speed. In July 2025 the research group METR ran a controlled study with 16 experienced open-source developers on 246 real tasks. With AI tools they took 19 per cent longer. They had expected to be 24 per cent faster. Afterwards they still believed they had been 20 per cent faster. If experienced developers cannot feel the difference, “it felt fine” is not evidence. [1]
The gaps are real when someone looks. Escape Technologies examined about 1,400 vibe-coded applications and reported 2,038 highly critical vulnerabilities, more than 400 leaked secrets and 175 instances of exposed personal data. That is a vendor’s own study, so read it as one. But every one of those apps was, as far as its builder knew, working. [2]
And an instruction is not a control. In July 2025 an AI coding agent deleted a live production database belonging to SaaStr’s founder, Jason Lemkin, during a declared code freeze. The agent’s own summary afterwards: “I violated explicit instructions, destroyed months of work, and broke the system during a protection freeze.” The platform’s chief executive called it “unacceptable and should never be possible” and shipped separation between development and production within days. [3]
None of them began with a dramatic defect. In every case the software was running.
The trouble began at one particular moment: when someone other than the builder had to rely on it. A client paying the final invoice. An investor. A new developer. A father asked to connect a database. Sometimes the founder himself, a month later.
I have written before about that moment, between “it works” and “I can take responsibility for it”. [4] What these weeks added is that builders already know it. They just meet it one at a time, alone, usually on a bad day.
If you are about to receive an AI-built system, or hand one over, four plain things should come with it:
what was examined;
what was not;
which findings actually matter;
what is still uncertain.
And ask how each was established. Laurence Svekis wrote about turning AI prototypes into production software, and I added one distinction under his post that I will stand by: review is not the same as evidence. An AI reviewing its own code, or a second AI reviewing the first, is useful. It is not independent. The strongest checks are the ones a model cannot talk its way past. Run the tests. Sign in as a different user. Restore the backup. Deploy from cold.
Do not ask “has this been reviewed?” Ask “what has actually been shown to be true?”
I am not saying AI-built software is worse. Most of what these builders described is ordinary software trouble, arriving faster.
I am not saying a review makes an app secure, or that any honest report can say “pass”. A review is one copy of the code, on one day, and it should say what it did not check.
I am saying we have become very good at measuring what AI gives us: speed, cost, access. We are much less disciplined about what we traded for it.
My son’s app taught me that. So did six of my own.
If you have been on either side of a handover like this, tell me what went wrong and how you found out. It is the most useful question I know.
Next week, Part 2: the six objections I keep hearing to controlling what AI is allowed to do, and what each one taught me.
References
METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, 10 July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
Escape Technologies, “The State of Security of Vibe Coded Apps”. https://escape.tech/state-of-security-of-vibe-coded-apps
eWeek, on the Replit agent and the SaaStr production database, July 2025. https://www.eweek.com/news/replit-ai-coding-assistant-failure/
Mohan Iyer, “Before You Release the Final Payment: From ‘It Works’ to ‘I Can Take Responsibility for It'”. https://trustbeforeship.com/before-you-release-can-i-take-responsibility-for-it/ – and “A Clean Result Is Not the Same as a Clean System”. https://trustbeforeship.com/a-clean-result-is-not-the-same-as-a-clean-system/
Andrej Karpathy, post on X, 2 February 2025. https://x.com/karpathy/status/1886192184808149383 – quoted as given in Wikipedia, “Vibe coding”. https://en.wikipedia.org/wiki/Vibe_coding
Disclosure: I run Trust Before Ship, which reviews AI-built software before launch. That is a direct commercial interest in the argument I am making here.
Why a series. The public argument about AI is being held
The Announcement On 28 September NVIDIA announced its Open Agent Safety
Sometimes Zero Bytes Is the Right Result A system that stops
Guardrails Aren’t Enough. Who Verifies the Guardrails? A control is an