Roman Hromenkov, founder · Tallinn
Our own website once carried numbers nobody had measured, and every automated check passed. Why a passing test can prove nothing, and three questions anyone can ask before accepting that something is done.
When AI writes most of the code, the bottleneck moves. Writing is fast now. Knowing whether what was written actually works – that is the slow, expensive part, and it is where most of our effort goes.
The trap is the green tick. Tests pass, the pipeline is green, the report says "done". It feels like proof. Often it is only proof that the checks ran.
Our own website, every check green
In August we went through our own website line by line and found claims nobody had measured: "save 10–20 hours a week", "up to 80% less manual work". They read like case results. They were not. Nobody had invented them on purpose – they had arrived with templates and early drafts and simply stayed.
Every automated check on the site was green. The end-to-end tests clicked through every page. Visual regression compared every screen pixel by pixel. Performance scores were near perfect. And not one of those checks could notice that a sentence on the page was untrue, because none of them was asked to.
We removed every number that had no measurement behind it. What replaced them are seven real systems, each with a result we measured, and a rule the build enforces: a claim on a card must match its source document byte for byte, or the build fails.
The test that can't tell
The harder version of the same problem lives inside the code.
On a side project we spent seven rounds of review on two pieces of work. The reviews kept finding the same kind of defect, and it was one that green checks cannot catch by construction: a test that passes whether the code works or not.
It looked like this. A guard was supposed to reject a bad upload for four different reasons. There was a test for the guard, and it was green. But the test used one sample file that broke all four rules at once, and it only checked that some error came back. Remove any one of the four rules and the test stayed green – the other three still produced an error.
In the same work, an image library was quietly stripping location data from photos on its own. So the obvious test – "the location doesn't leak" – would have passed forever, without ever testing our code.
The fix is simple to say: every rule gets its own test that fails when that rule is removed, and only then. We now check this on purpose. We break the guard deliberately and watch the test turn red. If it doesn't, the test is decoration.
(That particular feature was later removed by the product's owner for product reasons. The method stayed – it is now a check that runs on the whole project.)
Checked in a browser, every time
Before any of this, our most tedious job was clicking through the product by hand after every AI-written change, to make sure it really worked. So we automated the clicking too: after each change, an agent opens the product in a real browser, walks through what changed, and brings back evidence – screenshots, the database rows it created, the exact steps.
The point is not the screenshots. The point is that "done" now has to arrive with something you can look at. A report that says "verified" without a run behind it doesn't count.
Three questions anyone can ask
You don't need to read code to hold a developer – or an AI – to this. Before accepting that something is done, ask:
- Have you seen this check fail? A test that has never been red may not be testing anything. Ask what happens if the feature is broken on purpose.
- What does it prove, in one sentence? "The page loads" and "the invoice total is correct" are very different claims. Green only covers the one that was written down.
- Can I see it run? Not the summary – the run. A screenshot, a recording, the rows in the database.
None of this is slower. It is the difference between knowing and hoping, and with AI writing the code, hoping gets expensive fast.