Roman Hromenkov, founder · Tallinn
A new AI model came out at a tenth of the price. We moved our farm product to it one task at a time, each step measured on our own cases. Seven tasks got about twenty times cheaper, five stayed where they were, and the most useful finding was a switch that did nothing.
When a new AI model comes out cheaper than the one you use, the tempting move is to change one line and enjoy the smaller bill. We didn't, and this is what that cost us and what it saved.
Our farm product uses AI in about a dozen places: it reads a farmer's voice note, answers questions in the chat, pulls a field list out of a government PDF, suggests the next field activity, reviews compliance with subsidy rules. Until this week, most of that ran on two models, one cheap and one capable.
Then a new small model came out. By its price list, it costs ten times less than the cheap one we had: $0.10 per million tokens read, against $1.00. There is a catch in the small print: it counts about 30% more tokens for the same text. So the real price for our text is about an eighth, not a tenth. Still a big number – if the answers hold.
One question per task, not one per product
"Is the new model good enough?" has no answer. "Is it good enough to extract parcels from this PDF?" does. So we asked it twelve times, once per task, and let each task get its own verdict.
For every task we already had a set of real cases: real voice notes, real PDFs, real chat questions, each with the answer we expect. Both models answered the same cases in the same run, through the same code the product uses. The rule for "holds" was written before the run:
- no more hard failures than today – an invented id, a broken structure, a missing step;
- the pass rate and the average score drop by no more than 0.05;
- for the chat, a stronger model judges both answers, and no part of the scorecard may get worse.
The whole run made 873 paid calls and cost $13.71, against a cap of $18 we set before starting.
What the measurement said
Seven tasks held: classifying a request, polishing a brief, reading voice notes, summarising crop damage, extracting parcels from a 38-parcel PDF, suggesting an activity, and simple chat questions – where the new model actually did better, 15 of 20 against 11 of 20.
On those seven, the same cases cost $4.26 on the old models and $0.21 on the new one – about twenty times less. That is measured, on our cases, with the same price arithmetic the product's cost ledger uses.
Five tasks did not hold, and they stayed where they were. Harder chat conversations scored lower. The compliance review – checking a farm's records against subsidy rules – went from 1 hard failure to 9: the new model often didn't return its review in the required form at all. We measured the heavy tasks a second time two days later, and the result was the same. For those, a cheaper wrong answer is not a saving.
What a farm's month looks like
For a typical farm, our calculation puts the AI cost at about €2.25 a month instead of €5.48, which takes AI from 11.4% to 4.7% of the smallest plan's price.
This is a calculation, not a bill: it multiplies the measured size of a typical conversation by the new prices, and only the chat moves in it. We don't have a month of production usage on the new model yet. When we do, the ledger will say whether the estimate was right.
The switch that did nothing
The most valuable finding of the week wasn't about the model.
We built an admin page where the platform admin can switch any task to another model, without a release. Roman switched the simple chat to the new model and ran a test conversation on production. The cost ledger – which records every paid AI call with its model and its money – showed all 35 calls on the old model.
The page had saved the choice. The code that should have read it existed. But none of the places that actually call the AI asked for it: each one still used its built-in default. Every switch on that page had been a no-op since the day it shipped – and the page said "switched".
Nobody would have found this in a code review; it read correctly. It was found because a different part of the system recorded what really happened and someone compared the two. It is fixed now, with a test that fails if any caller stops asking.
Two smaller things of the same kind. The new model refuses some settings the old one accepted, so a few calls had to change before they could run at all. And one early version of the comparison happily reported "holds" for the new model measured against itself.
How Claude Code did the work
Most of this was done by Claude Code sessions running in parallel, each in its own copy of the repository and each on one item: one built the model catalogue and the admin page, one wrote the comparison, one probed which models support the features we rely on, one recorded the costs. One session collected the finished work into a single change for review and release.
The rules they worked under matter more than the speed. A defect gets a failing test before it gets a fix. Every claim of "it works" comes with a run. And a test is checked by breaking the code on purpose – reverting the model defaults turned six tests red, which is how we know they test something.
The decisions stayed human. Which tasks move, what counts as "holds", how much a run may spend, and who pays when an admin picks a cheaper model – Koru does, the farm's price doesn't change – were Roman's calls, each written down.
If you are about to switch models
- Ask per task, not per product. The same model was twenty times cheaper and just as good on seven tasks, and clearly worse on five.
- Write the pass rule before the run. Otherwise "close enough" moves to wherever the result landed.
- Read the price list to the end. A model that is ten times cheaper per token and uses 30% more tokens is about eight times cheaper.
- Check that the switch reached the call. A setting that was saved is not a setting that is used. Look at what was actually billed.