The AI bill was too high
A document-processing SaaS company was using a frontier model to pull structured data from insurance policy documents. It worked. Customers liked it. At $10,000 a month, it still cost less than hiring people to review the documents manually.
Then the company planned its next round of features. It wanted to support more document types, extract more fields, and process more customer volume.
The projected AI bill hit $50,000 a month.
Their CTO knew other models cost less, so he tested one in production. The results looked pretty great at first, but customers noticed enough mistakes to call support. He rolled it back three hours later.
After that, nobody wanted to try again. The current model was expensive, but changing it felt like gambling with customer trust.
Why the first attempts didn’t work
The team tried cleaning up its prompts and cutting token usage. That saved about 15%, but the system still depended on the most expensive model tier.
Next, they tested a cheaper model against five documents. Those documents looked fine, so they released the change. Real customer documents exposed problems the test hadn’t caught, and the team rolled it back.
They also received an $80,000 proposal from an offshore team to rebuild the AI infrastructure over six months. The proposal didn’t guarantee what the monthly bill would be afterward, or a long-term plan for upgrades.
Their developers were good. They just didn’t have a reliable way to measure whether a cheaper model could actully handle the workload.
How Fixation approached it
We started by defining what a good result actually was.
The team gave us 500 real production documents with outputs it had already accepted, by humans, assisted by frontier models. Those became the reference set. We weren’t testing against generic benchmarks; we were testing against the company’s real work.
Then, we built an automated evaluation suite that could run those documents through any model, and any prompt, and compare the results with the accepted outputs. It measured field accuracy, formatting, and the edge cases that had caused trouble in production.
The team could now compare models with actual numbers. One might score 94%. Another might score 96%. A third might score 91% while costing one-sixth as much.
Most of the cheaper models didn’t pass. One smaller model reached 89% accuracy, which wasn’t ready for production, but its mistakes looked fixable.
We adjusted the prompts one failure pattern at a time. Each change went through the full evaluation again. After about a dozen rounds, the smaller model reached 95% accuracy.
The CTO saw the results in weekly demos: the current score, the cases that failed, and what we’d work on next. He didn’t have to take our word for it. This was presented alongside all our other development work on the product.
We released the new model behind a feature flag. It received 5% of production traffic first. We watched its accuracy and error rates, then increased traffic to 20%, 50%, and finally 100%.
The full process took three weeks.
The result
The company’s monthly AI bill fell from $10,000 to $1,600. That’s $100,800 a year back in the budget.
Accuracy also improved because the evaluation work exposed prompt problems that had existed with the original model. Customers didn’t report any decline in quality.
The company owns the evaluation suite, test cases, and scoring code. Its developers can test a new model in an afternoon without risking production traffic.
Their CTO put it this way:
“Before, we were afraid to make changes. Anything we did meant potentially using customer trust. Now we can ask Fixation to test changes like any other code. The evaluation suite made it easy to upgrade models, which we have done twice now, and we’re using that money to re-invest in new engineering.”
Six months later, the company had released three more AI features using the same process. Its total AI spend remained below $2,500 a month, even with five times as much AI-powered functionality in the product.
Could your AI bill be lower?
A growing AI bill doesn’t always mean the work itself is expensive. Sometimes the company simply can’t measure quality without relying on its current vendor.
Once you can measure the results, switching models becomes a normal engineering decision. You can compare providers, respond when prices change, and add features without wondering what they’ll do to next month’s invoice.
Fixation helps companies build evaluation systems and AI architecture that keeps models replaceable. You own the code, the tests, and the ability to change providers.
If your AI costs keep climbing and changing models feels too risky, let’s talk.
Ready to transform your business?
Let’s discuss how we can help you achieve similar results.
