A lot of the AI your business could use is not a conversation. It is a (big)queue of small, repetitive judgment calls.
Which team gets this ticket? Is this urgent? Does a person need to look at this before we reply? How likely is a refund?
Today, if you automate those with a frontier model, you pay chat prices and wait chat seconds for a one word answer. If you switch to a small model to save money, you get a cheaper answer you cannot trust.
This week a third option appeared, and we wanted to measure it before talking about it.
What we did
We built a support inbox for a fictional bank: 300 tickets, in English and Portuguese, some clear, some deliberately messy. For each ticket we asked four questions with fixed answers. Priority, one of four. Team, one of five. Needs a human before reply, yes or no. Refund risk, one to five. Three readers set the correct answers.
Then we asked the same four questions, on the same 300 tickets, to three models. A current frontier model. A small, fast chat model. And Jev, the new model from TypeSafe AI, which we have in early access.
What we found
Cost. One million decisions a month costs about 1,522 dollars with the frontier model, about 424 dollars with the small model, and about 11 dollars with Jev. Measured on list prices, on our own run.
Speed. The frontier model took just under four seconds per ticket, the small model just over two, Jev 0.7. That is the difference between a decision that waits in a batch and one that happens while the customer is still on the page.
Accuracy. The frontier model and Jev tie at 80.7 and 80.4 percent. The small model drops to 65.8. All three were excellent at picking the team. The gaps opened on the judgment calls, priority and refund risk, where our human readers also disagreed with each other on same of the same tickets. On this task, 80 percent is close to what the task allows.
The number that matters
Every answer from every model came with a confidence score. We checked whether the scores could be trusted. The frontier model and Jev keep their word: when either says 90 percent, it is right about 90 percent of the time. The small model does not. It said 90 percent or more on more than half of all decisions and was wrong on a fifth of those. Worse, it was wrong in the same direction every time, rating medium tickets high and high tickets urgent, always with confidence. A cheap model that is confidently wrong in a consistent direction is the worst thing you can put in front of a queue.
So the question is not which model is best calibrated. It is how often a trustworthy model is sure enough to act without a person. That is the number that decides your headcount.
Set the bar at 95 percent confidence. Jev clears it on 38.8 percent of decisions and is right on 98.9 percent of those. The frontier model clears it on 15.2 percent. Set the bar at 90 percent and Jev handles half of all decisions alone at 96 percent accuracy, against 31 percent for the frontier model. The small model clears the 90 percent bar on more than half of all decisions, and gets a fifth of those wrong.
Same trust level, two and a half times the automation. The remaining decisions go to a person, exactly as they do today, except the person only sees the hard ones.
What it means for you
The design is simple enough to explain to a board and an auditor in one sentence: the system acts alone above a confidence you choose, escalates below it, and logs every decision with its score. The threshold is one line of code your team owns. Raise it when the stakes are high, lower it as the evidence builds.
Two honest caveats. This is one scenario with four questions, and Jev is early access at launch pricing. What it shows is not that Jev wins everywhere. It shows that a new class of model exists, it works, and the economics of automating small decisions moved by two orders of magnitude in a week.
This is the same lesson as our last post on AI written tests. The doing has become cheap. Deciding what to decide, writing the rules down, and checking the result are where the value sits now, and those are people's jobs.
Try it on one of yours
If you have a queue of repetitive decisions that a person still clears one by one, we will run this exact test on one of them: your questions, your data rules, three models, the numbers in a week. Send me the decision.
