top of page
Digital-Price-Works-logo

The Token Trap: Why Nobody Can Predict What AI Actually Costs

  • Writer: Jorge Rodriguez
    Jorge Rodriguez
  • 4 days ago
  • 6 min read





The Token Trap: Why Nobody Can Predict What AI Actually Costs


Last year, thousands of developers using the AI coding tool Cursor woke up to a different bill than the one they'd signed up for. The company had sold a flat $20-a-month, "use it as much as you want" plan — and then discovered that some users were consuming far more compute than that price could ever cover. Cursor switched to a credit-based model, the backlash was immediate, and the CEO ended up publicly apologizing within weeks.


This wasn't a careless vendor. It was a symptom. Across the AI industry, sellers keep changing their pricing after launch, and buyers keep getting surprised by bills that don't match their expectations. The usual explanation — "AI is just expensive" — isn't quite right. The real story is that almost nobody, including the companies selling it, has fully worked out how to measure what they're actually charging for.


To understand why, we need to open up the one word every AI invoice is built around: the token.


Two different bills hiding inside one product

Every AI model carries two separate costs that get talked about as if they were one.

Building the model (training) is a one-time, enormous expense — months of specialized computer chips running around the clock to teach a model how language works. Like building and outfitting a restaurant kitchen before it ever serves a customer.

Running the model (inference) is what happens every single time someone sends it a question. This is the ongoing cost, and over a model's life it dwarfs the training bill — simply because a popular model answers billions more questions than it took training runs to build it. The kitchen gets built once. It cooks meals forever after.

Pricing is really an inference problem. So that's where we'll stay.


A token is a bite, not a meal

A token isn't quite a word — it's a small chunk of one, roughly four characters. When an AI answers you, it isn't producing the whole response at once. It's plating one bite at a time.

Reading your question is fast: the kitchen can glance at the whole order in one go. Writing the answer is slow: each bite has to be individually cooked, one after another. That's the real, physical reason output tokens cost three to ten times more than input tokens across every major provider — it isn't a markup, it's genuinely more work.

Here's where it gets strange, though. You'd expect a hot dog to always cost about the same as a hot dog. In AI, it doesn't. Order the exact same thing twice, and you might pay hot-dog money once and Michelin-three-star money the next time — and almost nobody, buyer or seller, can fully explain why in the moment.



The kitchen's rent doesn't care if it's quiet tonight

Some of that unpredictability comes from the seller's side — specifically, from a question nobody outside data centers usually thinks about: how busy is the kitchen?

The chip running an AI model (the GPU) costs money by the hour, whether it's cooking or just sitting there warm. If the kitchen is slammed all night, that hourly rent gets spread across hundreds of meals — cheap per plate. If it's half-empty, the same rent gets spread across far fewer meals — expensive per plate, even though nothing about the food changed.


This isn't theoretical. Industry cost models show that a chip running at low utilization can end up more expensive per token than a premium, fully-loaded one — sometimes by an order of magnitude, on identical hardware. One detailed engineering estimate found the same GPU and model running a single real-time request, versus running efficiently batched requests, swung the cost per million tokens by roughly 17x — purely from how full the kitchen was, not from anything the customer asked for.



This is the first reason "$ per million tokens" is a much less stable number than it sounds.


The bite itself can be cut differently

Here's the second, stranger reason — and it's specific to AI in a way the kitchen analogy starts to strain against. The same output text can be chopped into tokens in more than one valid way, even when the words on the page are identical. Two customers asking for the same thing, getting the same answer, can be billed a different number of "bites" — and because the buyer can't see how the cutting happened, they have no way to check.

It gets sharper still: because providers profit from the length of the token sequence they report, researchers have pointed out a built-in incentive problem — billing by a unit the seller controls and the buyer can't verify is a textbook misaligned incentive, not a quirk. And the effect isn't evenly spread: processing text in some languages costs four times more than English, purely as a side effect of how the technology was built — nobody designed that as a pricing decision; it just happens.

So even before your own files enter the picture, the unit itself is slippery.


Then your own paperwork makes it worse

This is the part almost no buyer is warned about. How you feed information to an AI tool changes the bill dramatically — often by 10x or more — for reasons that have nothing to do with how complex your question is.

  • Format matters more than content. A messy PDF or an HTML page can carry up to 10 times more "tokens" than the same information cleaned up as plain text, just from formatting noise the AI has to wade through.

  • Scanned documents cost far more than digital ones. A single scanned invoice can run 1,000–5,000 tokens once processed — a cost the exact same invoice, received as a clean digital file, would mostly avoid.

  • Long conversations quietly compound. Each new message in a chat resends the entire conversation so far. By the tenth message, that single exchange can cost roughly seven times what the first one did — for what feels, to the person typing, like just one more line.

  • The model picked behind the scenes matters enormously. Routing the same task to a premium model instead of a budget one can multiply the bill by 10x or more, often invisibly to the end user.




None of these are things a typical buyer would think to ask about before signing up. They're infrastructure decisions, made upstream, that quietly decide whether your bill looks like a hot dog or a tasting menu.


Cheaper overall — and still impossible to predict

Here's the genuinely confusing part to close on: AI inference has also gotten dramatically cheaper over time. Independent research tracking cost-per-equivalent-performance found the price for a fixed quality benchmark fell from roughly $60 per million tokens in late 2021 to about $0.06 by late 2024 — a thousandfold drop in three years.

That's a real, well-documented trend. But it sits awkwardly next to everything above it: the average price is falling fast, while the specific price any one customer ends up paying remains genuinely hard to forecast — because it depends on infrastructure utilization the buyer can't see, a unit (the token) that isn't measured consistently, and document and conversation choices most buyers don't realize are pricing decisions at all.

That's the real definition of the token trap. A token was adopted as the billing unit because it happens to be what the machine already counts internally — not because anyone decided it was the right way to measure what a customer is actually getting. It tells you what the kitchen did. It doesn't tell you what the meal was worth.

Solving that properly will take more than a cleaner invoice — it needs a hard look at how AI pricing connects to actual value delivered, and at the market behavior (subsidized pricing, land-grabs, race-to-the-bottom token rates) currently masking how unstable the underlying economics really are. That's where this series goes next.

Sources & further reading

 
 
 

Comments

bottom of page