Glossary

Inference cost

Also: Token cost

The running cost per request to a language model, billed by tokens in and tokens out.

An AI system has a price per operation, and you should know it before volume grows. Billing is by token, meaning text fragments, split into input and output. Output is regularly several times more expensive than input, which explains the most important design rule: force short answers, long contexts are the smaller problem.

The cost curve between model tiers is steep, the quality difference on simple tasks small. For classification, extraction and matching, the smallest model that solves the task is almost always enough. The large models belong where language is genuinely produced or where reasoning runs over several steps.

The second big lever is batching. If you have twenty items rated one by one, you pay for the instruction twenty times, because the system prompt goes along with every call. Batched, you pay for it once. With short payloads and a long instruction, that makes up the bulk of the bill.

Caching works in two forms: provider prompt caching lowers the price for recurring parts of the context considerably, and your own cache in front of the model prevents paying twice for the same question. With recurring requests, the cheapest call is the one that never happens.

What regularly blows up the calculation is repetition. An agent working in a loop produces not one call per task but ten. For multi-step flows a hard upper limit therefore belongs in the code, otherwise the price per operation has no ceiling.

Measure per business unit, not per month: cost per call, per document, per ticket. Only that number tells you whether a use case carries itself. A rising total bill alongside faster rising usage is success, and without a reference figure the two cannot be told apart.

How you notice it

Nobody can say what a single operation costs in model spend.
The model choice was made once and never reviewed since.
There is no upper limit on steps per request.
Items are sent to the model one by one although they would go together.

Not to be confused with

Training cost: A one-off to create or adapt a model. Inference is the running operation and in almost every project the larger item.
Fine-tuning: Costs training once and often a higher inference rate afterwards. It pays off when it lets a smaller model suffice or shortens prompts drastically.
Context window: The technical upper limit for the input, not a price. But every token in it is paid for, which is why a large window is no reason to fill it.

When it fits

Before going live with any AI feature that runs per user or per operation.
When volume is meant to grow: a price per operation scales linearly with it.
With multi-step agents, where one operation produces many calls.

When it does not

For an internal tool with a handful of calls a day, the optimization is not worth it.
As an argument against quality. A model too small that classifies wrongly costs more than it saves.

How to approach it

Measure the price per operation, do not estimate itLog tokens per request and extrapolate with the expected volume. This single number decides whether the use case carries itself, and it is settled before you build.
Start with the smallest modelStep up only when the measurement shows quality is not sufficient. The reverse route is rarely taken, because nobody downgrades a running system.
Batch instead of asking one by oneHave several items rated in one call. The instruction is paid for once instead of twenty times. Size the batch to the rate limit, not to the maximum.
Limit the output lengthForce structured, terse answers. Output tokens are the most expensive part of the bill, and a model allowed to write freely writes long.
CacheProvider prompt caching for recurring parts of the context, your own cache for recurring questions. Both lower the price per operation without losing quality.
Set an upper limit and an alertA hard limit on steps per operation, a budget alert on the model cost. An agent in a loop is the one case where the bill explodes overnight.

Frequently asked

What is the fastest way to reduce inference cost?

In this order: check a smaller model, batch requests, limit the output length, cache. The first two usually deliver the bulk, and none of the four steps requires an architectural change.

Do input and output tokens cost the same?

No, output is regularly several times more expensive. That is why forcing a terse answer structure often has more effect than a shortened context, although intuition suggests the opposite.

Is running your own model in house worth it?

Only from considerable volume onwards, and then with a different cost shape: graphics cards run even when no request arrives. A provider model bills per request and is almost always cheaper under fluctuating load. Where data protection is the driver, the calculation is a different one.

How do you plan cost for an agent?

Not per call but per completed task. An agent makes several calls per task, and the number varies. That is why a hard upper limit on steps and a measurement of the actual distribution belong in the build, not in the review afterwards.