All terms

Glossary

Inference cost

Also: Token cost

The running cost per request to a language model, billed by tokens in and tokens out.

Unlike classic software, cost grows with usage rather than with user count. A flow with five model calls costs five times, even when the same person triggers it.

The levers are model choice per subtask, the length of supplied context, caching of recurring requests, and the question whether a model is needed at all.

How you notice it

  • Cost per transaction was not calculated before rollout.
  • A flow calls the model several times without caching.
  • The context contains documents the question does not need.

Frequently asked

How do I reduce the cost of an AI workflow?

In this order: shorten the context, cache recurring requests, pick a smaller model for simple subtasks, and check whether a step needs a model at all. Model choice delivers least and gets discussed most.

Where this comes up in my work

AI in production

The infrastructure behind AI systems that actually ship: MCP servers, controlled tool access, LLM integration with real permissions and cost control.

To the service

Reducing AWS costs

I find where your AWS budget leaks away, and cut it measurably without giving up performance or availability.

To the service

Related terms

Context windowRAG