What an AI answer really costs to produce

Training makes the headlines. Inference, answering each request, is the bill that repeats. Here is where the money goes and how teams cut it.

By Team Inspiropedia

What an AI answer really costs to produce
Photo captionPhoto credit

Key points

  • Training is a large one-off cost per model version; inference is paid on every request.
  • Cost follows the tokens processed and the chip memory and time each request occupies.
  • Batching, caching, smaller models and routing cut the bill without changing the product.

When a chatbot answers a question, a data-centre chip does real work: it reads your prompt, then writes the reply one small piece of text at a time. That work is called inference, and unlike training it happens again for every request. For a product with millions of users, it becomes the main running cost.

Tokens are the unit of billing

Language models read and write tokens, short chunks of text, often a few characters or part of a word. Most providers price per million tokens, and output tokens usually cost more than input tokens because each one has to be generated in sequence. Longer prompts, long chat histories and wordy answers all add tokens, so they all add cost.

Why generation is slow and expensive

Producing each new token means moving the model’s weights and a running memory of the conversation through the chip. In practice the limit is often how fast data can travel from memory, not how fast the chip can multiply numbers. That is why memory capacity and bandwidth matter so much in AI hardware.

The model also keeps a “KV cache”, stored intermediate results for the text so far, so it does not recompute the whole conversation for every new token. The cache saves computation, but it grows with the length of the conversation and takes up scarce memory.

How teams cut the bill

  • Batching. Serving many requests together keeps the chip busy. The trade-off is a slightly longer wait for each user.
  • Caching. Several providers let you reuse a repeated prompt prefix, such as a long set of instructions, at a lower price.
  • Quantisation. Storing weights at lower numerical precision, for example 8-bit instead of 16-bit, shrinks memory use, with a small quality cost that has to be tested.
  • Smaller or distilled models. A compact model trained to imitate a larger one can handle routine tasks at a fraction of the cost.
  • Routing. Easy requests go to a cheap model; hard ones escalate to a larger one.

What to measure

Cost per token is a supplier’s number. What a business needs is cost per successful task. If a cheaper model needs two attempts, or a person has to fix the output, the saving can vanish. Teams that track task success alongside spend usually make better trade-offs than those chasing the lowest price per token.

The metric that matters is not the cheapest token. It is the cheapest correct answer.

More in Artificial Intelligence

Keep reading

All news
SearchSubscribe