Controlling AI Costs: Token, GPU, and Inference Economics

The bill nobody modeled

AI pilots are cheap. AI in production is not, and the surprise usually arrives on the second or third invoice, when a feature that cost a rounding error in testing is scaling into a real line item. The cause is almost always the same: nobody modeled the unit economics before shipping, so cost grew silently with usage while everyone was watching quality.

Controlling AI cost is a solvable engineering and FinOps problem, but only if you understand where the money actually goes. There are two spending models — paying a provider per token through an API, and paying for GPU capacity you run yourself — and each has its own levers. This article walks the drivers of both and the tactics that move them, without sacrificing the quality that justified the project.

Token economics: what you are actually paying for

With a hosted model, you pay per token — both the tokens you send (input) and the tokens the model generates (output), usually at different rates, with output typically the more expensive of the two. Every cost lever traces back to one of three quantities: how many tokens go in, how many come out, and which model prices them.

• Input tokens are driven by prompt size: system instructions, few-shot examples, and — the usual culprit — retrieved context and conversation history stuffed into every call. Long RAG passages and full chat transcripts replayed on each turn are where input cost quietly balloons.
• Output tokens are driven by how much you ask the model to generate. Verbose formats, unnecessary explanations, and unbounded responses all cost money by the word.
• Model choice sets the per-token rate, and the spread between a frontier model and a small one is large.

The first discipline is simply measuring these per request and per feature. You cannot control what you have not instrumented, and most cost overruns are invisible until someone attributes spend to a specific prompt.

The highest-leverage tactics

A handful of techniques do most of the work. In rough order of impact:

• Right-size the model per task. The single biggest lever. Do not run a frontier model on a task a small one handles. Route each request to the cheapest model that meets the quality bar — a classifier to a small model, open-ended reasoning to a large one — with a fallback to escalate only when needed.
• Cache aggressively. Many prompts share a large, stable prefix — the same system prompt and instructions on every call. Prompt caching lets the provider reuse that computation at a steep discount. And cache results: identical or near-identical queries should return a stored answer, not a fresh model call. For repetitive workloads, response caching can remove a large fraction of calls outright.
• Trim the context. Retrieve fewer, better passages instead of stuffing the whole knowledge base into the prompt. Summarize and compress conversation history rather than replaying it verbatim. Better retrieval cuts cost and improves quality at the same time.
• Bound the output. Set token limits, ask for concise or structured responses, and do not generate prose a downstream system will discard.
• Batch where latency allows. Offline and asynchronous workloads — nightly classification, bulk extraction — can use batch processing tiers that cost substantially less than real-time calls.

None of these trade away quality. Trimming bloated context and caching stable prefixes generally improve the system while cutting the bill.

GPU economics: when you run the model yourself

Self-hosting flips the model from per-token to per-hour: you pay for GPU capacity whether or not it is busy, so the entire game becomes utilization. An idle GPU is pure loss.

• Utilization is everything. A GPU serving a handful of requests an hour is burning money. Consolidate workloads, and scale capacity to real demand instead of provisioning for a peak that rarely arrives.
• Batching and concurrency. Modern inference servers raise throughput dramatically by batching concurrent requests and reusing computation across them. The right serving stack can multiply the requests one GPU handles.
• Quantization. Running a model at lower numerical precision cuts its memory footprint and speeds inference, often letting a model fit on smaller or fewer GPUs with modest quality loss. This is one of the highest-leverage self-hosting tactics.
• Right-size the hardware to the model. A small specialized model may not need premium accelerators at all. Match the GPU to the workload rather than defaulting to the biggest available.

The crossover math matters: hosted APIs win at low and spiky volume because you pay only for what you use; self-hosting wins at high, steady volume where you can keep expensive hardware busy. Do the arithmetic at your real throughput before committing either way.

Treat AI spend as a FinOps discipline

Tactics without governance drift back into waste. AI cost needs the same operational practices you already apply to cloud spend.

• Attribute cost to features, teams, and use cases. Tag and track spend so you know which application drives the bill. Unattributed cost never gets optimized because no one owns it.
• Set budgets and alerts. A runaway loop, a prompt-injection abuse pattern, or a viral feature can spike spend overnight. Caps and alarms catch it before the invoice does.
• Rate-limit and quota per consumer. Protect against both accidental and malicious over-use, especially on any endpoint exposed to users.
• Route policy through one control point. An AI gateway centralizes model routing, caching, and rate limiting — and the same enforcement point that redacts sensitive data can meter and cap spend. The intSignal AI Gateway exists to make policy, privacy, and cost control one layer rather than three scattered concerns.

This is FinOps applied to a new resource, and it belongs beside the rest of your platform, not bolted on — one reason cost governance runs through the broader intSignal product approach rather than a standalone tool.

Where to start

Instrument first: attribute current spend to specific features and find where the tokens or GPU-hours actually go. In almost every case the top finding is the same — an expensive model doing a job a cheaper one could, and stable context paid for on every call instead of cached. Fix those two, add budgets and alerts so nothing spikes unseen, and revisit the API-versus-self-host math only once your volume is steady enough to justify it.

AI cost is not an inevitable tax on using the technology; it is an engineering variable you control. If you want help modeling your unit economics and building the routing, caching, and governance to keep spend in line with value, talk to our team.