When teams budget an AI feature, they price the model API and stop. Then the feature ships, and the real cost structure reveals itself: the engineering around the model, the evaluation machinery, the monitoring, and a maintenance tail that behaves nothing like normal software. None of this argues against building. It argues for budgeting honestly, because features priced on tokens alone get cancelled in month six when the rest of the bill arrives.
Inference: variable, and shaped by design choices
Unlike most software costs, inference scales with every single use, and the per-request price is a product of decisions you control: which model tier, how much context you stuff into each call, how long the outputs run, how many retries and chained calls sit behind one user action. A feature that assembles generous context and double-checks itself with a second model call can cost several times its naive estimate. The design levers matter more than the rate card: route easy cases to smaller models, trim context to what the task needs, cache what repeats, and cap output length where the product allows.
Latency is a cost too
Model calls are slow relative to everything else in your stack, and chained calls multiply the wait. Latency does not appear on an invoice, but it shows up as abandoned features and workflow friction, and engineering around it, streaming, precomputation, moving work off the interactive path, is real project scope. Fast-enough often decides the model choice as much as smart-enough.
The costs nobody puts on the first slide
- Evaluation: building the eval set, grading machinery, and the human review hours to keep the grader honest. This is the price of knowing the feature works.
- Monitoring and logging: tracing every interaction, dashboards for quality signals, storage and retention handled per your privacy obligations.
- The human loop: if outputs are reviewed before use, that review time is an operating cost of the feature and belongs in its business case, along with the plan for shrinking it as measured quality improves.
- Guardrails and security: input validation, injection resistance, output filtering, and the incident process for when something slips through.
- Vendor churn: models are deprecated and replaced on a cadence traditional software does not prepare you for. Each transition costs a re-evaluation run and sometimes prompt rework.
The maintenance tail
Normal software degrades when you change it. LLM features can degrade when you change nothing: usage drifts toward inputs the eval set never covered, source content goes stale under a retrieval system, a provider updates behavior. Budget a standing fraction of the build effort, not zero, for the ongoing loop of reviewing quality signals, refreshing evals, and tuning prompts. Teams that plan this keep their features healthy; teams that do not quietly watch adoption erode.
Budgeting that survives contact with reality
- Model the unit economics: cost per assisted task at expected volume, including retries and chained calls, against the value of the minutes saved or quality gained per task.
- Set a per-feature cost budget and alert on it, attributed through a gateway so each feature's spend has an owner.
- Plan optimization as a phase: ship with a capable model to prove value, then engineer the cost down with routing, caching, and context discipline once real usage shows where the money goes.
The pattern to internalize: the API bill is usually the smallest honest line item, and the feature's economics are decided by design choices and operational discipline, both of which are in your control. Priced honestly, good AI features still clear the bar comfortably. Priced on tokens alone, even great ones become budget surprises.