The problem
An AI feature that costs almost nothing in a pilot becomes a monthly line item once the storefront sends it every product question. Nobody sees the bill until the month closes, and by then the spend has already happened.
What goes wrong
- Every request goes to the largest model. A stock check and a disputed return cost the same to answer, and the stock checks are most of the traffic.
- The context grows without anyone deciding it should. Each call carries the whole catalog, the whole returns policy, and the whole conversation history, every time.
- The cap has no owner and the bill has no breakdown. A limit sits in a provider dashboard, nobody is alerted as it approaches, and the invoice cannot say which feature, store, or day drove it.
The approach
- Route by task weight. Small models for lookups and classification, the large model for judgment, set in configuration so the split changes without a rebuild.
- Track model spend as it accrues and attribute it to your engagement, instead of finding it on the invoice.
- Cap spend per workspace and alert the support and maintenance team before the cap is reached.
- Shrink what each call carries. Retrieve the paragraphs that answer the question instead of the whole catalog, and cache the prompts that repeat.
When it works
When the business can name the handful of request types that make up most of its traffic, and one person owns the number: someone who reads the spend report and decides what changes.
Why this is an operating problem
Token spend, what a model provider bills for the text going in and out, is the one cost in an AI build that scales with success. A storefront AI that answers a few hundred questions in a pilot is a rounding error on the bill. The same AI on a holiday weekend, with every shopper asking about shipping cutoffs, is a line item.
The fix is treating spend as a number your team runs, the way you run inventory or ad budget: visible every day, owned by someone, with a ceiling.
Where the money goes
Model choice
Most requests are small: is this in stock, what is the return window. Those belong on a small, fast model. A handful need judgment: a customer disputing a charge, a question that touches three policies at once. Those earn the large model. When everything goes to the large model, you pay the judgment rate for lookups.
Context size
Every call carries context: the catalog, the policies, the conversation so far. Retrieval keeps it small. The call carries the paragraphs that answer the question, not the document they came from.
Repeats
The same system prompt and policy text go out thousands of times a day. Prompt caching stores that repeated part, so each later request reads it at a fraction of the normal input price instead of paying full price again.
How we build it
Cost control is part of the build, not a cleanup after. When usage approaches a workspace's cap, the support and maintenance team brings you the decision: raise it, route more traffic to the small model, or hold.
Roo is the cockpit a company runs its AI crew from. The crew works inside the company's own systems, under its rules. The Brain belongs to the company. Early access February 2027.
A retail example
Illustrative, with example data and no live actions. A home goods store runs AI on its site. In setup, the pod sorts its traffic into four request types: stock and shipping lookups, order status, return policy questions, and everything else, including disputes. The first three go to the small model with retrieval over the policy and catalog text. Disputes and everything else go to the large model. When a promotion lands, the spend report shows lookup volume climbing while cost per request stays flat, and the cap stays where it is.
When it does not work
It does not work when nobody owns the number: a cap without an owner is a shutoff, not a control. Nor does it work if the routing rule is never revisited as models and prices change.
Where to start
If you already run an AI feature and the bill is the problem, the paid assessment delivers an AI inventory, risk tiers, the top 3 workflows with today's numbers and a 90-day plan. Running AI in production covers who watches the spend after launch. The same routing and caps apply to an agent answering quote requests at volume in logistics and freight. For a straight answer first, ask Roo, Webaroo's AI: How do we keep our AI spend predictable as usage grows?
Book 15 minutes with Connor