The question on every first call
Early in the first call, someone asks which model we use. Behind that question is a bigger one: if we pick wrong, how much of this do we throw away?
Providers ship a new model every few months. Last year's pick can become the slower, pricier option while nothing on your side changed. Prices move. Terms move. Models get retired. If your system is wired to one model by hand, every one of those moves is a rewrite, and the rewrite lands on your budget.
So the honest answer to "which model" is short: it should matter less than you think. Here is how we build so that it does.
What is yours and what is rented
Start by splitting the system in two.
Yours: the memory of your organization, the permissions, the approval rules, the logs, the workflows, and the fixed set of real cases you test against. This is where the value lives, and it is specific to you.
Rented: the model. It is a supplier. A good one, but a supplier. Suppliers get replaced when a better one shows up. Your architecture should make that replacement boring.
Most AI builds get this backwards. The business rules go into a long prompt tuned to one model's habits. The conversation history lives inside the provider's own tools. The provider's library gets called straight from application code. Six months later the rented part owns the part that was yours, and switching means starting over.
One layer between your code and the model
Build it so every model call goes through one layer you own: a router. It takes the job and returns the answer in the shape your application already expects. The provider's name, the model version, the prompt version, and the settings live in configuration, not in code.
That one decision changes what a switch costs. A new model becomes a config change plus a test run, not a rewrite.
It also lets you route by task. A refund dispute with an angry customer gets the strongest model. Tagging ten thousand support tickets gets a cheaper tier. Each route is a setting, so when a provider cuts its price or ships a better small model, you move the routine work without a deploy.
The default model is a choice we retest, not a habit. Every candidate model runs against real cases before we adopt it. If the default gets slow, or its terms change in a way your compliance team cannot accept, the route is changed in configuration and the test set says whether the switch is safe.
We pick the model that fits the job.
The test that makes the choice reversible
An architecture that can switch is only useful if you know whether the switch worked. So keep a fixed set of your real cases, names stripped: the invoices that confused the old process, the calls that went wrong, the questions your team asks most.
That set runs against every candidate model before a switch, and again after every version or prompt change. The result is a score your team can read, not a vendor's demo. If the new model does worse on your own work, you do not switch.
That answers the fear behind the vendor question. You are not betting the company on a provider. You are betting it on a test set you own and can rerun any time.
Where Roo comes in
Roo is the cockpit a company runs its AI crew from. The crew works inside the company's own systems, under its rules. The Brain belongs to the company. From early access in February 2027, Roo runs inside every Run client.
What to ask any vendor, including us
- Where does the model name live: in config or in code?
- Can you show a switch, on my cases, with the score before and after?
- What happens to my memory and history if I leave the provider?
- Who reads the data terms before any data flows, and where is that written down?
- What is logged on every call, and can my compliance team export it?
If a vendor cannot answer the first two in plain words, the model is not a setting. It is a dependency, and you are the one carrying it.
The playbooks behind this post
The vendor playbook covers what to test and which terms matter. The architecture playbook covers the layer itself. The paid 10-day assessment is where every engagement starts.
Book 15 minutes with Connor