The problem
An agent that passed every test in week three can be wrong in week nine, and nothing on the screen looks different. The model updates, the schedule template changes, a new insurance plan starts showing up in calls, and the first person to notice is a patient who never got booked.
What goes wrong
- Nobody measures after launch. The demo worked, the team moved on, and the first report of a problem comes from a customer or a patient, weeks late.
- The wrong things are watched. Uptime is green and response time is fine while the agent books the wrong visit length or stops asking for the insurance plan.
- Changes ship untested. A model update, a prompt edit, or a new rule goes live without being run against the cases that used to work.
The approach
- Build an eval set from real cases: recorded calls, real forms, real tickets, each with the right answer written down. Re-run every case after every change. If the score drops, the change does not ship.
- Watch the job, not just the server, every day: booked rate, handoff rate, intake completeness, and errors, posted to your Slack each morning with an alert when one moves outside its usual range.
- Keep a friction log. Every correction from the review queue and every 'it should have done this' from your staff goes in. The support and maintenance team reviews it monthly and turns each entry into a rule, a fix, or a new eval case.
- Report monthly on one page: what the agent handled, what it handed off, what changed, and what the eval set says now against last month.
When it works
When there is a number your team already cares about, such as visits booked or intakes completed, a fixed set of real cases with known answers, and a named person on each side who reads the daily check and the monthly report.
A green dashboard is not enough
Most monitoring answers one question: is it up. For an AI agent that is the least useful question. A clinic phone agent can be up all week while it books thirty-minute slots for visits that need forty-five, or stops asking which plan the patient carries because a prompt edit dropped the line. Nothing crashes. The schedule just gets worse.
So we measure two things: whether the agent still does the job, and whether the last change made it better or worse.
Evals: the fixed set of real cases
An eval is a test the agent has to pass again every time something changes.
Example: for a clinic phone agent, sixty recorded calls. New patients, reschedules, a refill question that should go to a nurse, a caller who gives the wrong date of birth twice, a plan the clinic does not accept. Each one has a known correct outcome: the slot that should be booked, the fields that should be captured, the point where a person should take over.
The whole set runs again after a model update, a prompt edit, a new rule from the friction log, or a new provider on the schedule. The set grows every time the review queue catches something new, so a case that failed once stays in the set for good.
Monitoring: the daily check
Example, for the same clinic: a nightly job counts yesterday's calls. How many ended in a booking, how many went to a person and why, how many intakes came back complete, and how many errors were logged. It posts the numbers to your Slack by 7 a.m. and flags anything outside its usual range. A handoff rate that doubles overnight means a rule broke or something new started coming in.
Response time and error triage still run underneath. They tell us the system is alive. The daily check tells us it is useful.
The friction log
The most valuable artifact after launch is a plain list. Every correction your staff makes in the review queue, every 'it should have asked for the referral,' every call that should have gone to a person sooner. One place, written by the people who see it.
What a month looks like
Daily: the check posts to Slack.
After every change: the eval set runs, the score is compared, and the change ships or waits.
Monthly: one page. What the agent handled, what it handed off, what changed, the eval score against last month, and what the friction log says to fix next.
When it does not work
It does not work if nobody on your side reads the daily check. Then the alert is noise.
It does not work when the eval set is made up. Synthetic cases pass synthetic tests. The set has to come from real calls and real forms, with answers written by someone who does the job.
Where to start
Our dedicated support and maintenance team runs monitoring and evals every month on every agent the pod ships. The pod's first measurement comes at its production checkpoint: Day 90: what a production checkpoint means. If you are not sure which agent to build first, ask Roo, Webaroo's AI: Which AI agent should we build first, and how would we measure it?
Book 15 minutes with Connor