The problem
The answer exists in a document somewhere, and the person who needs it asks the one colleague who always knows instead. The pilot that was supposed to fix this answered confidently, could not show its source, and was turned off inside a quarter.
What goes wrong
- It was loaded with everything, so it quoted a file to someone who should never have seen it.
- It answered without a source, so nobody could check it.
- It was loaded once, so it kept answering from the old policy.
The approach
- Set access before loading a single file. The agent answers only from the folders and channels your team shares with it, agreed in writing. Excluded folders and channels are written down and editable any time.
- Retrieve the paragraph, not the gist. Retrieval is tuned on your own questions, and every answer names the document and section it came from, with a link.
- Run a weekly corpus check that picks up new and changed files and flags stale ones.
- Tune on 30 to 50 questions your team actually asks, then run it beside three to five named users before opening it to everyone.
When it works
When your team has written policies, procedures, and manuals that live in shared folders and get asked about in chat all week, and one person owns the list of questions the agent could not answer.
Why these pilots stall
Most teams have tried this once. Someone pointed a chat model at a folder, asked it a question, and got a confident answer that was wrong. Or right, with no way to tell. Three people used it for a month and it was quietly turned off.
It is not the model. It is three things around the model.
It was never told what it is allowed to see
The fastest demo loads everything. That is also the fastest way to get the pilot killed. The first time it quotes the executive pay file to a loan processor, the whole thing is shut down. Retrieval that ignores permissions is a leak with a chat window.
Nobody could check the answers
An answer without a citation is an opinion. Nobody acts on an opinion from a chatbot.
It went stale
Documents change. Policies get revised. The pilot was loaded in week one, and by week six it was answering from last year's credit policy.
How we build it
The method has a name, retrieval-augmented generation (RAG): the agent retrieves the paragraph first, then writes its answer from it.
First comes access. We agree the folder list in writing, and the excluded paths with it: board folders, HR channels, executive files. If a folder was never shared, nothing in it can show up in an answer. The security page covers how we handle your data.
Next is loading and tuning, on the questions loan officers and claims handlers ask every week, not on sample questions.
After that, a few named users run it beside their normal work before it opens to the team.
One rule does more for trust than anything else: when the answer is not in the corpus, the agent says so. It does not fill the gap from general knowledge.
What it looks like when it works
A lending team example. A loan officer asks in Teams what the credit policy allows when a borrower's financial statements are more than a year old. The answer comes back with the paragraph from the credit policy, the date it was last revised, and a link. She did not have to find the document or interrupt the credit officer.
A processor asks the same agent about executive pay. The agent says it has nothing on that, because it does not. The folder is excluded, so the agent never read it.
A claims team at an insurer works the same way: a coverage question comes back with the paragraph from the claims manual and a link.
Every question and answer is logged where your admin can read it. The monthly report lists what was asked, what could not be answered, and which documents were cited most. The unanswered questions become the documents you write next.
When it does not work
It does not work if the documents do not exist. If the answer lives in one senior person's head, write it down first. And it does not work without an owner who reads the unanswered-question list each month and adds what is missing.
Where to start
Start with the Knowledge Agent. It fits any team whose answers live in documents: the credit policy at a bank or lender, the claims manual at an insurer, or the supplier contracts at an electronics manufacturer. If you are not sure documents are the right place to start, ask Roo, Webaroo's AI: Could an AI agent answer my team's questions from our own documents?
Book 15 minutes with Connor