An internal chatbot that answers policy questions and cites its source

Business context

The client is a digital marketing agency with about 60 staff. It grew from 20 people in three years, and its policies grew with it: a leave document here, an expenses sheet there, an IT access guide in a shared drive. New starters and long-serving staff both asked the operations manager the same questions week after week.

We built an internal knowledge base chatbot inside the chat app the team already used. It answers from approved documents only, shows the source for every answer, and hands anything personal to a person.

The problem

Policies were spread across a dozen files, some two years old. Nobody knew which version was current. The operations manager fielded around 25 questions a week about leave, expenses, and tool access, and answered most of them by pasting the same paragraph into a direct message.

New staff were the worst affected. In their first month they had no idea who to ask, so they asked the manager. When she was on leave, the questions sat unanswered. The agency had tried a wiki once. It went stale within a quarter because nobody owned it.

What we built

We connected the chatbot to a small set of documents the operations team owns: leave and holiday policy, expenses, remote work, IT access, and the onboarding checklist. The assistant reads only those files. If a document does not contain an answer, it says so and offers to raise the question with the operations team.

The knowledge base

The documents live in one place and are versioned. When a policy changes, the operations team replaces the file and the assistant picks it up on the next index. There is no separate copy of the policy to maintain, which is what killed the old wiki.

Answering with sources

Every answer ends with a link to the paragraph it came from, with the document name and the date it was last updated. That one detail changed how people used it. Staff stopped asking the manager to confirm the answer, because they could check it themselves in two clicks.

Routing to a person

Anything personal or sensitive goes to a person. That covers pay, performance, complaints, and anything mentioning a named colleague. The assistant replies with a short message and passes the thread to the operations manager in the same chat, with the question attached.

Rollout

• Week 1: Interviewing operations and reading every policy file. Two of them contradicted each other.
• Week 2 to 3: Building the index and the answer flow, then testing it against 200 real questions from the last year.
• Week 4: Opening it to the operations team, then to all staff, with a feedback button on every answer.

Before and after

Step	Before	After
Where policy lives	Scattered files, no owner	One versioned set of documents
Common questions	Direct messages to one manager	Answered in the team chat
Source of truth	Whoever was free	Link to the exact paragraph
Personal questions	Mixed in with the rest	Routed to a person on purpose

Results

In the first month the assistant answered about 120 questions a week. Around 72% of them ended without a person, and most of the rest were personal questions that were meant to be routed. The operations manager estimated she got back five hours a week that she used to spend on repeat answers.

We also learned which policies were the problem. The monthly report of unanswered questions showed that two topics, expenses and equipment, produced a third of all escalations. Rewriting those two documents cut escalations by half in the second month. Related work for a SaaS product, where a support chatbot hands over to Intercom, showed the same pattern: the gaps found are worth more than the answers given.

Lessons learned

• Showing the source document with every answer was the whole reason people trusted it. We treated the citation as a nice extra at first, and almost cut it for time. That would have been a mistake.
• The two policy files contradicted each other on holiday carry-over. The assistant surfaced the conflict by giving different answers to the same question, which forced a decision the agency had avoided for a year.
• Adoption was slower among the longest-serving staff. They already knew the policies and saw no need. The feature that won them over was searching for the exact wording during a dispute, not getting an answer faster.