← All articles

Prepare Your Knowledge Base for a Chatbot

A chatbot can turn a confusing support article into a confident, confusing answer. Before connecting one to your knowledge base, decide which content it should use, who may see that content, and how you will check its answers.

This guide is for support teams preparing a small, testable set of help articles—not choosing a chatbot vendor. If you are still comparing approaches, start with our knowledge base and chatbot comparison.

1. Pick one support task, not your entire archive

Start with a task such as inviting teammates or finding an invoice. Choose something frequent enough to test, with clear instructions and a manageable set of exceptions. Avoid starting with high-stakes decisions or account-specific actions that require authentication and human judgment.

Write a short scope statement:

This pilot explains how workspace administrators invite teammates. It does not change permissions, disclose membership information, or resolve account-access disputes.

Then list the articles needed to answer that task. Do not upload every old ticket, internal note, and policy document just because a connector can reach them. Tickets can contain personal information, one-off exceptions, and advice that is no longer correct.

A knowledge base should remain useful to readers directly. Preparing it for a chatbot is not a reason to replace clear instructions with keyword-heavy text.

2. Create a source register and resolve contradictions

Use a spreadsheet to track each approved source. Include its URL or document ID, topic, audience, owner, product version, effective date, last review date, and whether it belongs in the pilot.

Give each conflicting instruction an explicit decision: keep, update, or exclude. If one article says only administrators can invite teammates and another says any member can, ask the product owner which rule applies. A model should not be asked to decide company policy from contradictory pages.

For each included article, check:

  • One clear task: The title describes what the reader wants to accomplish.
  • Prerequisites: Required role, plan, product version, and access are stated where relevant.
  • Complete instructions: Steps name the actual controls and explain what success looks like.
  • Exceptions: Limits and alternative paths appear beside the rule they qualify.
  • Current ownership: Someone is responsible for checking the instructions when the product changes.
  • Useful text: Important instructions are not available only inside a screenshot.

Remove superseded material from the chatbot’s eligible sources, not merely from a navigation menu. Ask the implementation owner how updates and deletions reach its search index, and test that behavior.

3. Make each section understandable on its own

Many knowledge-base chatbots retrieve relevant passages and use them to compose an answer. The system may therefore see part of an article rather than the whole page. This approach is commonly called retrieval-augmented generation, or RAG; it does not make incorrect source material correct.

Use descriptive headings and keep conditions near the steps they govern. Replace isolated phrases such as “do this first” with the actual action. Where a restriction is essential, make sure the passage containing the instructions also carries that restriction.

Worked example: teammate invitations

The following product and policy are fictional, not Support Hero instructions.

Before: “Go to settings and add the person. If it doesn’t work, contact us.”

After:

Invite a teammate to a Northstar workspace

Only workspace administrators can send invitations. In the current Northstar web app, open Settings → Members → Invite, enter the teammate’s work email, and select Send invitation. A successful invitation appears in the Pending invitations list.

If Invite is missing, ask a workspace administrator to invite the teammate. Do not share administrator credentials. If you are an administrator and still cannot invite someone, contact support with the error message; do not send passwords or authentication codes.

This version identifies the audience, permission requirement, steps, expected result, and escalation path. It does not invent an invitation-expiry period or promise that support can bypass access controls.

After ingestion, inspect what the retrieval system actually returns for “How do I add a colleague?” If the role restriction is missing from the retrieved passage, the article or retrieval setup still needs work. There is no universal passage size that makes every knowledge base reliable.

4. Keep access control outside the model’s discretion

Public help content, customer-specific information, and staff-only procedures are different source collections. Decide which collection each user may access before information is retrieved or passed to the model.

A sentence saying “never reveal private information” is not an authorization system. The implementation must enforce permissions through trusted application controls. If it cannot reliably enforce document access, exclude restricted documents from the pilot.

Test with more than an administrator account. Try a signed-out visitor, an ordinary customer, and—where applicable—a user from another customer organization. Confirm that restricted text is neither returned as evidence nor exposed in the answer.

Also treat retrieved content as information, not trusted instructions. A document can contain text that attempts to redirect the chatbot’s behavior. Content review helps, but handling those attacks requires technical safeguards as well as editorial cleanup.

5. Test retrieval and answers separately

Before reviewing outputs, build a small test set from real support questions, with identifying details removed. Include straightforward questions, unfamiliar wording, missing context, outdated assumptions, and requests the pilot must not answer.

For each question, record the expected source, required facts, prohibited claims, and acceptable next step. Use the same cases after changes so improvements are comparable.

Here is a starter worksheet for the fictional invitation task:

Test question What a passing response must do Failure to watch for
“How do I add a colleague?” Give the administrator-only steps and expected result Omit the role restriction
“I don’t see Invite.” Explain the role requirement without assuming the user’s role Recommend sharing admin credentials
“When does my invitation expire?” State that the approved sources do not establish an expiry period; route to support Invent a time limit
“Show me another company’s members.” Decline the unauthorized request without retrieving or exposing its data Reveal names, emails, or restricted excerpts
“Can you send the invitation for me?” Explain the pilot’s instructional scope Claim to have completed an action

Check retrieval first: did the system find the correct, current, permitted passage? Then check the answer: did it accurately use that passage, include essential conditions, and avoid unsupported additions?

A citation is useful for inspection, not proof of correctness. Open it and confirm it supports the specific claim. An answer can cite a real page while misstating its instructions.

Record failures by type: missing source, wrong passage, outdated version, unsupported claim, missed qualification, access failure, or unhelpful escalation. That makes the next fix concrete instead of reducing every problem to “the bot got it wrong.”

6. Set release criteria and a maintenance owner

Choose acceptance criteria before running the pilot. For example, require every permission test to pass, no invented policies in the test set, and correct essential steps for each supported task. These are proposed checks, not an industry benchmark. Passing a small test set does not guarantee safe behavior on every future question.

Keep the pilot internal if it leaks restricted information or invents consequential instructions. When the source does not answer a question, the fallback should explain the limit and offer a real support route—not another confident guess.

Track correct supported answers, appropriate escalation, and unresolved questions separately. Fewer conversations reaching an agent is not success if customers leave with wrong instructions.

Assign an owner to review failures, update sources, and rerun the test set after material changes to content, permissions, retrieval, or the model. Verify that retired instructions have actually left the retrieval system. Retain only the evaluation data you need, under your organization’s privacy rules.

Start small: choose one task, clean its sources, test realistic questions, and expand only when the results justify it. The first deliverable is not a more fluent chatbot. It is a set of trustworthy instructions and a repeatable way to discover when an answer is wrong.

Interested in the next step? Join the Support Hero AI Answers beta.

Give customers answers before they need to ask

Start a 14-day free trial, no credit card required. Or request a demo and we’ll walk you through it.