Building an Internal Knowledge Assistant on Your Own Docs

The demo always works. You paste in twelve markdown files, wire up an embedding call, ask “how do we deploy staging?” and the thing answers perfectly. Then you point it at 4,000 pages of real internal documentation, half of which contradicts the other half, and it confidently tells a new hire to use a deployment script that was deleted in 2023.

We’ve built an internal knowledge assistant three times: once for our own studio docs, twice for clients with support teams drowning in Notion. The first attempt was a weekend of work and got abandoned in a month. The third took six weeks, most of which had nothing to do with the model, and it’s still in daily use. The difference wasn’t the LLM. It was retrieval quality, honest refusals, and a feedback loop that told us which answers were wrong.

This is the version I’d want to read before starting. It assumes you can write SQL and ship a Node or Python service, and that you’ve already skimmed the “build a RAG chatbot in 50 lines” posts and noticed they stop exactly where the hard part begins.

Build the eval set before you build the pipeline

This is the step everyone skips, and it’s the one that decides whether the project survives. Sit with the people who’ll actually use the thing (support, onboarding, whoever answers the same Slack question every week) and collect real questions. Not questions you invent. Real ones, in their real phrasing, including the badly worded ones.

For each question, record the document or section that contains the answer. That’s your ground truth. Now you can measure two things separately, which matters enormously:

• Recall@k: did the correct chunk appear in the top k retrieved results? This is a retrieval problem and it’s where 80 percent of your bugs live.
• Answer correctness: given the right context, did the model produce a correct, cited answer? This is a prompt and model problem.

Keep them separate. If recall@10 is 0.55, no amount of prompt engineering will save you. You’ll waste a week fiddling with system prompts because that’s the fun part. Ask me how I know.

Also include about ten questions your docs genuinely cannot answer. The correct behaviour is a refusal. If your assistant answers those confidently, it’s hallucinating on the rest too. You just haven’t caught it yet.

Chunking: where most internal chatbot docs projects quietly break

Fixed-size chunking with a 200 character overlap is the default in every tutorial. It’s also usually the wrong choice for documentation. Docs have structure. Use it.

Split on headings, keep each section together if it fits, and only fall back to paragraph splitting for sections that run long. Crucially, prepend the heading path to the chunk text before embedding. A chunk that starts “Deployment > Staging > Environment variables” carries context that a bare paragraph does not. It’s the single cheapest retrieval improvement available.

import { fromMarkdown } from 'mdast-util-from-markdown';
import { toString } from 'mdast-util-to-string';

const MAX_CHARS = 2200;   // ~500 tokens, comfortable for most rerankers
const MIN_CHARS = 400;    // below this, merge into the next section

export function chunkMarkdown(md, { docTitle, url }) {
  const tree = fromMarkdown(md);
  const sections = [];
  let current = { path: [docTitle], body: [] };

  for (const node of tree.children) {
    if (node.type === 'heading' && node.depth  {
    const heading = s.path.join(' > ');
    const text = s.body.join('\n\n').trim();
    if (!text) return [];
    return splitLong(text, MAX_CHARS).map((part, i) => ({
      url: url + '#' + slug(s.path.at(-1)),
      heading_path: heading,
      // the embedded text includes the breadcrumb; the displayed text does not
      embed_text: ${heading}\n\n${part},
      content: part,
      part_index: i,
    }));
  }).filter(c => c.content.length >= MINCHARS || c.partindex > 0);
}

Two things worth knowing. Tables and code blocks should never be split mid-structure: a truncated SQL example is worse than no result. And if your source is Confluence or Notion, export to markdown first and normalise it. Their HTML exports produce div soup that destroys any heading-based strategy.

One trade-off worth flagging: heading-based chunks vary wildly in size, from 300 to 2,200 characters. That’s fine for retrieval, but it means your token costs per query aren’t predictable. Cap the total context you assemble, not the individual chunk.

Hybrid retrieval, because embeddings can’t spell

Semantic search is excellent at matching “how do I roll back a bad release” to a section titled “Reverting deployments”. It’s terrible at “ERRMODULENOT_FOUND in the build step” or “does v4.2 support RTL”. Exact tokens, version numbers, error codes and product names are where dense vectors go vague.

You don’t need Elasticsearch. Postgres with pgvector plus a tsvector column covers it, and for corpora under a few hundred thousand chunks it’s fast enough that the network hop to a dedicated vector DB costs you more than the search does.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
  id           bigserial PRIMARY KEY,
  space_id     text NOT NULL,          -- permission boundary
  url          text NOT NULL,
  heading_path text NOT NULL,
  content      text NOT NULL,
  updated_at   timestamptz NOT NULL,
  embedding    vector(1536) NOT NULL,
  tsv          tsvector GENERATED ALWAYS AS (
                 totsvector('english', headingpath || ' ' || content)
               ) STORED
);

CREATE INDEX ON docchunks USING hnsw (embedding vectorcosine_ops)
  WITH (m = 16, ef_construction = 64);
CREATE INDEX ON doc_chunks USING gin (tsv);
CREATE INDEX ON docchunks (spaceid);

Then fuse the two result sets with Reciprocal Rank Fusion. RRF ignores raw scores (which aren’t comparable between cosine distance and tsrankcd) and works purely on rank position, with a constant k of 60 that comes from the original paper and that almost nobody needs to tune.

WITH vec AS (
  SELECT id, RANK() OVER (ORDER BY embedding <=> $1) AS r
  FROM doc_chunks
  WHERE space_id = ANY($3)
  ORDER BY embedding <=> $1
  LIMIT 50
), kw AS (
  SELECT c.id, RANK() OVER (ORDER BY tsrankcd(c.tsv, q) DESC) AS r
  FROM docchunks c, websearchto_tsquery('english', $2) q
  WHERE c.tsv @@ q AND c.space_id = ANY($3)
  ORDER BY tsrankcd(c.tsv, q) DESC
  LIMIT 50
)
SELECT c.id, c.url, c.heading_path, c.content,
       COALESCE(1.0 / (60 + vec.r), 0) + COALESCE(1.0 / (60 + kw.r), 0) AS score
FROM doc_chunks c
LEFT JOIN vec ON vec.id = c.id
LEFT JOIN kw  ON kw.id  = c.id
WHERE vec.id IS NOT NULL OR kw.id IS NOT NULL
ORDER BY score DESC
LIMIT 40;

Note websearchtotsquery rather than plainto_tsquery: it tolerates quotes and stray operators from real user input instead of throwing. On our corpus, adding the keyword arm was the single largest recall improvement of the whole project. The queries it rescued were exactly the ones support staff ask most: specific names, specific errors, specific versions.

If you use a 3,072-dimension model, note that pgvector’s HNSW index caps vector at 2,000 dimensions. Cast to halfvec for the index or drop to a 1,536-dimension model. The accuracy difference on documentation retrieval is small. The operational simplicity is not.

Rerank before you generate

Retrieve 40 candidates, rerank with a cross-encoder, keep the top 6 to 8. A cross-encoder reads the query and the chunk together rather than comparing two independently computed vectors, and it’s dramatically better at judging relevance. Cohere Rerank, Voyage’s reranker, or a self-hosted bge-reranker-v2-m3 all work. Expect 200 to 400ms of added latency for a batch of 40.

Worth it? On our eval set, reranking moved more questions into “correct answer, correct citation” than upgrading the embedding model did, and it cost less. If you only have budget for one addition beyond hybrid search, make it this.

Then assemble context deliberately. Deduplicate chunks from the same section, order them oldest-document-last so recency wins ties, and hard-cap the assembled context at something like 6,000 tokens. Stuffing 30 chunks into a 200k window is not free: the model gets worse at identifying the relevant one, and your per-query cost triples.

Answers, citations and knowing when to shut up

The prompt matters less than people think, with two exceptions: citation format and refusal behaviour. Both need to be explicit and both need to be verified in code, not trusted to the model.

const system = `You answer questions using ONLY the numbered sources below.

Rules:

Every factual sentence must end with a citation like [3].
If the sources do not contain the answer, reply exactly:
  "I couldn't find this in the docs." Then list the 2 closest sources.

If sources conflict, say so and cite both. Prefer the more recent one.
Never infer configuration values, credentials or commands not present in the sources.

Sources:
${chunks.map((c, i) => [${i + 1}] ${c.headingpath} (updated ${c.updatedat})\n${c.content}).join('\n\n')}`;

Then post-process: parse every [n], drop any that’s out of range, and if a response claims a fact with zero valid citations, flag it rather than displaying it. We render citations as links straight to the anchor in the source doc, and roughly a third of users click through instead of reading the answer. That’s a feature. The assistant’s real job is often navigation, not authorship.

Include the updated_at date in the context and in the UI. Stale documentation is the number one cause of confidently wrong answers, and showing “source last updated 14 months ago” transfers the judgement back to a human who can actually make it.

Permissions, freshness and the unglamorous half

Access control belongs in the WHERE clause. Resolve the user’s accessible space IDs from your identity provider on every request and pass them as a parameter, as in the query above. Do not tell the model which documents the user may see. Prompt-level permission is not permission.

For freshness, run incremental indexing off webhooks where you can (Notion, Confluence and GitHub all emit them) and a nightly full reconciliation to catch deletions, which webhooks miss constantly. Store a content hash per chunk and skip re-embedding unchanged text. Embedding is cheap at roughly two cents per million tokens for small OpenAI models at current pricing, but re-embedding 4,000 chunks every night for no reason is still sloppy.

Deletion is the failure that bites hardest. A page gets archived, nobody reindexes, and six months later the assistant is still quoting a policy that no longer exists. Reconcile by source ID nightly and hard-delete orphans.

Feedback is the product

Log every query, the retrieved chunk IDs, the final answer and a thumbs up or down. Review the down-votes weekly. In our experience most negative feedback isn’t a model failure at all: it’s a documentation gap, and the assistant just made it visible for the first time. That log becomes a prioritised backlog for the docs team, which is arguably worth more than the chatbot itself.

If your docs are on a static site and you don’t want to build a backend just to collect ratings, an endpoint like WebForms will take the POST and push it into Slack where people will actually read it. We’ve done exactly that for two client documentation sites.

When not to build this

If your documentation is under about 200 pages and well organised, a good search box and a well-maintained index page will outperform a knowledge base AI on both accuracy and cost. Genuinely. We’ve talked two clients out of this project.

If your docs are contradictory, undated and scattered across four tools, an assistant will amplify the mess rather than paper over it. Fix the corpus first. The work of consolidating and standardising documentation pays off with or without an assistant on top: it’s the same discipline behind maintaining a reusable block pattern library, where the value comes from consistency, not from the tooling.

And if the answer to most questions is “it depends on the customer’s account state”, you don’t need retrieval over documents. You need a tool-calling agent with read access to your systems. Different project, different risks, much longer timeline.