Retrieval is not enough

What I tested, what came out, and what I think has to be done Anton Schelepov — Commercial Intelligence / Data Engineering — 30 July 2026

1. What I did

I took two things: a live internal AI-tooling service — recent, actively used, with maintained Confluence documentation — and a set of questions about that service that I collected from Google Spaces. Then I tested one narrow question:

Does the documentation actually contain the answers people ask for, and does search find them?

  • Corpus: the complete documentation tree for that service.
  • Questions: nine questions taken verbatim from our own Google Spaces. Four from general channels, five from the service’s own user group where the maintainers are present. One asked in German.
  • Method: a simple retrieval implementation — chunking plus two embedding models, one English, one multilingual. Two separate measurements: is the answer present in the text at all (verifiable with grep, model-independent), and does semantic search surface it.
  • Discipline: every expected outcome written down before the run. Questions used exactly as typed.

2. Results

All nine questions were run against the complete documentation. The categories below overlap — one question can be both unanswered and answered wrongly:

  • Five had no answer in the documentation at all. Not badly worded, not hard to find — absent. Including where model traffic is processed and under what data-retention terms, and whether a specific commercial licence may be used through the service. First-order facts about where company data goes existed only as chat messages.
  • Two were fully answered and retrieval found them.
  • Two came back confidently wrong. Retrieval returned a real, relevant-looking, correctly-citable passage that answers a different question. In one case that wrong passage ranked first of 71 chunks, at one of the highest similarity scores in the entire run — and a second page repeats the same misleading recommendation, so the corpus pushes the wrong answer twice.
  • Two required a person, not a document. One answer was a permission grant. One was a question about future intent — documentation describes current state; intent lives in the maintainer’s head.
  • Two structural findings. Nearly half the indexed content was configuration code, which degraded the top result on five of nine questions. And the documentation is entirely English, while one of the nine questions was asked in German.

Two things the questions themselves showed, which no metric captures: one asker had already read the documentation and it was still insufficient to decide. And colleagues were already routing questions to named individuals by hand — one thread’s first reply was simply “@[name] could you provide any info?”, followed by another participant noting they had asked the same person the same thing in a private message. Same question, two askers, one answer, given where it helped nobody else.

3. The main point: retrieval is not enough

The instinct on seeing “five of nine unanswered” is write better documentation. That instinct is not wrong, but it does not reach the problem.

Better documentation and better search fix four of these nine cases. They cannot fix the other five, because there is nothing to find.

Wrong answers, provided with confidence, are the ones that should worry the most, because they are not a documentation-quality problem. The offending passage was real, well-written, current, topically adjacent, and correctly cited. It is documentation doing its job — although it answers a different question than the one asked. Better documentation would still have ranked it first.

Our company has many departments, different documentation cultures, different vocabularies for the same thing. Documentation quality is uneven by construction, not by neglect. A retrieval layer that assumes a complete, consistent, single-language corpus is assuming away our actual conditions.

4. The source of truth is still people

In every one of the five unanswered cases, an answer existed. It was given by a named person, in a Space, a chat, or a private meeting — within minutes, correctly, and then forgotten, until the same exact question is asked again.

The company’s knowledge is not primarily in Confluence. It is in people, and it surfaces conversationally, in response to a question that is relevant to them. Documentation is the small, lagging, partially-written subset.

The people side already works reasonably well. Colleagues route questions to the right individual by hand, and those individuals answer. What fails is memory — nothing captures the answer, so the same question is asked and answered indefinitely, and half of it happens in private messages where the answer helps exactly one person.

The conclusion I draw is not “get people to document more.” It is: meet the answer where it is already given, and capture it there.

5. What I think we need

Three capabilities on top of retrieval. Each addresses a specific failure above; each is separable.

The proposed flow: retrieval feeds a sufficiency check; sufficient results return an attributed answer, insufficient ones abstain and name the nearest person, whose answer is captured and — after human approval — written back into the documentation.

Orange marks the two steps that require a person. Dashed lines are the feedback path: everything captured re-enters the index, which is what makes coverage grow with use rather than stay fixed at whatever was pre-loaded.

a. Abstain, then name a person. When retrieval cannot answer, return the person whose recent work is closest to the question — not silence.

b. Index the conversational channels as a first-class source. Our Spaces and chats already contain answers to questions the documentation does not cover — including, in this test, the exact answers to cases the docs missed entirely. Indexed as attributed, dated, searchable records, they close most of the gap immediately.

c. A sync path from channel to documentation. For each channel, a mechanism that identifies newly-answered questions and proposes an update to the matching documentation, with an LLM applying our documentation conventions in between:

  • Human approval before anything lands.
  • Append to clearly-matching pages only. In this test the corpus already contained a passage that contradicts the correct answer to one question — and a second page repeating the same misleading recommendation. An appending LLM produces a page asserting both. Contradictions get flagged for a human, never resolved automatically.
  • No auto-creation of pages. Where to put a new page, and who owns it, are judgment calls.

We are leaving Google Workspace by the end of 2026, and whatever replaces it will need exactly this capability. The Google Spaces history is already scheduled for export; the point here is that once exported, it should be indexed and written into the system rather than archived and forgotten.

What this test does and does not establish

It demonstrates mechanisms, on nine questions about one service. The retrieval stack was deliberately simple: small embedding models. A production stack would do better on the documented cases. It cannot do better on the absent ones.

The one genuinely unvalidated assumption is human, not technical: whether a person actually responds when a question is routed to them. For myself I can say: I am not thrilled about answering the same question over and over again.

Full test, corpus, and reproduction scripts available. Question catalogue with complete threads held separately.