Most retrieval failures are definition failures

A user asks about a customer. The system returns four documents. Two are about a different legal entity with a similar name, one is about the right company but a closed contract, and one is correct. The team spends the next fortnight tuning chunk size and rerankers.

The retriever is not the problem. The problem is that the word customer means four different things in that organisation and nothing in the system knows which one was meant.

What a definition failure looks like from the outside

Technically related, practically wrong. Everything returned genuinely mentions the term. None of it answers the question, because relevance was computed on words rather than on entities.

Correct answers about the wrong subject. Right document, wrong site. Right invoice, wrong subsidiary. Confident, coherent and useless.

Answers that fluctuate with phrasing. Asking the same thing two ways produces different results, because there is no resolved entity underneath — only text similarity.

Aggregates that do not tie out. Two departments count the same thing and get different totals. The reconciliation exists in an analyst’s head and cannot be given to a machine.

Why tuning does not fix it

Retrieval quality has a ceiling set by how well the underlying things are identified. Better embeddings and cleverer rerankers move you toward that ceiling; they do not raise it.

If the corpus contains three companies called Nordwind and nothing records that two of them are subsidiaries of the third, no ranking function can decide which the user meant. The information needed is not in the text.

The cheap version of the fix

A full ontology programme is not the first step, and comprehensive enterprise ontologies are a well-documented way to spend two years. The useful minimum is narrow.

Resolve the entities that matter. Usually customers, sites, products and contracts. One identity per real thing, with the aliases recorded rather than deduplicated away.

Write down the relationships. Parent and subsidiary, site and organisation, contract and version. Three or four relationship types cover most of the failures.

Attach documents to entities. Not just to folders. Once a document is linked to a resolved customer, retrieval can filter before it ranks, which is where the improvement comes from.

Settle the contested definitions. Where two departments disagree, record both, name them distinctly, and map between them. An average nobody uses is worse than two definitions everybody understands.

The result is usually visible immediately

Filtering to a resolved entity before ranking removes most of the wrong-subject failures in one step. It is a smaller change than reworking the retrieval stack and it addresses the actual cause rather than compensating for it.

The test worth running first: take twenty recent bad answers and classify them. Wrong subject, or right subject and poor passage? In most systems we have looked at, the first pile is much larger, and it is the pile that no amount of retriever tuning will shrink.

That is what an ontology design and engineering engagement starts with — narrow, tied to one or two real use cases, and pointed at something that consumes it within weeks, because a model nothing uses is a document that decays.