The answer was not wrong. The document was three months old

A support system had been answering questions about a returns policy for six weeks. The answers were clear, well sourced and consistent. They also described the policy as it stood before a change in March, because the ingestion job that pulled the policy documents had been failing silently since then.

Nothing crashed. No alert fired. The answers were internally coherent and cited a real document. The only signal was a slow rise in complaints that nobody connected to anything.

Why freshness fails silently

A pipeline that fails still leaves data behind. Yesterday’s index is still there and still answers. Absence of new data looks exactly like a quiet week.

Nobody watches for nothing happening. Monitoring alerts on errors and on volume spikes. A job that stopped running produces neither, and a job that runs and silently loads zero rows produces even less.

The corpus has no visible age. Users see an answer with a source. They do not see that the source was last updated in February, and no interface tells them.

Deletions are invisible. A withdrawn document usually stays in the index. The system keeps answering from a policy that no longer exists, which is worse than not answering at all.

The checks that catch it

Cheap, and largely absent from systems we are asked to look at.

Freshness thresholds per source. This source must have loaded within twenty-four hours. Not loading is an alert, exactly like an error. This single check catches most of these incidents.

Row and document counts with expected ranges. A load that returns zero, or ten per cent of normal, fails loudly rather than proceeding.

Deletion propagation. Something that removes documents from the index when they are removed at source, verified rather than assumed.

Embedding recency. In retrieval systems the embedding can be stale even when the document is current, if re-embedding was skipped after an update.

Age surfaced in the answer. Show when the source was last updated. It costs a line of interface and it converts a silent failure into something a user reports on the first day.

The test that finds it today

Take the five sources your system depends on most. For each, ask when it last successfully loaded and how you would know if it stopped.

If the answer to the second question is that somebody would eventually notice the answers getting strange, that source has no freshness monitoring, and the incident described above is already possible in your system.

Most of what we do in an AI data engineering engagement is this kind of unglamorous plumbing: checks that fail loudly, lineage that says where an answer came from, and age made visible. It is far less interesting than the model and it accounts for a much larger share of whether the system can be trusted.