--- title: "The model was fine. The data was three weeks old and missing the exceptions" url: "https://predictes.com/services/ai-data-engineering" description: "Pipelines, quality and lineage built for AI workloads: getting documents and records into a form a model can use, with the freshness and trust guarantees a system needs." --- # The model was fine. The data was three weeks old and missing the exceptions Almost every AI quality problem traced far enough turns out to be a data problem: stale, incomplete, inconsistent, or missing exactly the cases that matter. This service builds the pipelines, checks and lineage that let a system be trusted — for documents and unstructured content as much as for tables. - **Pipelines that run and are watched.** With alerting on freshness and volume, so a silent failure is caught before an answer is wrong for a fortnight. - **Documents in a usable form.** PDFs, scans, email and files parsed, chunked and enriched so retrieval finds the right passage rather than a plausible one. - **Quality you can state.** Completeness, freshness and consistency measured and published, so trust is calibrated rather than assumed. - **Lineage.** For any answer, where the underlying data came from. The first serious dispute will require exactly this. [Talk to us about this](https://predictes.com/contact) ## What is delivered - ### Ingestion From your systems, your document stores and the awkward sources — a shared drive, an inbox, a supplier portal — with incremental loads and replay rather than nightly full refreshes. - ### Document processing Parsing, layout handling, table extraction, OCR where needed, and chunking tuned to your documents rather than to a default that works on articles. - ### Quality checks Freshness, volume, distribution and referential checks running on every load, failing loudly. The cheapest reliability improvement available in any AI system. - ### Enrichment and linking Entity resolution against your model, so a record and a document about the same customer are known to be about the same customer. - ### Lineage and cataloguing Where each field came from, what transformed it and when it last changed — queryable rather than documented in a wiki. - ### Cost and retention Storage tiering, embedding cost control and a retention policy that satisfies both your finance team and data protection. ## How it runs 1. 01 ### Profile what exists Actual completeness, actual freshness, actual weirdness. This step routinely changes the plan, because the documented state and the real state differ. 2. 02 ### Build one path properly One source end to end with checks and lineage, rather than five sources without. The pattern then repeats cheaply. 3. 03 ### Measure quality Publish the numbers, including the uncomfortable ones. A quality metric nobody sees is a quality metric nobody acts on. 4. 04 ### Hand over Your team running it, with runbooks for the failures that will happen and documentation written during the work. ## A good fit when - Retrieval returns things that are almost right, and nobody knows why. - An AI feature works on the demonstration set and degrades on real data. - The knowledge you need is in PDFs, scans and email rather than in tables. - Nobody can say how fresh the data behind an answer actually is. ## Not the right service when - You need a general analytics warehouse. That is a different and well-served market. - The use case is not defined yet. Building pipelines before knowing what they feed is how data platforms become expensive furniture. - The data genuinely does not exist. No engineering compensates for that, and it is better said early. ## Frequently asked questions Is this different from ordinary data engineering? The pipelines are similar; the requirements differ. AI workloads care much more about unstructured content, chunking, embedding freshness and lineage down to the passage level, and much less about the star schema. The failure modes are also quieter, so monitoring has to be designed differently. Do we need a data warehouse first? Usually not. Plenty of valuable AI work runs against operational systems and document stores directly. A warehouse is worth building when several consumers need consistent history — not as a prerequisite for a first AI system. How do you handle documents that are badly scanned? Realistically. Some will not parse reliably and the honest answer is to identify those, measure how many, and decide whether the value justifies fixing them at source. A pipeline that silently produces poor text for ten per cent of documents is worse than one that flags them. What about personal data? Classified early, minimised, and kept out of anything it does not need to reach. Retention and the legal basis are decided before the pipeline is built rather than retrofitted, because retrofitting data protection is far more expensive than designing for it. How long before something improves? Quality checks on an existing pipeline can be in place in days and usually surface a real problem immediately. A properly built path for one significant source is typically three to six weeks. ## Keep reading - [ ### The answer was not wrong. The document was three months old Freshness is the quietest failure in an AI system. Nothing errors, nothing alerts, and the answers are perfectly consistent with a version of reality that has moved on. ](https://predictes.com/blog/the-answer-was-wrong-because-the-document-was-old) ## Other services - [ ### AI Strategy Which decisions are worth changing, what each would cost, and what would measurably be different. Including the ones where the answer is not AI. Read article](https://predictes.com/services/ai-strategy) - [ ### AI Transformation The part after the strategy: sequencing against real capacity, changing how work is done, and making adoption somebody's job rather than a hope. Read article](https://predictes.com/services/ai-transformation) - [ ### AI Agents Engineering Agents that do work rather than answer questions — with bounds, tools, approval gates and a decision log that survives the first incident. Read article](https://predictes.com/services/ai-agents-engineering)