The data decisions an engineer should not make for you

An engineer can build a pipeline that ingests documents, checks their freshness and records the lineage of every passage. What an engineer should not decide is how fresh the price list has to be before the system answers customers from it, or whether it is worth paying to rescan the contract archive.

These decisions look technical and are business decisions. When nobody on the business side makes them, the engineer does — by default, sensibly from where they sit, and without telling anyone. You find out at the first dispute.

Five decisions that belong to you

Who owns each source. Not the system, the content. Somebody who knows what an empty file on a Monday morning means, and who receives the alert when a load fails.

How fresh the data must be. The price list may need updating within hours, the terms and conditions within days, the archive of closed projects never. A single threshold for everything is either too expensive or too loose, and only somebody who knows what the data is for can tell the difference.

What to do with documents that cannot be read. Some scans will not parse reliably, whatever the tool. Whether to fix them at source, exclude them openly or route them to a person depends on their value, and the business knows that value, not the pipeline.

Personal data: legal basis and retention. Which personal data needs to reach the system at all, on what basis, and how long to keep it. That is settled with your lawyers, and with your data protection officer if you have appointed one — before the pipeline is built, because retrofitting data protection costs far more.

Who sees the quality numbers. When completeness or freshness drops, who finds out and what can they do about it. A quality metric nobody sees is a quality metric nobody acts on.

Documents that cannot be read

This decision is easy to postpone, because it looks like a detail. One question simplifies it: what is this document worth if the system is to read it correctly?

Infographic

Which documents are worth fixing at source

Reads cleanlyHard to read

Take as they areEnter the pipeline with no extra work and no priority.

Main pathFull checks, lineage and the tightest watch on freshness.

Exclude openlyFlagged and counted, so nobody assumes they are in the system.

Fix at sourceRescanning or manual verification, a budget decision for the business.

Low valueHigh value

A pipeline that quietly produces poor text from part of the documents is worse than one that flags and counts them. Flagging turns a hidden problem into a budget line somebody can decide on.

What it costs in your time

Less than it seems, if it is planned. The source owner: a few conversations at the start and receiving alerts afterwards. Decisions on freshness and documents: a short meeting per source, with proposals and consequences prepared by the team. Personal data: one round with the lawyers before the build rather than three after it. Altogether it is usually less time than one meeting explaining why the system answered a customer from an outdated price list.

The most expensive option is no decision. The project either waits, or proceeds on defaults that somebody discovers at the least convenient moment.

What a decision properly made looks like

Short and in writing. For each source: who owns it, what freshness threshold applies and who gets the alert when it is breached. For documents: which classes go down the main path, which are excluded openly and how many there are. For personal data: what reaches the system, on what basis and for how long.

That record fits on one page per source and is worth more than its length suggests. At the first dispute — with a customer, an auditor or between departments — the answer to why the system said what it said starts there. Without it, the answer starts with a search for somebody who remembers.

How to tell the decisions are not being made

Questions from the data team sit in an inbox for weeks. The documentation fills up with sentences beginning “we assumed that…”. The schedule slips although nobody reports a technical problem.

Each of these means somebody is making a business decision without a mandate. Not because they want to, but because the pipeline has to know what to do with the data and nobody has told it.

In AI data engineering we put these questions on the table at the very start, while profiling what exists — because at that point an answer costs one meeting rather than a rebuilt pipeline.