AI data pipelines: architecture, governance and production delivery

The feature works. Then somebody asks which document the answer came from, and the room goes quiet. Nobody recorded which version of the source, which version of the instructions or which model ran. The demo took two weeks. Answering that one question takes a rebuild.

An AI data pipeline is more than an ETL job, the routine that copies records from one system to the next. It also holds the rules, tests, sign-offs and feedback loops that decide whether the system can be trusted for one task.

The goal is simple. A product team should be able to say what data the feature used. Which prompt (the written instructions given to the model) and model version ran. What checks applied. How a risky output is held back. The NIST AI Risk Management Framework is a useful reference. It treats risk work as part of design and use, not a review held at buying time.

A feature nobody can trace costs you twice: once for the rebuild, and again for every week the team will not ship it. Autonomous Technologies is an engineering firm. We build and run the data and AI systems behind Shopify stores and commerce businesses, for the founders and operators who answer for the product after launch. Our engineering work starts at the approval boundary, the point where a person must sign off, because that is where the rebuild cost hides and where the second payment for the same feature is avoided.

This is for the person who will own the feature in six months

Read this if you have an AI feature in production or heading there, it reads company data, and you are accountable when it produces a wrong or unsafe result. It assumes you are buying models through an API (a service your software calls over the internet) rather than training your own.

Skip this if you are still testing whether a model can do the task at all. Run that experiment first, on copied data, with nobody depending on it. Come back the moment a second person starts using the output to make a decision.

It covers features that read company data: support assistance, document extraction, product enrichment, internal analysis. It names no model provider. The architecture follows the task and the failure modes you can live with.

What makes an AI data pipeline different?

Analytics pipelines turn source records into a governed dataset or report. AI data pipelines do more. They control how context reaches a model. They control how output is judged, and who signs it off. They control how feedback improves the system without polluting it.

That means the data path has two directions. One path moves approved inputs toward an AI task. The other captures evaluation results, user corrections, incidents and outcome evidence. Both need lineage, the record of which input and which setup produced which output. A pipeline that makes an answer but cannot rebuild the input context or evaluation result is hard to run responsibly.

Start with the decision and its approval boundary

Do not begin with a vector database (a store that finds text by meaning) or a model API. Begin with the decision the product supports and the cost of a wrong answer. A system that drafts an internal reply has a different approval boundary from one that changes a price or emails a customer.

Write the product contract in plain language: who can use the feature, which sources it may use, what it must not decide, when it must show evidence, and when a person must approve the output. That makes the choices reviewable before a prototype quietly becomes production.

A gate only counts when a named person stands behind it. On Sene’s agent-authored features, thirty-seven pull requests (proposed code changes) reached Sene’s order systems between 14 July and 8 September 2026. A named person accepted every one. The gate was the design, not a net bolted on later.

The five stages of an AI data pipeline

The names vary, but a reliable design separates the following concerns.

1. Capture and classify source data

Take in only data with a stated purpose. Keep the source id, the version or changed time, the access class (who may see it) and the provenance (where it came from). Permissions are part of the pipeline. If a user cannot open a document at source, the feature must not surface it.

2. Prepare task-specific inputs

Convert files to one consistent format. Drop duplicates. Split long documents. Attach the metadata, the labels you will filter and explain with. More context is not better. Stale or clashing context makes a good model worse.

For structured extraction (pulling set fields out of documents), preparation may mean defining allowed document types and fields. For a knowledge assistant, it may mean reading documents into clean text, keeping headings and permission fields, and setting a review date. Preparation should produce a saved, versioned output you can repeat, not an opaque one-off script.

3. Evaluate before release

An evaluation set is a small, picked set of real inputs with known right answers. It holds plain cases, hard cases, and cases where the safe move is to decline or hand off. Keep it apart from live feedback until each example has been checked for privacy and access.

Evaluation is not a single accuracy number. For an extraction task, examine field correctness, missing fields and unsupported values. For an assistant, look at grounding (whether answers stick to the sources), citation quality, task completion, and refusals. Check that the answer stays within who may see what. For each run, record the model, the prompt, the retrieval setup (how the system looks up source text) and the data version.

4. Release through a controlled interface

Ship the smallest screen that makes the limits clear. Show the source when it helps a person review. Keep costly actions behind an approval. Keep a log of the request, the state of the output and any user fix, so you can audit it later.

5. Observe, correct and improve

Monitor how fresh the data is, input failures, latency (response time), cost, evaluation regressions (tests that used to pass and now fail) and user corrections. A dashboard is useful. An operating response matters more: who gets the signal, what they inspect, when the feature is paused, and how a corrected artifact returns through evaluation.

Lineage is the debugging path

Lineage ties an output back to the inputs and setup that made it. In a normal data platform that means jobs, datasets and columns. For AI, add the prompt version, the model id, the retrieval setup, the policy checks and the test run.

Use an open standard where it fits. Do not log private content without a reason. The OpenLineage project is an open framework for recording dataset and job lineage. It covers the data half of the chain. Your own telemetry, the usage records your system collects, covers the rest.

A clearly labeled illustrative example

Take an internal catalog assistant that proposes missing fields from supplier documents. This is a sketch, not a claim about a live client system.

The pipeline accepts only documents tied to a supplier record and classifies them by document type. Preparation pulls out the text and keeps page references. It tags each prepared file with supplier, product family, document date and access policy. The task contract lets the assistant propose fields from an approved list of terms, but it cannot publish them.

The test set holds plain spec sheets, vague documents, clashing revisions, and documents that lack the field. Before release, reviewers check three things. Every proposal cites a page. Every value comes from the allowed list. Any thin field is left blank. In production a catalog owner signs off each one.

That is a useful first build. It has bounded inputs, a human approval step and observable corrections, and no agent changes a customer-facing system.

What we saw on a real pipeline

We run one of these ourselves. It feeds a repair assistant for one vehicle model from factory manuals, forum threads and video transcripts. Each source type has its own small importer, a program that reads that source in. Every importer emits the same six fields: text, source id, section or page, URL, engine tags and market tags. A record without them never reaches the index, the searchable store the assistant looks things up in.

The test set was written before the search code. It is 20 questions, each with the words a correct answer must contain, and the runner prints pass or miss per question. When a new manual goes in, the set runs again the same day. The observe stage is a plain query log. It drives the one improvement loop we allow: a query that misses gets an alias row (an extra search synonym), 23 so far, not a model change.

What it does not prove: one corpus, one owner, no permission model.

What breaks, and who owns it when it does

Here is the objection worth taking seriously. Most of these pipelines do not fail on model quality. They fail on ownership, and the failure is quiet.

The pattern repeats. One engineer builds the prototype in a fortnight. It is useful, so access widens. The engineer moves on, and the feature now has users but no owner, no maintained evaluation set, and a preparation step running on somebody’s laptop. The first incident finds all three at once.

Name three owners before you widen access, and write them somewhere a new person will find. The product owner decides what the feature may and may not do. The service owner keeps it running and is alerted when it stops. The data owner is accountable for each source and its permissions. One person can hold more than one role. Nobody can hold none of them.

Then rehearse the uncomfortable cases before a customer finds them for you.

Cost drivers and risks to surface during buying

Model tokens (the units a provider bills by) are visible. They are rarely the biggest number. Preparation, indexing (building the searchable store), testing, monitoring, human review and incident work usually cost more. The shape of the product drives that cost, not the model. The platform choice, Databricks or self-managed Spark, moves that cost between licence and engineering time rather than removing it.

Your next step is one task, one owner, one evaluation set

Choose one narrow task where the sources, the responsible owner and the human review path are already known. Write the contract and what the system does when it declines. Assemble a small evaluation set before you integrate anything. Build source capture and permissions. Produce a versioned prepared output. Release to a limited group, with a clear way to collect feedback. Then run the four drills above and scale only once the uncomfortable cases behave.