Medallion architecture for AI: Bronze, Silver and Gold explained

Finance says 9,400 units shipped last month. The warehouse feed says 9,655. Both numbers come from the same platform. Someone now spends a day working out which one to put in the board pack.

The direct answer. Medallion architecture is a way of organising your data into named stages, and each stage raises the bar. Bronze keeps what the source sent. Silver holds records that passed checks. Gold holds data shaped for one business question. The names matter less than the rules at each step.

The payoff is the day a month you stop spending on reconciling two systems by hand. Autonomous Technologies is an engineering firm. We build and run the systems behind Shopify stores, for founders and operators who own the store after launch. Our ecommerce integrations work is priced on that saved day: we name who owns each record first, so the reconciling stops before any layer is drawn.

Three named folders prove nothing. The real test is whether you can answer four questions for every dataset that matters. A dataset is a table or file of related records, such as all your orders. Where did this record come from? What checks did it pass? Who owns its meaning? What happens when a check fails? The Databricks reference architecture, from a large data platform vendor, calls the layers a recommended pattern, not a product feature.

Who this is for, and who it is not for

This is for engineering, data and operations leads who feed reporting and model work from one platform. Models are the trained programs that make predictions. This guide is vendor-neutral, so it favours no particular software supplier. The tables and the example are design aids, not a technical specification.

It is not for a team with one source, one report and one owner. Write one job, name the person who fixes it, and spend the saved months on the product. It is also not a rescue for unclear business rules. Three folders will not settle what net revenue means.

What each layer is actually for

Bronze holds captured source data and the details of how it arrived. Silver holds records that have been checked, de-duplicated and lined up. De-duplicated means repeat copies of the same event are collapsed into one. Gold holds data cut for a stated business question.

A layer is only as good as its contract. A contract names the allowed inputs and the shape of the data. It also names the freshness goal (how recent the data must be), the checks, the access rules, the owner and the failure path. Without those, a Gold table can be as opaque as a raw export.

Bronze is not an analyst sandbox. Databricks advises keeping raw state and its history, with little cleanup, so you can rerun the processing and audit the results when the data feeding it changes. Silver is where you check the shape of the data and handle missing values, duplicates and late records. Gold is where totals and models make a business question easy to answer.

When the pattern is worth the cost

Use it when several teams need the same source data at different levels of trust. It earns its keep when sources change, late records are normal, definitions are argued over, or product teams need data with clear rules and owners outside a dashboard.

Write the contracts before you pick the tool

Start from the decision that needs to improve. An operations lead may want a daily view of fulfilled orders, cancellations and return reasons. That one request settles a chain of choices. Which order ID. Which system owns fulfilment state. What an amended order means. Which timezone sets the business day. Which test catches a missing carrier event.

For every dataset that crosses a layer, record six things.

1. Grain and key. What one row means, and the stable key (a unique ID) used to match it.
2. Source and owner. The system allowed to set each important field.
3. Quality gate. The checks, the threshold, and whether a bad record is rejected, held or passed with a warning.
4. Freshness goal. How recent the data should be, in plain words, with one visible timestamp (the date and time of the last update).
5. Recovery path. How an owner finds, fixes and reruns a failure without quietly editing history.
6. Access and retention. Who can see what, matched to how sensitive the data is, and raw data kept only as long as it is needed (retention).

Silver is where most teams gain ground

Silver builds reusable records of the things that matter, without bending them to one dashboard. For orders, that may mean a checked stream of order events (each change to an order), a single order record, and separate fulfilment events. Each carries both the source time and the time it arrived.

Keep bad and unclear records visible. Holding a record helps only if someone can see why, who owns the fix, and how it rejoins the flow. A hidden table of rejected records turns a data problem into a reporting debt.

When a source changes shape, keep the received record in Bronze. Catch the break at the promotion step, the point where a record moves up a layer. Tell the owner, with enough detail to act. Do not quietly force a new value into an old category to keep a dashboard green.

Treat Gold as a product with an owner

A Gold dataset answers a set question for a set group. A weekly revenue summary is not just a total. It encodes rules about refunds, tax, currency, order state and the business calendar. A business owner approves what those rules mean. Engineering owns whether the build is reliable.

Keep Gold products small and named. A finance view and a growth dashboard may differ on purpose, as long as the difference is written down. The danger is two products with one name and two rules.

What we saw on a real pipeline

Our Shopify audit pipeline, a chain of automated steps that runs each audit, is a medallion pipeline that never uses the word. Every run gets one folder. Raw captures land under evidence, split into static and browser, exactly as received. A merge step combines them into one manifest, a checked list of everything captured, with a content hash per source. A content hash is a fingerprint that shows a file has not changed. No stage starts until the output before it validates against its contract. Findings, the revenue projection and the two reports sit at the top. Only one of the two reports may leave the building.

The last row is the rule that earns its keep. A probe is a small script that checks one thing on a page. When a probe cannot read a page, it stops with exit code 2 (a number a script returns to say what happened) and the dependent claim is dropped. I would rather the pipeline say nothing than read an absence as a zero. That is what Silver is for.

How the pattern serves model and RAG work

For training a model, record the dataset version, the feature rules and the time window each run used. Features are the input values the model learns from. A clean Silver table can still leak the future into a past example, if a join (a step that matches records from two tables) uses a value nobody had at the time. Build repeatable snapshots and test that before shipping a model. Databricks documents point-in-time feature joins for this.

RAG serves something else. RAG (retrieval-augmented generation) means the app looks up source text and hands it to the model before it answers. Bronze can hold source versions. Silver can hold parsed text with stable IDs, permissions and checks. The index, the searchable copy the app looks up, has to carry edits, deletions and permission changes through. Gold is not a vector database (a store that finds text by meaning), and an embedding (a list of numbers that represents a piece of text) proves nothing about accuracy.

Microsoft’s RAG design guidance keeps ingestion (loading data in) and retrieval choices apart from generation and scoring.

Cost drivers and buyer risks

Storage is the small line. The running cost comes from how often you ingest, how much you process and how much history you keep, plus transfer, monitoring and the people who clear exceptions. Databricks makes the trade-off plain: continuous ingestion costs more and waits less than batch loading, which runs in scheduled chunks.

A staged sequence that proves recovery

Pick one domain, such as orders, and one decision that keeps needing manual work. Map the real sources and keys, then build Bronze capture and one Silver model.

Publish one Gold product with a named owner and an acceptance test. Break it on purpose before you widen. This proves the team can recover, not just demo a clean run.

The ownership rule is the part that survives a change of platform. Thirty-seven pull requests (proposed code changes) reached Sene’s order systems between 14 July and 8 September 2026, and a named person accepted every one (Sene, agent-authored features behind a human gate).

If the next question is where these jobs run, our Databricks vs Apache Spark comparison covers platform and operating duties.

Start with the number people argue about

Take the one figure two systems report differently. Trace it back to the event each system received. That is an afternoon of work. It tells you whether you need layers or only an owner. The free scan does that first trace for you. It reads what your storefront and its connected systems record and lists the numbers that already disagree. It then ranks them by what each gap costs you in margin or in hours spent reconciling. You get back the shortest path to one number the board pack can trust, before a single layer is drawn.