Building Trustworthy Data Products Module 2 · Know What You Were Given

What Curated Guarantees

Last reviewed · content updated

Intermediate

What you'll learn

~18 min
  • Read the raw, processed, and curated zones as a consumer rather than a builder
  • State what a promotion to curated asserts and the four things it never asserts
  • Locate the responsibility boundary between the platform team and you, before you need it

The three zones, from the outside

Meridian’s lake is organized the way most are, in three tiers that get called the medallion pattern:

  • Raw — exactly what arrived, unmodified, immutable. Wrong values, duplicate rows, and a column labeled TEMP_FIX_2019 all survive intact, on purpose. Raw exists so that any downstream mistake can be re-derived from the truth as it was received.
  • Processed — parsed, typed, deduplicated, conformed to consistent column names and data types. Structurally clean. Semantically untouched.
  • Curated — modeled for consumption. Joined, enriched, given business-meaningful names.

As a consumer you will mostly read curated, occasionally read processed, and should read raw only when you are investigating a discrepancy. That last one matters: when your number disagrees with the source system, raw is the only zone that can tell you whether the problem arrived or was introduced.

Prompt first: what the zone does not tell you

I have read access to a curated table: [name].
Before I use it, tell me what I can and cannot conclude from the fact
that it is in the curated zone. Then list the specific questions I
should answer about THIS table that its zone does not answer for me -
completeness, accuracy of the upstream source, whether column names
match their contents, and fitness for a reliability-analysis purpose
as opposed to whatever it was originally modeled for.
Do not query anything yet. I want the question list first.

Note what is being asked for: a question list, not an assessment. The agent has no way to know whether Meridian’s service_start_date is misnamed. It can reliably tell you that “verify column names against contents” belongs on the list, which is the part that gets skipped.

KNOWLEDGE CHECK

A curated table passes every platform quality check and shows no errors. Your monthly totals are 12% below the source operational system. What is the most likely explanation to check first?

What a promotion to curated actually asserts

This is the part that costs people. When the platform team promotes a table to curated, they are asserting a specific and narrow set of things:

It asserts:

  • The schema is stable and changes are announced
  • Types are correct and consistent
  • The load ran, completed, and is tracked
  • Structural duplicates from the ingestion process have been removed
  • The table is documented well enough to find

It does not assert:

  • That the values are accurate. Curated means the pipeline is trustworthy, not that the source system was right. A meter that reported wrong for six months produces six months of faithfully-transported wrong values in curated.
  • That the data is complete. A source that stopped sending on a Tuesday produces a curated table that is clean, well-typed, and missing Wednesday onward — often with nothing visibly wrong.
  • That it means what its column names suggest. service_start_date in Meridian’s curated customer table is the date the record was created in the 2019 conversion, not the date service started. Nobody renamed it because a dozen things read it.
  • That it is fit for your purpose. Curated for billing is not curated for reliability analysis. The population rules differ, and nothing in the table announces that.
⚠The one-sentence version

Curated is a statement about the pipeline, not about the world. It tells you the data arrived intact. It tells you nothing about whether the data was right when it left.

The responsibility boundary

Find this before you need it, because you will need it at the worst moment. At Meridian the line runs like this:

QuestionOwner
The load failed last nightPlatform team
Schema changed without warningPlatform team
The source system sent bad valuesSource system owner — platform team transported them faithfully
The column name is misleadingNobody, historically. This is the gap.
My join produced double rowsYou
The number is wrong for my purposeYou

Two rows deserve attention.

“The source system sent bad values” is the most common real problem and the one with the most orphaned ownership. The platform team is correct that they transported it faithfully. The source system owner is correct that their system works for its own purpose. Meanwhile your number is wrong. Escalating this well means arriving with evidence — which is what Lesson 2.2 produces — rather than with a complaint.

“The column name is misleading” is genuinely unowned in most organizations, and it is where the largest silent errors live. When you find one, the useful move is not to get it renamed. It is to record it somewhere your future self and successor will look. Module 6 gives that record a home; for now, keep a running list.

Key takeaway

Curated is a claim about the pipeline, not about the world: it asserts stable schema, correct types, a completed load, and structural deduplication, and it asserts nothing about accuracy, completeness, whether column names match their contents, or fitness for your particular question. The responsibility boundary has two soft spots — bad values from a source system, and misleading column names — and both land on you by default. Lesson 2.2 turns that from a warning into evidence.

Search lessons