The Catalog Is Not the Data

What a catalog is for, and what it is not

A data catalog is a searchable inventory of what exists: tables, columns, descriptions, owners, classifications, lineage, and freshness. Meridian runs one, most organizations do, and it is genuinely useful — it is how you find that circuit_faults exists at all.

The failure is treating it as authoritative. A catalog records what somebody wrote down at some point, and the gap between that and what the data is doing today grows every day nobody looks.

Three specific gaps matter.

Prompt first: test the catalog’s claims

For table [name], I have the catalog entry [paste it].

Write queries that test each catalog CLAIM against the data:
- for every column description asserting a unit, range, or meaning,
  a query whose result would contradict it if it were wrong
- compare the declared freshness against the actual max load timestamp
- compare the declared row-level grain against observed distinct counts
- list the columns present in the data but absent from the catalog,
  and vice versa

Report each as CONFIRMED, CONTRADICTED, or UNTESTABLE. For untestable
claims, say what evidence outside the data would settle it.

The UNTESTABLE category is the one that earns its place. “This column contains the customer’s primary service address” is not something a query can confirm — but knowing that it is untestable from data alone tells you it needs a human to verify, which is a different task you can schedule rather than an assumption you carry.

Gap 1: descriptions written at creation, never revised

The column description for fault_duration says “duration of fault in minutes.” Your profile in Lesson 2.2 established it holds seconds. Both are in the catalog: the description says one thing, the data does another, and the catalog has no mechanism that would ever notice.

This is not negligence. Descriptions get written once, by someone building the table, describing what they intended. The data drifts, sources change, a downstream fix alters a unit — and no one revisits a description that nobody is complaining about.

Gap 2: lineage that stops at a boundary

Lineage shows what feeds what. It is genuinely valuable for impact analysis: change this column, and here is what breaks.

What matters is where it stops. Automated lineage is captured from things the platform can observe — pipeline definitions, SQL parsed from jobs, warehouse query logs. It cannot observe:

• A transformation inside a source system before the data was ever extracted
• A file someone drops on a schedule
• Anything a person did in a spreadsheet and re-uploaded
• Logic embedded in a tool the catalog does not integrate with

At Meridian the lineage for circuit_faults traces cleanly back to an ingestion pipeline reading the outage management system — and stops there. What it does not show is that the outage management system itself applies a business rule suppressing momentary faults under 30 seconds. That rule is invisible to the catalog, and it is the single most important fact about the table for anyone counting fault events.

The question to ask of any lineage graph: what happened upstream of the leftmost node? That is where the transformations you cannot see live.

Gap 3: freshness claims nobody re-checks

Catalogs display a freshness or SLA field: “updated daily.” Sometimes that is measured from actual load timestamps, which is reliable. Often it is a value someone typed when the table was registered, which is a claim about intent from two years ago.

Distinguishing them is quick: compare the catalog’s freshness claim against max(load_timestamp) in the table itself. If the catalog says daily and the newest row is eleven days old, you have learned two things — the table is stale, and the catalog’s freshness field is decorative here.

Fix it or record it?

You will find discrepancies. Fixing every one is not the goal, and attempting it is how this turns into a governance project that never ships a data product.

The practical split:

Fix it when the discrepancy would mislead the next person into a wrong number, and fixing is cheap — a wrong unit in a description, a missing owner, a freshness claim contradicted by the data. These are typically a pull request against catalog metadata or a message to a table owner.

Record it when the fix is not yours to make or is expensive: a suppression rule inside a source system, a column whose name cannot change because a dozen reports read it, lineage that structurally cannot capture a manual step. Write it into your own profile artifact from Lesson 2.2 and move on.

Escalate it when it affects more than you: if fault_duration is in seconds and three other teams are reading it as minutes, that is not a note for your file. That is a message to the table owner with your evidence attached, today.

Key takeaway

A catalog records what somebody wrote down once; the data records what is true now, and when they disagree the data wins. Three gaps recur: descriptions never revised after creation, lineage that stops at the boundary of what the platform can observe rather than at the origin of the data, and freshness claims that were typed rather than measured. Ask of every lineage graph what happened upstream of its leftmost node. Fix cheap discrepancies, record the ones that are not yours, and escalate immediately when other teams are reading the same column wrong. Lesson 2.4 covers the one class of finding you cannot simply record and move past.