The Layer That Nobody Shipped
The modern data stack has become good at certain things. Fivetran or Airbyte replicates your source tables. dbt transforms them. Snowflake, BigQuery, or Redshift stores and queries them. Orchestration tools like Airflow or Prefect schedule the whole sequence. Walk into most data teams today and this layer is in reasonable shape.
But walk into those same teams and you will also find someone with a Python script, an Excel lookup table, or just a lot of manual effort, doing the thing none of those tools address: figuring out whether "Acme Corp," "ACME Corp.," and "Acme Corporation" represent the same company.
That gap is not an oversight. It is structural.
Why Schema-First Tools Cannot Solve a Probabilistic Problem
The tools that make up the modern data stack are fundamentally schema-first. They assume that if you can define a schema, a transformation, and a destination, you have described the data problem. Fivetran replicates table structures. dbt transforms column values according to rules you declare. These tools are excellent at what they are built for.
Entity resolution requires a different assumption: records pointing to the same real-world object may look completely different across sources, and resolving them is a probabilistic, domain-specific matching problem. No SQL transformation handles that natively. No connector can infer it from a schema definition. You have to explicitly model the matching logic, tune it for your data distribution, and decide what confidence threshold makes a match actionable versus something a human should review.
The modern data stack tooling did not ignore entity resolution. The tools scoped it out deliberately. Resolution is messy and domain-specific in a way that does not fit cleanly into a declarative transformation framework. Building it well requires a different class of algorithms: blocking, similarity metrics, confidence scoring.
What Happens Without It
When entity resolution is not a first-class pipeline step, it happens informally. The patterns are recognizable:
A data analyst writes a GROUP BY company_name query and notices 47 variants of what is obviously the same company. They write a CASE statement to normalize the most common ones. Six months later, the vendor updates their export format and the CASE statement misses a new variant. The analyst is busy and it takes two weeks for anyone to notice the downstream segmentation is off.
A data engineering team builds a custom fuzzy-matching script for one vendor. When a second vendor is added, the script gets extended with vendor-specific logic. When a third vendor arrives, someone writes a separate script. Eventually the two scripts produce contradictory canonical names for the same entity on edge cases, and neither team knows which one to trust.
A CRM team runs a quarterly deduplication job against their Salesforce export. Between runs, duplicate records accumulate and contaminate downstream models. The segmentation model treats "Karbon Systems" and "Karbon Systems Ltd." as two separate companies with independent revenue histories.
Each of these workarounds is locally reasonable. None of them compose with the rest of the stack. They produce isolated fixes, not a canonical entity registry that downstream systems can query consistently.
A Scenario: Catalog Deduplication at a Logistics Data Team
Consider a data team managing product catalog records from three suppliers. Each supplier delivers a CSV weekly, with product names, supplier identifiers, and physical dimensions. No supplier uses the same naming convention. "Gearbox Assembly (Standard)" from one supplier is "GEARBOX-ASSY-STD" from a second and "gearbox assembly std" from a third.
For two years, a junior analyst maintained a lookup table in Google Sheets that mapped variant strings to canonical product names. The table grew to 2,400 rows. When a supplier changed their format, the analyst updated the sheet. When new products arrived, the sheet grew.
The analyst moved to a different team. The lookup table fell out of sync within a sprint. The downstream inventory model started treating two variants of the same product as separate items, predicting independent reorder points. The anomaly surfaced five weeks later during a physical count. The pipeline had run without errors throughout. This was not a pipeline failure. It was an entity resolution gap.
When the team ran their catalog through HyperNorm with a Jaro-Winkler threshold of 0.87 and blocking on product category, the first pass resolved a 4,200-row supplier catalog down to 1,400 canonical products. The 500-record discrepancy between the old lookup table and the resolved output became a concrete audit target rather than a vague "data quality concern."
What a First-Class Pipeline Step Actually Means
When we say entity resolution should be a first-class pipeline step, we mean something specific: a component that sits after ingestion and before your transformation layer, takes records from multiple sources, runs blocking and matching logic, and outputs a canonical entity record with a confidence score and a provenance trail. That output should be queryable. If a downstream model needs to know which canonical entity ID maps to a given raw string, that should be a simple lookup, not another script.
Here is what a resolved record looks like in HyperNorm's output format:
{
"canonical_id": "ent_08f3a1c9",
"canonical_name": "Acme Corporation",
"confidence": 0.97,
"source_records": [
{ "source": "vendor_a", "raw_name": "Acme Corp", "raw_id": "A-1042" },
{ "source": "vendor_b", "raw_name": "ACME Corp.", "raw_id": "B-7831" },
{ "source": "vendor_c", "raw_name": "Acme Corporation", "raw_id": "C-0019" }
],
"matched_at": "2026-05-28T14:22:17Z"
}The canonical record is not a guess. It is a confidence-weighted determination based on blocking strategy and similarity scoring, with full provenance so you can trace any downstream anomaly back to its origin record.
We Are Not Saying the Modern Data Stack Is Broken
To be direct: dbt, Fivetran, Snowflake, and the tooling that surrounds them are excellent. The modern data stack solved a real and significant problem for data teams. We are not arguing that entity resolution should replace the transformation layer, or that SQL-based transformation is inadequate for most data problems. It is more than adequate for most data problems.
What we are saying is that entity resolution is a categorical step the modern data stack was never designed to include. It requires probabilistic matching, not deterministic transformation. It requires a shared canonical entity registry, not a one-off script bolted to the side of a dbt project. And it requires ongoing maintenance as source data formats drift, which they always do eventually.
If your data team spends more than a day or two per sprint on manual deduplication or ad-hoc string normalization across vendor feeds, that time is a measurable cost with a measurable alternative. At a team of four engineers, that is somewhere between 5 and 10 percent of capacity spent on work that should be automated pipeline infrastructure.
The Practical Starting Point
Treating entity resolution as a first-class pipeline step means declaring it explicitly, versioning it, monitoring it, and alerting on failures. In practice that involves four things:
- A normalization config that maps known field variants to canonical field names, versioned alongside your dbt models so schema changes are tracked and reviewable.
- A matching config that specifies blocking strategy, similarity metric, and confidence thresholds, tuned per entity type. Company names and product codes have different matching characteristics and different acceptable false-positive rates.
- An entity registry that downstream pipeline steps join against, so each step is not independently re-deriving matching logic from raw strings.
- Alert rules that fire when match rejection rate exceeds a threshold, so schema drift from a vendor does not silently propagate for weeks before someone notices.
The tooling to do this well is recent. Most modern data stack categories had mature options by 2020 or 2021. Entity resolution as a composable, maintainable pipeline component has lagged. For teams ingesting data from more than two vendors with overlapping entity domains, the cost of continuing without it tends to compound faster than teams expect when they finally measure it.
