The Data Engineer's Guide to Multi-Vendor Feed Integration

The pattern we see repeatedly: a data engineering team builds a custom transformation script for their first vendor feed. It works fine for a few months. Then the vendor ships an API update, renames two fields, and the script starts producing NULLs for those fields without raising an error. The team finds out six weeks later when a downstream metric looks wrong.

Multiply this by four vendors and you have a maintenance picture that is hard to justify. This post covers the actual failure modes in multi-vendor ingestion and what a more maintainable architecture looks like.

The Five Failure Modes, in Order of Frequency

After looking at a lot of vendor integration setups, these are the problems that show up in production most often:

1. Field name variation: the same concept has different names across vendors
2. Unit mismatches: currency, temperature, measurement units that differ between sources
3. Date format inconsistency: ISO 8601 vs. MM/DD/YYYY vs. epoch vs. European format
4. Schema version bumps with no advance notification
5. Null encoding differences: null vs. "N/A" vs. 0 vs. empty string

Numbers 1, 3, and 5 cause data quality issues. Number 4 causes both quality issues and pipeline failures. Number 2 is the worst because the failures are invisible: numerically plausible wrong values that pass all validation.

Field Name Variation Is More Extensive Than You Think

A single concept like "company revenue" can appear across four vendors as:

Concept	Vendor A	Vendor B	Vendor C	Vendor D
Revenue	gross_revenue_usd	total_rev	RevAmount	revenue_eur
Company name	company_name	companyNm	entity_name	NAME
Record date	created_at	record_date	DATE_CREATED	ts

None of these is wrong from each vendor's perspective. They're conventions their engineers chose independently. Your transformation script either hard-codes the field names (brittle, fails on any vendor change) or tries to auto-detect them by pattern matching (error-prone at the edges). The canonical alias map approach handles this more reliably, because the detection step runs once at schema inference time rather than on every record.

Why Silent Unit Errors Are the Worst Class of Failure

Unit errors don't cause pipeline failures. The pipeline runs. Data lands in your warehouse. Models train. Dashboards update. The wrong numbers look like right numbers. A revenue field in EUR treated as USD doesn't produce a type error; it produces a value that is off by roughly a factor of 1.1 to 1.2 depending on the current exchange rate. For revenue forecasting, that's a material error that your downstream model will learn as signal.

The most expensive silent unit error we encountered on my previous data engineering team: temperature sensor data from two equipment vendors, one sending Celsius, one sending Fahrenheit. Both were ingested as-is into a Celsius column. For several months, the anomaly detection model was trained on a mixed-unit dataset. Normal readings from the Fahrenheit vendor (say, 75F) were being interpreted as 75C, which is a significant temperature anomaly for the equipment category. The model was systematically misfired in one direction for those vendors.

This is why HyperNorm's pipeline layer declares units as a first-class attribute on every canonical field, not just as a comment in the schema documentation.

Why Custom ETL Glue Keeps Breaking

Custom ETL scripts fail for one structural reason: they are point-to-point. Each script knows about exactly one vendor and exactly one target schema. There is no shared canonicalization layer. When the target schema changes, every vendor script needs to be updated. When a vendor changes their schema, that script needs a patch.

In a team of two data engineers handling five vendor feeds, this produces a constant stream of interrupt work. The scripts work until someone leaves the team. The person who joins six months later inherits five custom scripts with no documentation of why specific field name remappings exist or which vendor uses which date format.

The alternative is to treat the canonical schema as a first-class artifact, stored separately from any pipeline script. Every vendor connector maps inbound fields to the canonical schema. When the canonical schema changes, the mapping updates in one place. When a vendor changes their schema, you update only that vendor's alias map.

connectors:
  vendor_karbon:
    field_mappings:
      gross_revenue_usd: revenue
      company_name: entity_name
      created_at: record_ts
    date_formats:
      record_ts: "%Y-%m-%dT%H:%M:%SZ"
    unit_declarations:
      revenue: usd

Schema Conflict Resolution

When two vendors send the same canonical field with conflicting values for what appears to be the same record, you need an explicit resolution strategy. This is not an edge case. Common examples: two vendor feeds have different prices for the same product, or two CRM exports have different contact emails for the same company.

Resolution strategies by field type:

Conflict type	Available strategies
Numeric (price, revenue)	Prefer most recent, prefer specific vendor, average, flag for review
String (company name)	Prefer entity resolution canonical output
Timestamp	Use latest, use earliest, flag
Boolean	Prefer true, prefer false, flag

The key point here: conflict resolution needs to be declared explicitly, not left to last-write-wins behavior in the destination database. Last-write-wins is an implicit data policy that no one on the team will remember choosing, and it produces different results depending on the order connectors run.

Monitoring a Multi-Vendor Pipeline

The monitoring surface for multi-vendor ingestion is larger than for single-source pipelines. Beyond standard pipeline health metrics (run duration, row count, error rate), you want per-vendor metrics on every run:

• Field coverage: percentage of expected canonical fields present in the run
• Coercion error rate: percentage of rows with at least one type coercion failure
• Schema version: which alias map version was used
• Unmatched fields: field names in the inbound data not present in the current alias map

A spike in unmatched fields for a specific vendor, with no spike in coercion errors, typically means the vendor added new fields. A spike in coercion errors with a coverage drop typically means they restructured their schema. These two patterns point to different responses: the first is usually safe to ignore temporarily; the second requires immediate investigation.

A Note on What This Approach Does Not Fix

A shared canonical schema and alias-map approach solves the structural problem of multi-vendor integration maintenance. It does not solve the entity identity problem. Two vendors sending data about the same company with slightly different names and no shared ID field still produce two rows in your canonical output. Entity resolution is a separate step that runs after normalization.

We are also not claiming that a normalization config eliminates all manual work. Writing and maintaining the alias map for a new vendor connector is work. It is less work than maintaining a custom ETL script, and it is more recoverable when something goes wrong, but it is not zero.

Getting Started Without Rewriting Everything

If you're currently running three or more vendor feeds with custom scripts, the most valuable first step is not to migrate everything at once. Start by writing down your canonical field names as a document. Then create a mapping table for each vendor: for each of your canonical fields, what is the corresponding field name in that vendor's output.

That mapping document is the artifact a normalization layer maintains automatically. Once you have it written down, you can see exactly where the gaps are (canonical fields with no vendor mapping, vendor fields with no canonical target) and prioritize which connectors to migrate first based on which ones fail most often.