Source contracts and onboarding

Research cut-off: 2026-08-23. Scope is restricted to source contracts and rapid, trustworthy onboarding.

Maturity labels:

• Established — broadly deployed with well-understood operating models.
• Emerging — credible implementations or standards exist, but interoperability and operating experience remain incomplete.
• Hypothesis — plausible design proposition requiring Dataheart experiments.
• Vendor claim — must be independently benchmarked before becoming an architectural assumption.

Executive thesis

Dataheart should define ultra-fast onboarding as time to first trustworthy, replayable and policy-correct dataset, not time to first successful connection.

The architectural unit should be a customer-owned source contract bundle:

1. A machine-readable contract describing identity, schema, meaning, keys, compatibility, quality, authorization, retention and deletion.
2. A replayable change protocol describing snapshot boundaries, source offsets, transactions, ordering, updates, deletes and truncates.
3. A connector capability manifest.
4. Executable compatibility and conformance tests.
5. Evidence from discovery, profiling, shadow ingestion, reconciliation and production runs.
6. Customer-owned checkpoints, schema history and deletion receipts.

The contract is the invariant. Power Query, Fabric Mirroring, Purview, Debezium, ADBC, Flink, Iceberg and metadata catalogs are replaceable deployment-profile components.

The decisive distinction is:

Schema registries prove that records can often be decoded. Dataheart must prove that the resulting state still has the intended business meaning, authorization, deletion behavior and recovery semantics.

State of the art 2024-2026

Development	Maturity	Assessment
Machine-readable data contracts	Emerging	ODCS 3.1.0, released 2025-12-08, covers schema, relationships, quality, SLA, infrastructure, roles and authoritative definitions. It is a useful interchange basis, but its JSON Schema validates documents rather than enforcing source or consumer behavior. Dataheart needs a stricter source-profile and runtime tests on top. ODCS 3.1.0
Federated catalog metadata	Established for discovery; incomplete for execution	DCAT 3 became a W3C Recommendation in 2024 and added explicit version relationships and dataset series. It is suitable for catalog exchange, not CDC checkpoints, field compatibility or deletion execution. W3C DCAT 3
Runtime lineage and lifecycle events	Established/emerging de facto practice	OpenLineage represents dataset schema, versions, lifecycle changes and quality assertions as facets. It is an evidence channel, not a source contract or authorization system. OpenLineage object model
Schema compatibility registries	Established	Backward, forward, full and transitive checks are established practice. Compatibility remains format- and configuration-dependent; non-transitive defaults can leave old retained data unreadable. Confluent compatibility documentation
Log-based CDC plus consistent snapshot	Established	Debezium’s PostgreSQL connector illustrates the standard pattern: capture a log position, snapshot, then continue from that position. Under faults it can emit duplicates, deletes without adequate identity are incomplete, and schema changes during incremental snapshots are restricted. Debezium PostgreSQL connector
Batch/stream convergence	Established as a programming model, not perfect semantic equivalence	Flink treats boundedness as the principal difference and recommends streaming-first logic where feasible. Batch still has different watermarks, checkpoints, shuffles and supported operations. Flink bounded/unbounded execution
Lakehouse row-level changes	Iceberg v2 established; v3 emerging	Iceberg v2 encodes equality and position deletes over immutable files. V3 introduces deletion vectors and deprecates position-delete files for new updates. These are analytical storage semantics—not proof of downstream erasure. Apache Iceberg specification
Standard analytical connector API	Emerging	ADBC provides a vendor-neutral, cross-language API for bulk Arrow-native query and ingestion. It does not standardize CDC, source-log offsets, deletion propagation or all authentication behaviors; driver coverage remains uneven. Apache Arrow ADBC
Automated profiling	Established with serious caveats	Profilers increasingly support sampling, partition scoping and metric selection. They still impose source cost, can expose sensitive values, and can miss rare violations. OpenMetadata explicitly advises selective profiling and sampling. OpenMetadata profiler
AI-generated mappings and contracts	Hypothesis	LLMs can draft names, semantic mappings, descriptions and tests from metadata and samples. There is insufficient evidence to let them authorize access, select authoritative keys, approve destructive semantics or accept production contracts without human review.
“Connect in minutes” or “near real-time”	Vendor claim	These descriptions usually measure configuration or nominal replication latency. They do not establish gap-free snapshot handoff, replay determinism, permission parity, delete closure, recovery time or source cost.

Load-bearing concepts

1. A Dataheart source contract profile

ODCS is a useful outer syntax, but Dataheart should make these fields normative:

• Stable source, object and field IDs independent of mutable names.
• Physical and logical types, precision, scale, timezone and null semantics.
• Authoritative business definitions and mapping confidence.
• Primary, candidate and deduplication keys; whether keys can mutate.
• Snapshot consistency level and snapshot watermark type.
• CDC position, transaction and ordering model.
• Insert, update, delete, soft-delete, truncate, drop and rename semantics.
• Before/after-image completeness.
• Schema compatibility mode and deprecation window.
• Freshness, completeness, reconciliation and recovery SLOs.
• Classification, purpose, residency and source/target authorization rules.
• Retention, time travel, backup and erasure obligations.
• Connector version, capabilities, resource budgets and support ownership.
• Evidence requirements and required conformance suites.

Every field should carry provenance: declared by producer, inferred by profiler, observed at runtime, or approved by a named organizational role.

2. Five compatibility layers

A change is safe only if it passes all applicable layers:

1. Wire compatibility — can old/new payloads be decoded?
2. Shape compatibility — do tables, columns, constraints and relationships remain usable?
3. Semantic compatibility — does the same field retain the same unit, population, meaning and null/default interpretation?
4. Operational compatibility — do keys, partitioning, ordering, freshness, transaction and deletion behavior still work?
5. Policy compatibility — are classification, authorization, purpose and retention still correct?

An optional field addition may be wire-compatible while breaking an exhaustive switch, a SELECT * export, a privacy classification rule or a uniqueness assumption.

3. Canonical change envelope

Regardless of source technology, the adapter should emit a canonical envelope containing at least:

• source_id, object_id, contract_version, schema_id
• operation
• stable record key
• source transaction ID
• source position/LSN/offset
• position within transaction
• snapshot flag and snapshot watermark
• source commit time and Dataheart observation time
• before/after-image presence flags
• idempotency key
• connector build and configuration digest
• authorization/classification reference
• cryptographic content digest where appropriate

Clock time must never substitute for a log position.

4. Snapshot plus CDC as one logical history

The preferred pattern is:

1. Establish a repeatable source position.
2. Start or reserve log capture.
3. Read a bounded snapshot.
4. Buffer or overlap CDC.
5. Deduplicate by source position and record identity.
6. Reconcile snapshot state against replayed state.
7. Activate only after convergence tests pass.

Batch and stream should use the same transformation and validation rules when possible. This does not imply the same physical execution plan: Flink documents material differences in batch checkpointing, watermarks and operators.

5. Deletion is a lifecycle, not one event

Dataheart needs distinct terms:

Event	Required result
Source hard delete	Keyed CDC delete, removal from current materialization, replay-consistent state
Soft delete	Retained record with explicit business status; not represented as physical deletion
Broker tombstone	Compaction instruction; not proof that every downstream copy was removed
Truncate/drop	Dataset lifecycle event with ordering and scope evidence
Lakehouse row delete	Row hidden by equality/position delete or deletion vector; old files may remain until expiry/vacuum
Retention expiry	Scheduled removal from serving, history, indexes, caches and derived products
Data-subject erasure	Policy decision plus propagation and evidence across every governed copy
Media/backup disposal	Separate sanitization or expiry process

The 2026 EDPB enforcement report found recurring difficulty with retention decisions, backups and ineffective anonymisation used in place of deletion. A CDC tombstone therefore cannot be treated as an erasure receipt. EDPB, 2026-02-18

6. Automated onboarding is an evidence-producing state machine

Recommended states:

discovered → metadata-scanned → profiled → contract-draft → shadow-snapshot → shadow-cdc → reconciled → approved → active

Failure states should be explicit:

quarantined, log-gap, schema-blocked, policy-blocked, reseed-required, retired.

Automation can progress between states only when its evidence predicates pass. Human approval remains mandatory for business meaning, authoritative keys, sensitive classification, access scope and deletion policy.

Reference patterns

Vendor-neutral invariants versus deployment choices

Concern	Vendor-neutral invariant	Microsoft-native profile	Self-hosted/open profile
Canonical contract	ODCS-compatible Dataheart source profile, versioned as customer-owned text	Azure DevOps/Git-backed contract; projected into Purview and Event Hubs registries	Git-backed ODCS; projected into OpenMetadata/another catalog and an open registry
Batch/API connectivity	Capability-based connector interface and golden tests	Power Query SDK for pull-oriented APIs, OData/ODBC and semantic refresh scenarios; its M/.mez packaging is Microsoft-specific. Power Query SDK	ADBC/JDBC/ODBC for bulk access; connector-specific SDK only behind the canonical interface
Database CDC	Snapshot watermark, source position, transaction order, idempotency	Native Fabric Mirroring where supported; otherwise a Dataheart publisher into Fabric Open Mirroring	Debezium or equivalent log-based connector into Kafka/Pulsar and Flink
Custom CDC landing	Canonical operations compiled to profile-specific format	Fabric Open Mirroring __rowMarker__, primary keys, full-row updates and ordered files	Canonical events plus sink into Iceberg/Delta/Hudi
Stream schema registry	Compatibility policy stored in the source contract	Azure Event Hubs Schema Registry; note that documented evolution enforcement is Avro-only and supports fewer modes than some registries. Azure Schema Registry	Apicurio or another open registry with format-specific compatibility tests
Materialized state	Deterministic replay, table snapshots, retained source positions	OneLake Delta tables	Iceberg on customer object storage, preferably through a portable catalog interface
Metadata	DCAT-compatible catalog export and OpenLineage evidence	Purview/Data Map/Unified Catalog	OpenMetadata, Marquez or another self-hosted implementation
Profiling	Scoped, least-privilege, non-authoritative evidence	Purview managed identity, private networking and data-quality scans. Purview profiling connections	OpenMetadata or a dedicated profiler with sampling and pushdown
Test harness	Same golden data, mutations and recovery assertions	Run against Fabric/Event Hubs/Purview	Run against Debezium/Flink/Iceberg/open metadata stack
Portability proof	Export contract, schema history, checkpoints, lineage and deletion ledger	Microsoft objects are derived projections, not sole copies	Open-stack objects are likewise derived projections

Microsoft-native pattern

Use three connector classes rather than one universal SDK:

• Power Query for scheduled pull and semantic/reporting connectivity.
• Native Fabric Mirroring for supported operational databases.
• A Dataheart publisher targeting Open Mirroring for unsupported/custom CDC.

Fabric Open Mirroring has explicit update/delete/upsert row markers, ordered files and full-row update requirements. Column removal or type changes require table recreation, and processed input files have their own cleanup lifecycle. Open Mirroring format, 2025-09-10

Treat Fabric limitations as contract admission checks. Azure SQL Mirroring currently has a 1,000-table limit, does not propagate source row/object permissions or masking to OneLake, can truncate large LOBs, and reseeds after some DDL changes. Fabric Azure SQL limitations, 2026-02-27

Self-hosted/open pattern

A credible profile is:

• ADBC/JDBC/ODBC or source APIs for discovery and snapshots.
• Debezium for supported log-based CDC.
• Kafka plus Flink for ordered processing and bounded/unbounded execution.
• Iceberg on customer-owned object storage for canonical materialization.
• An open schema registry.
• OpenLineage for runtime evidence.
• DCAT-compatible catalog export and OpenMetadata or equivalent for discovery/profiling.

No single component is the system of record for the contract. Dataheart’s customer-owned bundle is.

Failure modes and counter-evidence

Schema and semantic drift

• Registry compatibility may check only the previous version, not all retained versions.
• Defaults can make old records decodable while inventing a value that was never observed.
• Type widening may still lose precision at a target.
• A rename without a stable field ID becomes delete-plus-add.
• Enum additions can break exhaustive consumer logic.
• Nullable keys and mutable primary keys invalidate delete/upsert assumptions.

Mitigation: mutation testing against retained schemas and representative consumer assertions, not a registry check alone.

CDC and replay

Debezium documents exactly-once behavior only under normal/carefully managed operation; recovery may emit duplicates. It also notes that PostgreSQL deletes without adequate replica identity may not include a usable before-image, truncate events lack a key, and incremental snapshots do not support concurrent schema changes. This is direct counter-evidence to treating a connector checkbox as an end-to-end guarantee.

Mitigation: idempotent sinks, source-position deduplication, transaction-aware ordering, chaos recovery tests and replay/hash reconciliation.

Profiling

• Full scans consume source CPU, I/O and warehouse credits.
• Random samples miss rare invalid values and minority classifications.
• Samples may contain sensitive or secret values.
• High-cardinality profiles inflate metadata storage.
• Profiled values can become a shadow data copy with unclear retention.

Mitigation: metadata-first discovery, column allowlists, pushdown aggregates, bounded samples, sketches with recorded error bounds, redacted evidence and automatic expiry of sample values.

Deletion and retention

• Source delete, broker tombstone, target row delete, vacuum and backup expiry occur at different times.
• Time travel and snapshot retention deliberately preserve older files.
• Derived aggregates, vector indexes, caches and exports are easily omitted.
• An immutable evidence log can accidentally retain the data being erased.

Mitigation: a deletion plan per contract, opaque subject tokens in receipts, explicit exception/legal-hold state, propagation graph and periodic end-to-end erasure drills.

Authorization failure

• Discovery or profiling credentials are often broader than streaming credentials.
• Replication targets may not inherit source row/column permissions.
• Connector logs and dead-letter queues can expose raw values.
• A catalog classification does not itself enforce access.

Mitigation: separate least-privilege principals by function, negative authorization tests, target-side policy compilation and log redaction tests.

Operational cost

Dominant cost centres are:

• Initial snapshots, resnapshots and profiling load.
• WAL/log retention when consumers lag.
• Broker retention and duplicate replay.
• Small-file compaction, row-delete application and table vacuum.
• Schema registry, catalog and lineage cardinality.
• Private networking and gateways.
• Connector upgrades following source API changes.
• Maintaining test fixtures and supported database versions.
• Erasure propagation and backup expiry.
• FDE investigation of source-specific edge cases.

“Zero ETL” removes authored transformation pipelines; it does not remove source reads, replication compute, storage maintenance, policy work or connector operations.

Lock-in

Microsoft-specific risks include Power Query M/.mez, Fabric landing-zone row markers, OneLake item identities and Purview-specific governance APIs.

Open-profile risks include registry API dialects, connector-specific state formats, uneven ADBC drivers, table-format feature divergence and nominally open projects whose enterprise functionality is proprietary.

The guard is a continuously tested export/import boundary—not merely choosing software with an open-source license.

Implications for Dataheart

1. Define Dataheart Source Contract Profile 1.0 as a strict ODCS profile plus extensions for keys, CDC, deletion, authorization, profiling and evidence.

2. Treat contracts as code, but keep runtime observations separate from producer declarations.

3. Build a small canonical connector SPI:

   • discover
   • check
   • read_snapshot
   • read_changes
   • commit_checkpoint
   • profile
   • explain_capabilities
   • run_conformance
   • request_reseed
   • retire

4. Require the customer to own contract history, schema history, checkpoints, raw change journal or recoverable source positions, lineage export and deletion ledger.

5. Compile one contract into Microsoft-native and open-profile configurations.

6. Make acceptance evidence identical across profiles even when the mechanisms differ.

7. Keep an FDE-developed connector out of the certified catalogue until its tests are source-versioned and reproducible.

8. Measure onboarding using:

   • engineer hours to trustworthy activation;
   • source CPU/I/O consumed;
   • manual decisions required;
   • reconciliation errors;
   • replay and recovery duration;
   • unclassified fields;
   • authorization mismatches;
   • delete propagation time.

9. Use AI only to propose mappings, descriptions and tests. Each proposal must retain its evidence, confidence and reviewer decision.

10. Maintain connector maturity levels such as experimental, shadow-verified, production-verified and certified.

Connections to other domains

• API/ABI evolution: Wire compatibility is analogous to ABI compatibility; semantic behavior still requires contract tests.
• Event sourcing: CDC becomes dependable when source positions, replay and deterministic materializations are explicit.
• Software supply chains: A connector should have a signed build identity, dependency inventory, configuration digest and reproducible conformance results.
• Zero trust: Every discovery, profile, snapshot and CDC activity is a separately authorized workload.
• SRE: Freshness, lag, reconciliation, log-retention headroom and reseed time are service-level indicators.
• GitOps: Contract changes should undergo diff classification, compatibility analysis, review and promotion.
• Data protection engineering: Erasure is a distributed workflow with exceptions, deadlines and proofs—not a storage command.
• Digital twins: A source adapter behaves like a protocol adapter for a digital twin: identity and state-transition semantics matter more than physical transport.

Recommended experiments and conformance tests

Priority experiments

1. Cross-profile golden workload

   Run PostgreSQL plus one SaaS API and one file/object source through both profiles. Require identical canonical envelopes and final logical state.

2. Snapshot/CDC handoff experiment

   Continuously insert, update, delete, mutate keys and alter schemas during a large snapshot. Kill connectors at every handoff phase. Acceptance: zero missed committed transactions; duplicates allowed only when replay produces identical state.

3. Schema evolution arena

   Test optional/required fields, defaults, precision changes, timezone changes, rename with/without stable IDs, enum additions, nested structures and semantic unit changes. Compare registry outcome with Dataheart’s five-layer classification.

4. Deletion closure drill

   Delete a subject represented in snapshot files, CDC, materialized tables, time travel, search/vector indexes, caches, exports and backups. Measure closure time and inspect the evidence ledger for retained personal data.

5. Onboarding automation study

   Compare manual contract authoring with AI-assisted drafting. Measure engineer time and false acceptance rates for keys, classifications, business definitions and deletion rules.

6. Connector portability drill

   Export the complete source bundle from Microsoft-native operation and resume in the open profile, then reverse the direction. No hidden vendor checkpoint should be required.

Minimum conformance suite

A certified connector should pass:

• Metadata discovery without reading unrestricted data.
• Secret and log-redaction tests.
• Type/precision/timezone round trips.
• Null, empty, missing and default-value distinctions.
• Composite, mutable, nullable and absent key cases.
• Snapshot consistency under concurrent writes.
• Snapshot/CDC overlap and reconciliation.
• Transaction and per-key ordering.
• Duplicate, replay and out-of-order delivery.
• Crash before and after checkpoint commit.
• Lost source-log position and controlled reseed.
• Insert/update/delete/soft-delete/truncate/drop/rename.
• Schema change during snapshot.
• Backward, forward and transitive schema mutations.
• Rare data-quality violation outside the profiler sample.
• Source-to-target authorization parity.
• Lineage completeness and evidence integrity.
• Contract and checkpoint export/import.
• Lag, source-load, storage and reseed budget enforcement.
• Connector retirement and credential revocation.

A strong initial acceptance bar is:

• zero missing committed source operations;
• deterministic state after any replay;
• deletion closure within the contract SLO;
• no unexplained authorization expansion;
• reconciliation hashes matching at activation;
• recovery and source-cost measurements below declared budgets.

Sources

1. Open Data Contract Standard 3.1.0 — 2025-12-08.
2. W3C Data Catalog Vocabulary 3 — Recommendation, 2024-08-22.
3. OpenLineage object model — current specification, accessed 2026-08-23.
4. Confluent schema evolution and compatibility — last published 2026-08-08.
5. Debezium PostgreSQL connector — stable 3.6 documentation, accessed 2026-08-23.
6. Apache Flink DataStream/Table integration — current documentation, accessed 2026-08-23.
7. Apache Iceberg specification — current specification including v2/v3 features, accessed 2026-08-23.
8. Apache Arrow ADBC — current release documentation, accessed 2026-08-23.
9. OpenMetadata profiler workflow — v1.12 documentation, accessed 2026-08-23.
10. Microsoft Power Query connector SDK — updated 2025-10-01.
11. Microsoft Fabric Open Mirroring landing format — updated 2025-09-10.
12. Fabric Azure SQL Mirroring limitations — updated 2026-02-27.
13. Microsoft Purview data-quality source connections — updated 2026-05-15.
14. Azure Event Hubs Schema Registry — updated 2025-04-30.
15. EDPB coordinated enforcement findings on erasure — 2026-02-18.