Change data capture
Change data capture (CDC) connects a source system's database directly, so changes flow into DataHub as they happen, without anybody writing an export.
It is the fastest route in when a source database is reachable and its structure is stable. It is the wrong route when the source is a vendor system whose structure changes on upgrade.
Change data capture is on the roadmap and the screens are not yet functional. The console has a CDC area and the data model behind it exists, but creating and testing an integration is not wired up end to end, so this page describes the intended behaviour rather than something you can use today.
To land data from a source system now, use the API or the SDK. Developer documentation
What it does
Most source systems have no useful integration interface, but almost all of them have a database underneath. CDC tails that database's change log and turns each change into DataHub data: rows become resources, signals become time series, and occurrences become events.
The value is that it is continuous and hands-off. Once configured, there is no scheduled export, no file drop, no script anybody has to remember to run, and no window where DataHub is out of date because last night's job failed quietly.
Setting one up
Hostname, port, username and password for the source database, plus the publication name the source exposes its changes under.
There is a test connection action. Use it. A CDC integration that cannot reach its source fails quietly rather than loudly, so an untested one can look configured and deliver nothing.
Add only the tables you have a use for. Pulling everything because it is easy produces a faithful copy of somebody else's schema, which is the problem you adopted DataHub to escape.
How often captured data is written down, in milliseconds. Shorter means fresher data and more write activity; longer means less overhead and more lag.
Which data set the incoming data belongs to, and therefore who can see it.
Existing integrations are listed under CDC Integrations, with a search box and a detail view per integration.
When CDC is the right tool
| Use CDC when | Use something else when |
|---|---|
| The source database is reachable from the platform | The source is only accessible through an application API |
| Its structure is stable across upgrades | The vendor reshapes tables on every release |
| You need changes continuously, not nightly | A daily batch genuinely suffices |
| Nobody owns an integration you could extend | You already have a supported integration path |
For a vendor system with a volatile internal structure, an API-based integration that you control is more robust, it breaks loudly when something changes, rather than silently capturing the wrong columns. Your integration team can build that against the platform's API. Developer and SDK documentation →
Landing CDC data well
This is the part that decides whether a CDC integration is an asset or a liability.
CDC gets data in; it does not decide what the data means. Plan the mapping before you switch an integration on:
- Which rows become resources, and what label each one gets
- Which columns become time series, with what unit and value type
- Which changes become events, and which resources each event concerns
- What the external id is, usually the source system's own primary key, which is stable and unique by construction
- Which resources each incoming record relates to. Landing data without linking it is contextualization left undone, and it is what turns an integration into a second silo
Getting the last one right matters more than the rest, because it is the field you cannot change afterwards. A source system's primary key is normally an excellent external id; a row's display name is normally a poor one.
Configure one table, let it land a few hundred rows, and look at what arrived in the console before adding the rest. Mapping problems are obvious at 200 rows and expensive at 200,000.
Security
A CDC integration holds credentials to a source system, so treat it as you would any other integration account:
- Give it a read-only account on the source database. CDC never needs to write.
- Give it write and not read on the DataHub side, if it does nothing but feed data in. Why →
- Rotate its credentials through your normal process.
When to involve a developer, or an agent
Configuring a CDC integration is administrator work and is done entirely in the console. Writing a custom integration, for a source with no reachable database, or one needing transformation on the way in, is developer work.
It is also work AI agents are unusually good at: exploring an unfamiliar source system, proposing the mapping onto the four building blocks, writing the integration against the API, and repairing it when the source changes shape. In an organisation of agents this is standing worker-agent work, running the integration and watching its data quality, not a one-off build. Since integration effort is normally the biggest driver of how long a first project takes, this is often the fastest payback available. AI agents and integrations →
Developer and SDK documentation →- Contextualization: linking incoming data to the things it describes
- Data sets: deciding where incoming data lands
- The resource graph: the model CDC data should connect into
- Time series: units and value types for incoming signals
- Events: what discrete source-system changes become
- Users and access: service accounts for integrations