Administration overview
Running DataHub is mostly four recurring jobs: keeping it available, managing who can see what, controlling how much data is kept, and onboarding new sources.
Three decisions are hard to reverse and worth getting right early: tenant boundaries, data set boundaries, and external id conventions. Everything else can be changed later.
The three decisions to get right early
Each tenant is a fully separate set of databases. Splitting or merging tenants later means moving data between databases, so decide the boundary before onboarding.
Data sets are the unit of access control. Moving entities between them changes who can see them and requires write access to both sides.
External ids can never change or be reused. A convention agreed after 40,000 items are created is a convention you cannot apply. Set a naming policy before the first import, and the platform holds the convention for you.
The recurring work
Keeping it available
DataHub runs as four services, all of which run as multiple instances behind a load balancer. Three of them scale by simply adding instances; one is deliberately single-active with a standby. The backing stores are standard, well-understood systems your infrastructure team likely already runs.
Architecture, without the jargon →
Managing access
Data set permissions govern the stored data, resources, time series, events and files. They are granted through organization groups in your identity provider, and read and write are independent of each other. → Data set permissions
Users themselves live in your existing identity provider, not in DataHub. → Users and access
Controlling retention and cost
Storage is the dominant infrastructure cost, and it grows with ingestion whether or not anyone reads the data. Value types, retention and housekeeping are what keep it proportionate.
Onboarding new sources
Each new source system means: deciding which data set it lands in, configuring the integration that feeds it, giving its service account the narrowest permissions that work, and making sure what arrives connects to the model rather than sitting orphaned.
→ Change data capture · → Data sets
What you do not have to do
Worth knowing, because these are often assumed:
- You do not manage user accounts in DataHub. Identity lives in your OAuth2/OIDC provider. Identity providers → DataHub authorises every request against what your provider currently says, so revoking access is a matter of removing a role or a group membership, and it takes effect within about a minute.
- You do not run database migrations by hand. They run automatically at startup, across every tenant, with locking that makes concurrent instance startup safe.
- You do not need session affinity on the load balancer. All services, including the live streaming endpoints, are stateless in the way that matters. The load balancer needs WebSocket-upgrade support and nothing else.
Sizing and scaling, in one paragraph
Each of the four services scales differently and independently. The API scales with user and integration traffic. The consumer that lands datapoints and events scales with ingest volume. The console is a thin UI layer. The consumer that applies changes to the graph is order-sensitive and runs one-active-plus-standby rather than scaling out, which is fine, because graph changes are low volume compared with measurement ingest.
The practical consequence: you can scale ingestion without adding API capacity, and restart the UI without interrupting ingestion.
Where to go next
Standing up an evaluation stack, and what changes for production.
How multi-tenancy works, and how to decide tenant boundaries.
Identity, roles, tokens and service accounts.
Keycloak as the trusted issuer, and bringing Entra ID in behind it.
The organization groups that control read and write on stored data.
Retention, backups, housekeeping and storage cost.
The posture, the isolation guarantees, and what to tell a risk committee.
- Architecture, without the jargon: what the moving parts are
- Change data capture: connecting source databases directly
- Frequently asked questions: including the architecture decisions