Administration overview
Running DataHub is mostly four recurring jobs: keeping it available, managing who can see what, controlling how much data is kept, and onboarding new sources.
Three decisions are hard to reverse and worth getting right early: tenant boundaries, data set boundaries, and external id conventions. Everything else can be changed later.
The three decisions to get right early
Each tenant is a fully separate set of databases. Splitting or merging tenants later means moving data between databases, so decide the boundary before onboarding.
Data sets are the unit of access control. Moving entities between them changes who can see them and requires write access to both sides.
External ids can never change or be reused. A convention agreed after 40,000 items are created is a convention you cannot apply. Set a naming policy before the first import, and the platform holds the convention for you.
The recurring work
Keeping it available
DataHub runs as three services, all of which run as multiple instances behind a load balancer and scale by simply adding instances. The backing stores are standard, well-understood systems your infrastructure team likely already runs.
Architecture, without the jargon →
Managing access
Data set permissions govern the stored data, resources, time series, events and files. They are granted through organization groups in your identity provider, and read and write are independent of each other. → Data set permissions
Users themselves live in your existing identity provider, not in DataHub. → Users and access
Controlling retention and cost
Storage is the dominant infrastructure cost, and it grows with ingestion whether or not anyone reads the data. Value types, retention and housekeeping are what keep it proportionate.
Each tenant also carries a budget of its own: how fast it may write, how much it may ingest in a day, and how many live connections it may hold, plus optional ceilings on how large it may grow in total. The defaults are conservative, and raising one for a single tenant is a database update rather than a deployment.
Onboarding new sources
Each new source system means: deciding which data set it lands in, configuring the integration that feeds it, giving its service account the narrowest permissions that work, and making sure what arrives connects to the model rather than sitting orphaned.
→ Change data capture · → Data sets
What you do not have to do
Worth knowing, because these are often assumed:
- You do not manage user accounts in DataHub. Identity lives in your identity provider, any system speaking the OAuth2 and OpenID Connect sign-in standards. Identity providers → DataHub authorises every request against what your provider currently says, so revoking access is a matter of removing a role or a group membership, and it takes effect within about a minute.
- You do not run database migrations by hand. They run automatically at startup, across every tenant, and starting several instances at once is safe.
- You do not need session affinity on the load balancer. All services, including the live streaming endpoints, are stateless in the way that matters. The load balancer needs WebSocket-upgrade support and nothing else.
Sizing and scaling, in one paragraph
Each of the three services scales differently and independently. The API scales with user and integration traffic, and keeping the knowledge graph in step with the model is part of its job, so no service needs to run single-instance for the graph. The consumer that lands datapoints and events scales with ingest volume. The console is a thin UI layer.
The practical consequence: you can scale ingestion without adding API capacity, and restart the UI without interrupting ingestion.
Where to go next
Standing up an evaluation stack, and what changes for production.
How multi-tenancy works, and how to decide tenant boundaries.
Identity, roles, tokens and service accounts.
Keycloak as the trusted issuer, and bringing Entra ID in behind it.
The organization groups that control read and write on stored data.
Retention, backups, housekeeping and storage cost.
The Prometheus metrics every service exposes, and which ports to scrape.
The per-tenant request, ingest and connection budgets, and how to raise one.
The three switches behind Ask AI, the model it runs on, and the settings that bound its cost.
The posture, the isolation guarantees, and what to tell a risk committee.
- Architecture, without the jargon: what the moving parts are
- Change data capture: connecting source databases directly
- Frequently asked questions: including the architecture decisions