High-throughput ingestion

A common job: push a steady, high-volume stream of sensor readings in so a dashboard or alerting rule can act on them. The SDK does the heavy lifting — it chunks the data into batches and sends them concurrently, retrying transient failures.

How it works

Datapoints are split into batches (default 10,000 per request — the store is optimised for large batches), sent concurrently up to a bounded in-flight limit, and transient failures (HTTP 429/5xx, network) are retried. The Java client returns an IngestResult summarising what landed and what didn't.

Ingest a million readings

Java

ingest groups datapoints by series external id and returns an IngestResult.

var client = DatahubClient.fromEnv();

// readings: Map<String, List<Datapoint>> grouped by time-series external id
IngestResult result = client.timeseries().ingest(readings,
    IngestOptions.builder()
        .batchSize(10_000)     // datapoints per request
        .parallelism(16)       // concurrent in-flight batches
        .maxRetries(3)
        .build());

System.out.printf("ingested %,d, failed %,d%n", result.succeeded(), result.failed());

Python

insert_from_lists takes whole arrays (NumPy / pandas) and batches them for you.

import numpy as np, pandas as pd

client = DataHubClient.from_env()

client.timeseries.insert_from_lists(
    timestamps=pd.date_range("2026-01-01", periods=1_000_000, freq="s", tz="UTC"),
    values=np.random.rand(1_000_000),
    ts="engine_temperature")

Rust

insert_datapoints auto-batches large inputs (chunks above ~100k points).

use dataplatform_rust_sdk::create_api_service;
use dataplatform_rust_sdk::generic::{DataWrapper, DatapointsCollection};

let api = create_api_service();

let mut dw = DataWrapper::new();
dw.add_item(DatapointsCollection {
    external_id: Some("engine_temperature".into()),
    datapoints: readings,        // Vec<DatapointString { timestamp, value }>
    ..Default::default()
});
api.time_series.insert_datapoints(&mut dw).await?;

Tuning (Java)

The ingest knobs let you trade throughput against load on the server:

Option	Default	Meaning
batchSize	10_000	Datapoints per request.
parallelism	8	Concurrent in-flight requests.
maxRetries	3	Retries for transient failures.
failFast	false	Abort on the first failed batch instead of collecting errors.

When failFast is off, inspect result.errors() for the per-batch failures. See the Time-series reference for the full result shape.

Then chart the trend

Once the data is in, roll it up to hourly or daily buckets for a dashboard — see Query & aggregate time-series.

Ordering

Batches are sent in parallel, so there is no cross-batch ordering guarantee — which is fine for time-stamped data, since each datapoint carries its own timestamp.