Skip to main content

Demand forecasting

At a glance

Effort: ~1 hour · You'll build: a lag + calendar feature set and a gradient- boosted forecaster · Stack: the SDK for data in/out, plus pandas, numpy, scikit-learn.

Forecasting turns history into a plan: how much power a feeder will draw tomorrow, how many units a store will sell next week, how much load a network will carry at the busy hour. The pattern is the same regardless of domain — past values plus calendar effects predict the next ones — and the forecast becomes a new series you can chart against actuals and alert on.

New to machine learning?

No background needed. Skim the gentle primer for the ideas in plain language — model, feature, training, and the algorithm itself — and use Generate sample data for a sandbox to run this against.

Need data to run this?

This reads a load history that already exists. Generate a sandbox first — section G ingests the feeder_f12_load_mw curve this build forecasts.

1. Load the history​

Pull a long, regularly-sampled history of the quantity you want to forecast.

import datahub_sdk, numpy as np, pandas as pd

client = datahub_sdk.DataHubClient.from_env()

rf = datahub_sdk.RetrieveFilter(
    ts="feeder_f12_load_mw",
    start=pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=120),
    end=pd.Timestamp.now(tz="UTC"),
    aggregates=["avg"], granularity="1h", limit=100_000)    # hourly history

points = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()
demand = pd.Series([float(p.average) for p in points],
                   index=pd.to_datetime([p.timestamp for p in points])).sort_index()
demand = demand.asfreq("1h").interpolate()        # regular hourly grid

2. Build lag + calendar features​

Demand depends on its recent past (an hour ago, a day ago, a week ago) and on the calendar (hour of day, day of week, season). Encode both into a supervised frame whose target is the value to predict.

def build_frame(series):
    df = pd.DataFrame({"y": series})
    for lag in (1, 24, 168):                 # 1h, 1 day, 1 week
        df[f"lag_{lag}"] = series.shift(lag)
    df["roll_24"] = series.shift(1).rolling(24).mean()
    df["hour"] = df.index.hour
    df["dow"] = df.index.dayofweek
    df["month"] = df.index.month
    return df.dropna()

frame = build_frame(demand)
FEATURES = [c for c in frame.columns if c != "y"]

3. Train the forecaster​

A gradient-boosted regressor handles the non-linear calendar interactions well and needs little tuning. Hold out the most recent slice to check it honestly.

from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

split = int(len(frame) * 0.85)
train, test = frame.iloc[:split], frame.iloc[split:]

model = HistGradientBoostingRegressor(max_iter=400, learning_rate=0.05)
model.fit(train[FEATURES], train["y"])

mae = mean_absolute_error(test["y"], model.predict(test[FEATURES]))
print(f"holdout MAE: {mae:.2f} MW")

4. Forecast the horizon and write it back​

Future lags aren't known, so forecast recursively — predict one hour, feed it back in as the next hour's lag, and step forward. Then publish the forecast as its own series alongside the actuals.

HORIZON = 48      # hours ahead
history = demand.copy()
preds = {}

for step in range(HORIZON):
    t = history.index[-1] + pd.Timedelta(hours=1)
    row = {
        "lag_1":   history.iloc[-1],
        "lag_24":  history.iloc[-24],
        "lag_168": history.iloc[-168],
        "roll_24": history.iloc[-24:].mean(),
        "hour": t.hour, "dow": t.dayofweek, "month": t.month,
    }
    yhat = model.predict(pd.DataFrame([row])[FEATURES])[0]
    preds[t] = yhat
    history.loc[t] = yhat        # feed the prediction forward

forecast = pd.Series(preds)
client.timeseries.create([datahub_sdk.TimeSeries(
    external_id="feeder_f12_load_mw_forecast", name="Feeder F12 load — 48h forecast", unit="mw", value_type="float")])
client.timeseries.insert_from_lists(
    timestamps=forecast.index, values=forecast.to_numpy(), ts="feeder_f12_load_mw_forecast")

With the forecast stored as a series, a threshold rule on it becomes a predictive alert — raise a capacity_risk event when the forecast, not the actual, is set to exceed a limit, hours before it happens.

Where to take it further​

  • Exogenous drivers. Add weather, price, or a promotions flag as features — usually the biggest accuracy win.
  • Prediction intervals. Train quantile models (loss="quantile") for a P10/P90 band instead of a single line.
  • Backtest rolling. Re-fit on a sliding window and score each step to track drift over time.

Further reading​

See also​