Skip to main content

Sequence forecasting with an LSTM

At a glance

Effort: ~2 hours · You'll build: a sliding-window sequence model that learns temporal dynamics directly · Stack: the SDK for data in/out, plus numpy, pandas, scikit-learn (scaling) and tensorflow/keras.

The gradient-boosted forecaster predicts from hand-engineered lag features. An LSTM takes a different route: it reads the raw sequence and learns the temporal structure itself — useful when the dynamics are non-linear and the right lags aren't obvious, like an oil well's production decline, where rate, pressure and water-cut interact over time.

We'll forecast a well's oil rate, but the pattern is the same for any signal with memory.

Need data to run this?

These steps read a series that already exists. Generate a sandbox first — section B ingests the decline curve this build forecasts.

New to machine learning?

No background needed. Skim the gentle primer for the ideas in plain language — model, feature, training, and the algorithm itself — and use Generate sample data for a sandbox to run this against.

1. Load the history​

Pull a long, regularly-sampled history and scale it — neural nets train best on values in a small range.

import datahub_sdk, numpy as np, pandas as pd
from sklearn.preprocessing import MinMaxScaler

client = datahub_sdk.DataHubClient.from_env()

rf = datahub_sdk.RetrieveFilter(
    ts="well_a12_oil_rate_bpd",
    start=pd.Timestamp.now(tz="UTC") - pd.Timedelta(days=730),
    end=pd.Timestamp.now(tz="UTC"),
    aggregates=["avg"], granularity="1d", limit=100_000)
pts = client.timeseries.retrieve_datapoints(rf)[0].get_datapoints()

rate = pd.Series([float(p.average) for p in pts],
                 index=pd.to_datetime([p.timestamp for p in pts])).sort_index().asfreq("1d").interpolate()

scaler = MinMaxScaler()
scaled = scaler.fit_transform(rate.to_numpy().reshape(-1, 1))

2. Frame it as input window → forecast horizon​

An LSTM learns "given the last LOOKBACK days, what do the next HORIZON days look like?". Slice the series into overlapping pairs: each input is LOOKBACK days, each target is the whole HORIZON that follows. Predicting the horizon in one shot (rather than one day at a time, feeding each prediction back in) avoids the error build-up that makes recursive forecasts drift off the trend.

LOOKBACK, HORIZON = 90, 90

def windows(arr, lookback, horizon):
    X = np.stack([arr[i:i+lookback, 0]         for i in range(len(arr) - lookback - horizon)])
    Y = np.stack([arr[i+lookback:i+lookback+horizon, 0] for i in range(len(arr) - lookback - horizon)])
    return X[..., None], Y          # X: (samples, LOOKBACK, 1), Y: (samples, HORIZON)

X, Y = windows(scaled, LOOKBACK, HORIZON)
split = int(len(X) * 0.85)
Xtr, Xte, Ytr, Yte = X[:split], X[split:], Y[:split], Y[split:]

3. Train the LSTM​

A single LSTM layer feeding a dense layer with one output per horizon step learns the whole forecast at once.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
from tensorflow.keras.callbacks import EarlyStopping

model = Sequential([LSTM(64, input_shape=(LOOKBACK, 1)), Dense(HORIZON)])
model.compile(optimizer="adam", loss="mse")
model.fit(Xtr, Ytr, validation_data=(Xte, Yte),
          epochs=60, batch_size=32,
          callbacks=[EarlyStopping(patience=8, restore_best_weights=True)], verbose=0)

4. Forecast the horizon and write it back​

The model outputs the whole horizon in a single call — feed it the most recent LOOKBACK days, inverse-scale the result, and publish it as its own series.

pred = model.predict(scaled[-LOOKBACK:].reshape(1, LOOKBACK, 1), verbose=0)
forecast = scaler.inverse_transform(pred.reshape(-1, 1)).ravel()
index = pd.date_range(rate.index[-1] + pd.Timedelta(days=1), periods=HORIZON, freq="1d")

client.timeseries.create([datahub_sdk.TimeSeries(
    external_id="well_a12_oil_rate_forecast", name="Well A-12 oil-rate forecast", unit="bpd", value_type="float")])
client.timeseries.insert_from_lists(
    timestamps=index, values=forecast, ts="well_a12_oil_rate_forecast")

The decline curve is now a stored series — chart it against actuals, and a threshold rule on it flags when the well is forecast to drop below its economic limit.

Where to take it further​

  • Go multivariate. Feed rate, tubing pressure and water-cut together (input shape (N, 3)) so the model uses their interaction — usually a big accuracy gain.
  • Encoder–decoder. For long horizons, a seq2seq LSTM decodes the horizon internally — more expressive than a single dense layer when the forecast shape is complex.
  • Quantiles. Train with a pinball loss for a P10/P90 band, not just a point forecast.

Further reading​

See also​