Forecasting in real-world financial market data

In this tutorial, we will load data from the Jane Street Kaggle competition and apply advanced neural forecasting models. We’ll evaluate the performance of these models and understand their suitability for this task.

WARNING

This tutorial requires a large amount of RAM. It is currently not yet possible to stream load the data. Experiments are conducted on a machine with 128GB of RAM.

Setup

First, we import the necessary libraries and configure logging

[1]:

import logging
import torch
from neuralforecast import NeuralForecast
from neuralforecast.losses.numpy import mae, mse
from neuralforecast.models import NBEATS, NHITS, BiTCN, NBEATSx
from fintorch.datasets.marketdata import MarketDataset

# Set up logging
logging.basicConfig(level=logging.INFO)
torch.set_float32_matmul_precision("medium")

/home/marcel/Documents/research/FinTorch/.conda/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
2024-11-30 12:04:18,606 INFO util.py:154 -- Missing packages: ['ipywidgets']. Run `pip install -U ipywidgets`, then restart the notebook server for rich notebook output.
2024-11-30 12:04:18,697 INFO util.py:154 -- Missing packages: ['ipywidgets']. Run `pip install -U ipywidgets`, then restart the notebook server for rich notebook output.

Defining parameters

We define hyperparameters such as the input size (past time steps for prediction), forecast horizon, and batch sizes:

[2]:

# Hyperparameters
input_size = 2  # Number of past time steps used for prediction
days = 1  # Number of days to forecast
steps_per_day = 5  # Number of steps per day
horizon = days * steps_per_day  # Forecast horizon
max_steps = 100  # Max training steps

# Validation and test set sizes
val_size = horizon
test_size = horizon

# Batch sizes
batch_size = 16
windows_batch_size = 16
valid_batch_size = 16
batch_size = 1  # Single series per batch

Dataloading

We begin by loading the dataset using the MarketDataset class from the fintorch library:

[3]:

# Load dataset
df = MarketDataset("~/.fintorch_data/marketdata-janestreet/")
df = df.data.collect()

# Number of unique series
n_series = len(df["unique_id"].unique())
print(f"Number of unique series: {n_series}")

INFO:root:Load train market data
INFO:root:Downloading dataset from Kaggle

Downloading jane-street-real-time-market-data-forecasting.zip to ~/.fintorch_data/marketdata-janestreet/raw

100%|██████████| 11.5G/11.5G [07:15<00:00, 28.3MB/s]

INFO:root:Processing: apply transformations to train market data
INFO:root:Processing: apply transformations to train market data

Number of unique series: 39

This code initializes the dataset from the specified path and collects it into a DataFrame for further processing.

Models

We utilize the NeuralForecast library to implement four models: NHITS, BiTCN, NBEATS, and NBEATSx. These models are well-suited for time series forecasting due to their unique architectures:

NHITS: Builds upon NBEATS by specializing its outputs in different frequencies of the time series through hierarchical interpolation and multi-rate input processing, enhancing long-horizon forecasting accuracy.
BiTCN: Employs bidirectional temporal convolutional networks to capture both past and future dependencies in time series data, making it effective for sequential data modeling.
NBEATS: A deep neural architecture with backward and forward residual links, capable of modeling trend and seasonality components in time series data.
NBEATSx: Extends NBEATS by incorporating exogenous variables, allowing the model to leverage additional information for improved forecasting accuracy.

We configure each model with parameters such as input_size, horizon, max_steps, and batch sizes, and specify any exogenous features to be included.

[4]:

# Initialize models
models = [
    NHITS(
        input_size=input_size,
        h=horizon,
        futr_exog_list=["feature_00", "feature_01", "feature_02"],
        scaler_type="robust",
        max_steps=max_steps,
        windows_batch_size=windows_batch_size,
        batch_size=batch_size,
        valid_batch_size=valid_batch_size,
    ),
    BiTCN(
        input_size=input_size,
        h=horizon,
        futr_exog_list=["feature_00", "feature_01", "feature_02"],
        scaler_type="robust",
        max_steps=max_steps,
        windows_batch_size=windows_batch_size,
        batch_size=batch_size,
        valid_batch_size=valid_batch_size,
    ),
    NBEATS(
        input_size=input_size,
        h=horizon,
        max_steps=max_steps,
        windows_batch_size=windows_batch_size,
        batch_size=batch_size,
        valid_batch_size=valid_batch_size,
    ),
    NBEATSx(
        input_size=input_size,
        futr_exog_list=["feature_00", "feature_01", "feature_02"],
        h=horizon,
        max_steps=max_steps,
        windows_batch_size=windows_batch_size,
        batch_size=batch_size,
        valid_batch_size=valid_batch_size,
    ),
]

# List of model names
model_names = ["NHITS", "BiTCN", "NBEATS", "NBEATSx"]

Seed set to 1
Seed set to 1
Seed set to 1
Seed set to 1

Cross-validation

To assess model performance, we perform cross-validation using the cross_validation method of the NeuralForecast object:

[5]:

# Create NeuralForecast object
nf = NeuralForecast(models=models, freq="10s")

# Perform cross-validation
Y_hat_df = nf.cross_validation(
    df=df,
    val_size=val_size,
    test_size=test_size,
    n_windows=None,  # Expanding window
).to_pandas()

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
Missing logger folder: /home/marcel/Documents/research/FinTorch/docs/tutorials/marketdata/lightning_logs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name         | Type          | Params | Mode
-------------------------------------------------------
0 | loss         | MAE           | 0      | train
1 | padder_train | ConstantPad1d | 0      | train
2 | scaler       | TemporalNorm  | 0      | train
3 | blocks       | ModuleList    | 2.4 M  | train
-------------------------------------------------------
2.4 M     Trainable params
0         Non-trainable params
2.4 M     Total params
9.591     Total estimated model params size (MB)

Epoch 2:  56%|█████▋    | 22/39 [00:37<00:28,  0.59it/s, v_num=0, train_loss_step=14.50, train_loss_epoch=14.40, valid_loss=0.270]

`Trainer.fit` stopped: `max_steps=100` reached.

Epoch 2:  56%|█████▋    | 22/39 [00:37<00:28,  0.59it/s, v_num=0, train_loss_step=14.50, train_loss_epoch=14.40, valid_loss=0.270]

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

Predicting DataLoader 0: 100%|██████████| 3/3 [00:12<00:00,  0.24it/s]

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

   | Name          | Type          | Params | Mode
---------------------------------------------------------
0  | loss          | MAE           | 0      | train
1  | padder_train  | ConstantPad1d | 0      | train
2  | scaler        | TemporalNorm  | 0      | train
3  | lin_hist      | Linear        | 80     | train
4  | drop_hist     | Dropout       | 0      | train
5  | net_bwd       | Sequential    | 1.1 K  | train
6  | lin_futr      | Linear        | 64     | train
7  | drop_futr     | Dropout       | 0      | train
8  | net_fwd       | Sequential    | 3.2 K  | train
9  | drop_temporal | Dropout       | 0      | train
10 | temporal_lin1 | Linear        | 48     | train
11 | temporal_lin2 | Linear        | 85     | train
12 | output_lin    | Linear        | 49     | train
---------------------------------------------------------
4.6 K     Trainable params
0         Non-trainable params
4.6 K     Total params
0.018     Total estimated model params size (MB)

Epoch 2:  56%|█████▋    | 22/39 [00:37<00:28,  0.59it/s, v_num=2, train_loss_step=13.90, train_loss_epoch=14.70, valid_loss=0.275]

`Trainer.fit` stopped: `max_steps=100` reached.

Epoch 2:  56%|█████▋    | 22/39 [00:37<00:28,  0.59it/s, v_num=2, train_loss_step=13.90, train_loss_epoch=14.70, valid_loss=0.275]

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

Predicting DataLoader 0: 100%|██████████| 3/3 [00:12<00:00,  0.24it/s]

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]


  | Name         | Type          | Params | Mode
-------------------------------------------------------
0 | loss         | MAE           | 0      | train
1 | padder_train | ConstantPad1d | 0      | train
2 | scaler       | TemporalNorm  | 0      | train
3 | blocks       | ModuleList    | 2.4 M  | train
-------------------------------------------------------
2.4 M     Trainable params
77        Non-trainable params
2.4 M     Total params
9.534     Total estimated model params size (MB)

Epoch 2:  56%|█████▋    | 22/39 [00:37<00:28,  0.59it/s, v_num=4, train_loss_step=0.441, train_loss_epoch=0.454, valid_loss=0.234]

`Trainer.fit` stopped: `max_steps=100` reached.

Epoch 2:  56%|█████▋    | 22/39 [00:37<00:28,  0.59it/s, v_num=4, train_loss_step=0.441, train_loss_epoch=0.454, valid_loss=0.234]

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

Predicting DataLoader 0: 100%|██████████| 3/3 [00:12<00:00,  0.24it/s]

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]


  | Name         | Type          | Params | Mode
-------------------------------------------------------
0 | loss         | MAE           | 0      | train
1 | padder_train | ConstantPad1d | 0      | train
2 | scaler       | TemporalNorm  | 0      | train
3 | blocks       | ModuleList    | 2.4 M  | train
-------------------------------------------------------
2.4 M     Trainable params
77        Non-trainable params
2.4 M     Total params
9.663     Total estimated model params size (MB)

Epoch 2:  56%|█████▋    | 22/39 [00:37<00:28,  0.59it/s, v_num=6, train_loss_step=0.428, train_loss_epoch=0.467, valid_loss=0.234]

`Trainer.fit` stopped: `max_steps=100` reached.

Epoch 2:  56%|█████▋    | 22/39 [00:37<00:28,  0.59it/s, v_num=6, train_loss_step=0.428, train_loss_epoch=0.467, valid_loss=0.234]

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

Predicting DataLoader 0: 100%|██████████| 3/3 [00:12<00:00,  0.24it/s]

This approach provides insights into how each model generalizes to unseen data. We then compute evaluation metrics such as Mean Squared Error (MSE) and Mean Absolute Error (MAE) to quantify the models’ predictive accuracy.

Evaluation

For each model, we calculate the Mean Squared Error (MSE) and Mean Absolute Error (MAE) as performance metrics:

[6]:

from neuralforecast.losses.numpy import mae, mse

# Iterate over models to compute metrics
for model_name in model_names:
    # Extract true values and predictions
    y_true = Y_hat_df.y.values
    y_hat = Y_hat_df[model_name].values

    # Reshape arrays
    y_true = y_true.reshape(n_series, -1, horizon)
    y_hat = y_hat.reshape(n_series, -1, horizon)

    # Compute metrics
    mse_score = mse(y_true, y_hat)
    mae_score = mae(y_true, y_hat)

    # Print results
    print(f"\nModel: {model_name}")
    print(f"MSE: {mse_score}")
    print(f"MAE: {mae_score}")


Model: NHITS
MSE: 0.1669229418039322
MAE: 0.28757739067077637

Model: BiTCN
MSE: 0.17169682681560516
MAE: 0.3052093982696533

Model: NBEATS
MSE: 0.12209504097700119
MAE: 0.257291316986084

Model: NBEATSx
MSE: 0.13161614537239075
MAE: 0.27656877040863037

Conclusion

By leveraging these advanced neural forecasting models, we can effectively tackle the time series forecasting challenges presented in the Jane Street Kaggle competition. Each model offers distinct advantages, enabling us to capture various patterns and dependencies within the financial data, ultimately enhancing our predictive capabilities.