Event-Driven Alpha: Modeling Framework

Overview

This repository implements a research-grade modeling framework for evaluating event-driven directional prediction in commodity markets (metals, energy, agricultural products).

The system integrates:

• Price-based technical features\
• Macro features\
• News embedding features\
• Event intensity conditioning\
• Volatility regime modeling\
• Walk-forward cross-validation\
• Linear and tree-based model families

The goal is to evaluate whether predictive signal is:

• Linear vs nonlinear\
• Regime-dependent\
• Event-intensity dependent\
• Horizon-sensitive

---

Architecture

Data Inputs

• Historical price data
• Macro time series data
• News headline dataset (embedded using transformer-based sentence embeddings)

Feature Engineering

• Log returns and lagged returns
• Rolling volatility measures
• Volatility regime classification
• News embeddings (fold-safe dimensionality reduction)
• Event intensity filtering (percentile-based)
• Optional interaction features

Leak-sensitive transforms (for example PCA over embedding features) are fit inside each walk-forward fold using train data only.

---

Modeling Families

The framework supports multiple model types via configuration:

1. Ridge Regression (Linear Baseline)

• Expanding-window walk-forward CV
• Directional prediction via regression sign
• Serves as linear benchmark

2. XGBoost (Tree-Based)

• Binary classification objective
• Captures nonlinear interactions
• Expanding-window CV

3. LightGBM (Tree-Based)

• Gradient boosting framework
• Fast and efficient tree construction
• Suppressed training verbosity for clean experimentation

All models share:

• Identical CV geometry\
• Identical target construction\
• Identical evaluation metric

This ensures fair cross-model comparison.

---

Evaluation Methodology

Walk-Forward Cross-Validation

• Expanding training window
• Strict temporal ordering
• No look-ahead bias

Primary Metric

Directional Accuracy (DA):

Predicted direction vs realized forward return sign.

Results are stored as:

• Full per-configuration results
• Per-model summary results
• Final cross-model comparison table

Publication-Hardening Defaults

• Embedding PCA is fit on train folds only and then applied to test folds.
• Event-intensity thresholds (Top50...Top90) are calibrated on the initial train window, then applied forward.
• Model identifiers preserve news provenance: GlobalNews__... vs AssetNews__....
• Full result tables now include n_test_obs, Wilson 95% confidence intervals, and one-sided p-values against a 50% directional baseline.
• Ridge, XGBoost, and LightGBM use the same expanding-window geometry.

---

Event-Intensity Gradient

The framework evaluates performance across percentile-based event intensity buckets:

• Full sample
• Top percentiles (e.g., Top 50, 60, 70, 80, 90)

This allows testing whether predictive power concentrates during high-information regimes.

---

Regime Modeling

Optional volatility regime interaction modeling is supported:

• Regime feature interaction
• Regime-specific training
• Conditional performance evaluation

---

Output Structure

Each model run produces:

• results_<model>_full.csv\
• results_<model>_summary.csv

A final comparison script merges model families into:

• final_model_comparison.csv

---

Key Questions

1. Is predictive structure linear or nonlinear?
2. Does signal concentrate in high-intensity event regimes?
3. Does volatility regime alter predictive strength?
4. Are improvements consistent across forecast horizons?

---

Key Findings

The findings below come from prior runs and should be treated as illustrative until regenerated with the current publication-hardened pipeline.

1. Event intensity unlocks the signal. Full-sample models average ~52–58% directional accuracy. Conditioning on high-information regimes (Top 60–90%) pushes performance to 65–78% DA across metals, energy, and broader commodities.

2. Filtering beats complexity. Moving to high-intensity subsets delivers larger gains than switching model families. Signal concentration drives performance more than algorithm sophistication.

3. Nonlinearity matters — but only where it should. Tree-based models add +3–8% DA in short-horizon, high-intensity environments. Longer horizons remain largely linear and macro-driven.

4. Horizon changes the game. Short-term forecasts (3–10 periods) show sharper nonlinear effects (~68–72% DA). Longer horizons (≈20 periods) reach 70–79% DA primarily via structured linear drift.

5. Regime-aware modeling improves selectively. Volatility interaction terms produce +2–9% DA improvements in targeted contexts, while hard regime segmentation reduces statistical stability.

---

Final Best Model per Asset

Asset	Model Family	Horizon	Best Model	Directional Accuracy
Energy_1	Ridge	20	Top60_Macro	0.7867
Metal_1	Ridge	20	Top90_All+RegimeInteraction	0.7778
Metal_2	LightGBM	5	Top90_News	0.7111
Metal_3	LightGBM	10	Top80_Price	0.7000
Metal_4	Ridge	10	Top80_RegimeSpecific	0.7000
Energy_2	LightGBM	3	Top90_News	0.6889
Commodities_1	LightGBM	10	Top90_Price	0.6667
Commodities_2	Ridge	20	Top50_Macro	0.6520

---

How To Run

Setup

python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install .

Notes:

• Dependencies are managed in pyproject.toml (single source of truth).
• lstm model type requires TensorFlow, which is not installed by default.
• First run downloads embedding model artifacts from Hugging Face; this can take a few minutes.
• Setting HF_TOKEN is recommended to avoid unauthenticated rate limits/timeouts.
• For tree models, set model.compute_device to cuda (or auto) to offload training where supported.

Run with a specific configuration:

python -m eda.cli --config configs/<model_config>.yaml

Available model types:

• ridge
• xgboost
• lightgbm

After running all model families:

python analysis/final_model_comparison.py

Generate a publication-filtered summary (minimum sample size + significance threshold):

python analysis/publication_report.py --outputs-dir outputs --min-test-obs 120 --p-threshold 0.05

Troubleshooting Long Runs

• If the pipeline appears idle after dataset alignment logs, it is usually running the walk-forward experiment matrix (multiple feature sets x datasets x horizons).
• The pipeline now emits per-block timing logs (for example Price done in ...s, All+RegimeInteraction done in ...s) so progress is visible during long model runs.
• If downloads are slow or retrying, set HF_TOKEN in your environment before running.
• xgboost and lightgbm can use GPU (model.compute_device: cuda); ridge remains CPU-bound.

---

Conclusion

This repository provides a clean experimental framework for testing whether directional signals in commodity markets are primarily:

• Linear
• Nonlinear
• Conditional on event intensity
• Regime-dependent

It is structured for clarity, reproducibility, and extensibility.