Dated futures curves: ingest, store, and analyze real term structure#21
Merged
Conversation
added 6 commits
June 14, 2026 16:17
Store the dated contract strip per commodity (daily settlements) in a new daily_contracts table keyed by (Commodity, ContractMonth, Datetime), with an idempotent ON CONFLICT upsert mirroring intraday_prices. Add DataFetchConfig knobs for the dated strip (enabled, lookback, contract count).
Enumerate forward monthly NYMEX contracts (CL/BZ/NG roots, month codes F-Z) and fetch each contract's recent daily bars, tagging Commodity and ContractMonth. A single empty/failed contract is logged and skipped. Expose a pure _dated_tickers helper for unit testing and a fetch_dated method on the DataSource adapter.
Real forward-curve analytics over dated contracts: curve_as_of, annualized term_structure_slope, per-pair roll_yield, and curve shape classification. Point the gated continuous-proxy FuturesAnalyzer methods at this new capability.
Add run_dated_ingestion (fetch -> upsert -> checkpoint), run it alongside intraday ingestion in the scheduler gated on dated_enabled, expose a /curve route, and report a daily_contracts row count from /healthz.
Add storage idempotence/PK tests, DatedFuturesAnalyzer backwardation/contango cases, pure ticker-enumeration unit tests, source/pipeline fetch_dated coverage, a sample_daily_contracts fixture, and README notes.
roll_yield now emits null (not silent inf) for same- or inverted-maturity adjacent pairs. Daily contract bars are tz-naive from the vendor, so stamp them UTC explicitly rather than relying on the session TimeZone at insert.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dated futures curves: ingest, store, and analyze real term structure
energex stored only the continuous front-month proxies (
CL=F,BZ=F,NG=F)in a single
intraday_pricestable, so term structure, roll yield, and basiswere structurally meaningless (see ASSESSMENT.md). This adds the dated contract
strip as a first-class dimension, so the platform can serve genuine forward
curves and cross-sectional term-structure analytics.
What's new
daily_contractstable (one daily settlement per datedcontract), keyed by
(Commodity, ContractMonth, Datetime)with an idempotentON CONFLICTupsert, mirroringinsert_intraday_data(column validation,canonical order, TIMESTAMPTZ/UTC).
reset()and init updated; migration pathuntouched.
EnergyDataFetcher.fetch_dated_contracts/fetch_all_datedenumerate the next N monthly NYMEX contracts per commodity (CME month codes,
year rollover handled), fetch each via the existing retry/timeout path, and
tag rows with
Commodity/ContractMonth. A pure_dated_tickershelpermakes the enumeration unit-testable without network. Daily bars are stamped
UTC explicitly.
DataSource.fetch_datedadded across base / yfinance /stubs;
run_dated_ingestionin the pipeline.DatedFuturesAnalyzerwithcurve_as_of,term_structure_slope(annualized front-to-back),roll_yield(annualizedadjacent-pair, null-guarded on degenerate spans), and
shape(backwardation / contango / flat). Sign-consistent across all three; correct
Polars 1.x idioms (no
GroupBy.mutate, no list-vs-scalar comparisons). Thecontinuous-proxy
FuturesAnalyzer's gated methods now point here./curveendpoint (commodity + optionalasof, 404 on empty),dated ingestion wired into the scheduler (gated on
dated_enabled, failureisolated from startup), and
/healthzreports adaily_contractscount.DataFetchConfig.dated_enabled/dated_lookback_days/dated_contract_count.Testing
analyzer math with asserted backwardated and contango cases, ticker
enumeration incl. year rollover, and stubbed source/pipeline coverage.
ruffclean on newcode; new code adds no
mypyerrors.Scope
Generic platform capability only; no commodity-specific market views.