Unified bitemporal energy-data platform (S1)#22
Merged
Conversation
added 30 commits
June 16, 2026 05:58
…ram) World-class rebuild design, adversarially stress-tested: - hexagonal core (zero framework imports) + Dagster/FastAPI/LangGraph drivers - bitemporal ArcticDB-on-MinIO store of record with per-symbol version-index vintage addressing (not datetime-as_of), crash-safe atomic commit protocol, and a vintage_reconstructed honesty flag for inline-revised sources - three revision modes (degenerate/merge/replace); single _to_polars adapter - ERCOT Public REST (not Data Miner 2), monthly nClimDiv weather - futures migrated off the DuckDB/APScheduler island onto Dagster+ArcticDB - always-on docker-compose (Postgres instance, Neo4j, MinIO) on the Mac Studio host
Bite-sized TDD tasks (34 tasks / 199 steps) for the unified data platform foundation: PRE-S1 infra/deps/CI, Phase 0 discovery spikes (EIA freeze, MinIO smoke, vintage addressing experiment), Phase 1 scaffold + framework-boundary test, Phase 2 crash-safe bitemporal storage core with 7 gate tests, Phase 3 pandera quality gate. Chrome verification at the MinIO console and Dagster UI milestones.
…ith dev/full profiles
…nIO arctic policy
added 28 commits
June 18, 2026 05:53
…m, NOAA, FRED) Add ScheduleDefinitions for the working data assets, each targeting a partitioned asset job and materializing the most recent partition per tick: - eia_gas_storage Thu 10:35 ET (35 10 * * 4) - eia_petroleum_status Wed 10:35 ET (35 10 * * 3) - noaa_degree_days monthly day 6 (0 12 6 * *) - fred_spot_prices weekday 09:30 ET (30 9 * * 1-5) All set execution_timezone=America/New_York. intraday_futures_bars stays manual (yfinance blocked). dagster definitions validate passes.
…inIO wiring - Dockerfile: parametrize installed extras via EXTRAS build arg; create /opt/dagster/home owned by nonroot so the named volume inits writable. - compose: dagster-webserver/daemon now build a dedicated amd64 image (energex:dagster) with orchestration+storage+quality+graph extras (arcticdb ships no linux/arm64 wheel -> run under emulation). Pass MINIO_ENDPOINT=minio:9000 (in-network), scoped ArcticDB service-account creds, NEO4J_* and the .env connector keys (EIA/FRED). Daemon loads the workspace explicitly via -w. - workspace.yaml: working_directory -> /app (the image WORKDIR; package is installed in the venv, not /opt/dagster/app).
The slim runtime image has no curl, so the curl-based /server_info probe always failed and the (functional) webserver reported unhealthy. Switch to the same python urllib probe the energex service uses.
…-on) Set default_status=RUNNING so the daemon auto-starts the EIA/NOAA/FRED schedules on an always-on deployment instead of leaving them STOPPED.
Switch from MIT to the PolyForm Noncommercial License 1.0.0 so noncommercial use (personal/research/education/nonprofit/government) stays free while commercial use requires a separate paid license, reserving a future commercial offering. - LICENSE: verbatim PolyForm Noncommercial 1.0.0 + Required Notice copyright, commercial-licensing contact, and a note that prior MIT releases stay MIT. - CITATION.cff: attribution metadata so users can cite the project. - README: source-available license section (not OSI 'open source'). - pyproject: license id + 'Free for non-commercial use' classifier. Note: this binds this and future releases; MIT grants already made are irrevocable for previously published versions.
…ed docstring ArcticDB 6.18.1's S3 backend requires creds embedded in the connection URI (the AWS env credential chain returns 403 against MinIO, verified). Keep EnvVar-sourced creds but: correct the docstring (it falsely claimed secrets never live in URI literals), and wrap Arctic() so a connection failure raises a redacted StorageError (endpoint/bucket + exception type only) -- preventing the cred-bearing URI from leaking via a traceback. Resolves the sole confidence>=8 finding from the branch security review.
…case-conflict Strengthen the local secret/safety gates alongside the existing gitleaks hook.
Cross-device (laptop + phone) frontend design brief for interacting with Energex data, inspired by the Orano Innovations immersive experience (Immersive Garden). Captures the open-core boundary: the app ships from a separate private repo and consumes only the S2 read API (as_of first-class); no frontend code lands in this repo. Covers the cinematic-shell / instrument-grade-core split, design system, responsive PWA strategy, the as_of time-travel headline interaction, the API contract seam, tech stack, phasing, and risks.
Remove the superseded ASSESSMENT.md audit, the internal docs/superpowers/ design artifacts (preserved in git history; architecture moves to the docs site), the original energex-ingestion-spec.md, and the phase-0 discovery spike scripts. Rewrite every dangling ASSESSMENT R## reference in code comments/docstrings/error messages to be self-contained, and drop the now-moot ASSESSMENT.md entries from .dockerignore and the CI mypy advisory.
- schemas: _business_days_between now localizes tz-naive timestamps to UTC before converting, instead of crashing on tz-naive freshness input. - orchestration/checks: derive as_of from the committed data (max read-back as_of) so the read-back quality gate uses the same knowledge time the asset wrote, avoiding false freshness failures. - core/storage: _to_polars guards tz_localize on Datetime/valid_time, only localizing when the column is tz-naive. - ci: connector-gate now also runs the EIA, weather, and readapi tests (adds the service extra for the readapi TestClient). - pyproject: add Python 3.11/3.12/3.13 classifiers (matches requires-python). - config/exceptions shims: replace dated "122 pre-S1 tests" wording with an evergreen backward-compatible-imports note. - docs/testing: give test_api_contract its own "Public API contract" row instead of misclassifying it as a connector test. - core/exceptions: QualityGateError.failures typed as pd.DataFrame. - connectors/weather: explicit empty-frame dtypes in _parse_file. - data_fetcher: use datetime.timezone.utc instead of pytz.UTC.
The platform is now Dagster ingestion + the S2 readapi serving seam, so the old serving/CLI stack is dead. Removed (no retained code imports them): - src/energex/service/app.py (legacy DuckDB FastAPI app) - src/energex/service/scheduler.py (in-process APScheduler) - src/energex/service/pipeline.py (DuckDB ingestion pipeline) - src/energex/main.py (CLI entry point) - tests/test_service.py, tests/test_main.py (only exercised the above) Kept src/energex/service/readapi.py (the S2 seam) and refreshed the service package + readapi docstrings to match. - docker-compose.yml: dropped the legacy `energex` service and its now-orphaned `energex-data` volume; the `api` (readapi) service is the serving layer. - Dockerfile: default CMD/EXTRAS now target readapi (service+storage) instead of the removed app module. - core/exceptions.py: removed the dead VintageImmutableError (never raised) and its assertion in test_core_exceptions. database.py / data_fetcher.py / sources/* / analysis/* / visualization/* are intentionally retained (still imported by the public package surface and tests).
…erpreter Fix two CI-only failures (local 3.12 + --all-extras masked them): - conftest.py guards its top-level 'import arcticdb' so jobs without the storage extra (the pandera quality gate) can still collect their non-storage tests. - test_dagster_definitions_validate_cli invokes dagster via sys.executable -m dagster instead of 'uv run dagster', which re-resolved to a fresh default env (no extras) and failed under the 3.11 matrix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilds Energex from a single-file DuckDB/yfinance library into a self-hosted, always-on bitemporal energy-data platform (point-in-time-correct store of record), shipped as S1 of a decomposed program (S2 serving · S3 agent · S4 frontend reserved).
Architecture
energex.core, zero framework imports, CI-enforced): connectors, ArcticDB-on-MinIO storage, pandera quality gate, symbology, schemas, config.as_of), crash-safe atomic commit, three revision modes (degenerate / merge / replace), and avintage_reconstructedhonesty flag (EIA/ERCOT revise inline, so true point-in-time only accrues going forward).energex.orchestration, the only Dagster importer): assets, asset-checks, partitions, 4 always-on schedules, reconcile/GC; Postgres-backed instance.fundamentals.eia), NOAA nClimDiv weather (weather), FRED benchmark spot prices (prices.spot); yfinance intraday futures dev-only (prices.intraday).energex.service.readapi): read-only FastAPI withas_offirst-class for the (separate, private) frontend;frontend/is gitignored + wired in compose.docker compose --profile full up -d→ MinIO + Dagster (webserver/daemon/Postgres) + Neo4j + the read API; verified running with real data ingesting on schedule.Quality & security
Project
website/): overview, architecture, quickstart, connectors, storage/point-in-time, orchestration, deployment, operations, testing, roadmap, frontend integration.