Release v0.3.0: defensible calibration, fair benchmarks, and staged inference - #5
Merged
Merged
Conversation
…tion, checkpoint v2 - EarlyExitConfig: per-head temperatures mapping (exits + final), explicit enabled_exits booleans, centralized validate() re-checked at wrapper construction and checkpoint load; legacy scalar temperature broadcast deterministically - routing: disabled exits never fire (even at confidence 1.0 / entropy 0.0); each head's softmax uses its own temperature; invalid runtime temperatures fall back to 1.0 instead of a razor-sharp 1e-6 clamp - calibration: staged pipeline (collect logits once, fit per-head temperatures with convergence status, greedy grid search on cache, evaluate); exits that never fire stay disabled; empty loaders fail fast; CalibrationResult carries exit distribution, sample count, objective, method and schema version - checkpoints: format_version 2 with exit-point metadata and cross-check; deterministic v1 migration (scalar temperature broadcast, sentinel thresholds -> enabled_exits=False) with warnings; custom_ee loads via a user-supplied factory - InferenceResult/BatchedInferenceResult: computation_used renamed to estimated_backbone_flops_fraction with a read alias
- benchmark_models: one measurement core; every compared model (backbone, early-exit wrapper, static baseline, quantised variant) sees the exact same preloaded sample sequence, warmup, eval/inference_mode setup, device placement and per-iteration synchronization - explicit boundary labels (model-only vs end-to-end incl. H2D copy); loader-vs-noise input source labelled; accuracy reported alongside speed for labelled loaders; batch-1 validated as the primary mode - BenchmarkResult carries dtype, input shape, warmup, sync policy, and avg_estimated_flops_fraction (computation_used kept as read alias) - CLI benchmark + scripts use the fair runner; docs/benchmarks.json restructured with pre-v0.3 records quarantined under legacy_v0_2 and marked not comparable
- estimate_layer_flops returns FlopsEstimate (fractions + method + reliable + notes + excludes_exit_heads); per_layer_flops kept as thin wrapper - reused/multi-call leaf modules are detected via a counting probe and produce a warned, low-confidence uniform fallback instead of a precise but wrong cumulative fraction - EarlyExitWrapper computes the estimate lazily on first inference (cached), so constructing large models (ViT) is fast; probe runs on the backbone's current device and is materialised before routing flags flip - estimator explicitly excludes exit-head FLOPs and routing overhead; documented on FlopsEstimate and InferenceResult
- TegrastatsMonitor.start() clears the stale stop event and sample buffer (a restarted monitor previously collected nothing), no-ops on double start, degrades to unavailable on spawn failure, and stop() closes the child's stdout pipe - missing telemetry fields parse to None instead of invented zeros; JetsonRun documents its power as instantaneous - integrate_energy(): pure trapezoidal integration over the timed window; EnergySummary distinguishes instantaneous power, window-average power and integrated energy, and reports energy-per-inference only when a real integral and inference count exist - profiler exposes profile_with_energy; CLI profile reports the energy block and nulls for missing data; parser covers Orin-style rails
- example_args/example_kwargs for the inspection forward (single-tensor path retained); dry-run tensors created on the backbone's own device (or an explicit device) instead of blindly probing a CUDA model with CPU input - per-layer feature_extractors convert tuple/dict/odd-rank outputs into the Tensor the exit head consumes, applied in the dry run AND in the routing hook (wrapper feature_adapters) - the dry run now validates each exit layer executes exactly once and in the listed forward order; reused, missing or out-of-order exits fail with actionable errors instead of building a mis-routing wrapper - freshly built exit heads placed on the backbone's device; wrapper FLOPs estimate degrades to a warned low-confidence uniform fallback when the probe cannot run the backbone
- earlyon.staged: StageSpec/Stage/StagedModel protocol — ordered stages emitting (continuation_features, exit_logits), routing applied between stages so later stages genuinely never run - staged_model() splits Sequential backbones (custom_ee shape), refuses anything it cannot prove, and self-checks staged==eager on a probe input; tests pin equivalence across policies, per-head temperatures, enablement - docs/STAGED_DEPLOYMENT.md: protocol, scope, per-stage ONNX/TensorRT procedure for later Jetson validation (no fabricated numbers) - export_all_exits_to_onnx alias makes the static multi-output export's compute-everything semantics explicit in the API name
- README: deployment-oriented framing, batch-1 primary use case, legacy labels on pre-v0.3 numbers, routing-overhead and estimator limitations, static-baseline guidance, per-head temperature + enablement description - DESIGN_DECISIONS: per-head temperature section now matches reality; new sections on explicit enablement and estimated-FLOPs naming - docs/CALIBRATION_AND_BENCHMARK_CONTRACT.md: split discipline, pipeline stages, estimator limits, benchmark fairness rules - examples: test-label hygiene note in 01; 02 rewritten on the fair runner and energy-aware profiler; pyproject description de-hyped - CHANGELOG unreleased, RELEASE_NOTES_DRAFT.md (0.3.0, not tagged), OVERNIGHT_REPORT.md
ruff --fix (unused imports), noqa for sys.path-dependent imports, rename ambiguous lambda vars, black/isort over scripts/ and examples/ which CI's earlyon+tests scope never checked
…ng claims - decisive counterfactual: a single global temperature misroutes where per-head temperatures do not (junk overconfident head vs final classifier) - batched routing pinned to per-head temperatures (previously untested path) - calibration determinism, NaN-emitting head safely disabled, clean NaN rejection in temperature fit, single-class calibration data - staged runtime proven to skip later stages structurally (counting module) - custom_ee dict-output feature extractor - genuine v1 checkpoint fixture written by the actual pre-hardening save_wrapper (git main) + full migration/round-trip test
…xture, evidence runner - MANIFEST.in prunes tests/ from the sdist (it shipped a partial test tree) - scripts/smoke_test.py: post-install smoke for an installed wheel, run from outside the repo (import/version, model build, training+inference forwards, deterministic calibration, v2 round-trip, v1 migration, staged==eager, benchmark smoke) - tests/fixtures/v1_cifar_resnet20.pth: genuine pre-v0.3 checkpoint written by the historical save_wrapper, force-added past *.pth ignore with a provenance README (the committed migration test needs it on fresh clones) - scripts/evidence_run.py: bounded seeded CUDA evidence experiment (CIFAR-10, explicit disjoint train/temperature/calibration/test splits, epoch caps, wall-clock budget, fair 3-model benchmark incl. static MobileNetV2 baseline, JSON output; negative results preserved)
… + reproducibility - SECURITY_REVIEW.md: checkpoint trust model (weights_only + structural validation), model-code execution assumptions, provenance of datasets/ weights/the one committed fixture, subprocess boundaries, scan results - docs/MIGRATION.md: v1->v2 checkpoint mapping table and code-level changes - PR_BODY_DRAFT.md (not opened) - README: ASCII architecture diagram with the training/calibration/routing separation, checkpoint-migration section, reproducibility commands
…k, release notes audit section
- fix evidence runner OOM on 6 GB GPUs (baseline trains at batch 32, models moved off-GPU between phases, batched eval at 64) - docs/evidence/cuda_evidence.json + CUDA_EVIDENCE.md: seeded 2+2-epoch ResNet-18 EE vs full backbone vs static MobileNetV2 on identical test samples; per-head temperatures all converged and genuinely different; 1.10x throughput with WORSE median latency and better tails, 1.15% test accuracy drop vs 1% calibration budget — negatives reported as-is - README: evidence table with the honest interpretation; trained weights artifact stays local (gitignored)
- version 0.3.0 in pyproject.toml, earlyon/__init__.py, README badge and citation - CHANGELOG: unreleased section dated as 0.3.0 (2026-07-22) - RELEASE_NOTES_DRAFT.md finalized as the canonical v0.3.0 release text (limitations, migration, bounded CUDA evidence with negatives preserved) - PR body header finalized
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v0.3.0: defensible calibration, fair benchmarks, and staged inference
What this PR is
A correctness and methodology release. No new backbones, no new headline
claims — the calibration math, the measurement methodology, the
serialization contract and the public claims are made defensible, and every
fix is pinned by tests (261 → 265+ tests, 95% coverage, strict mypy).
The four core defects fixed
the final classifier's logits and reused at every exit, though each exit
head is a differently miscalibrated classifier. Now: per-head fit with
convergence status, per-head application at routing time. A test proves a
single global temperature misroutes where per-head does not.
(confidence 1.0 / entropy 0.0) while routing compares with
>=/<=, soa float32-saturated softmax fired a "disabled" exit. Now: explicit
enabled_exitsbooleans; sentinel-carrying v1 checkpoints migrate.temperature was clamped to
1e-6, producing an artificially razor-sharpsoftmax. Now: centralized
EarlyExitConfig.validate()raises; runtimeguard falls back to the no-op 1.0.
on different inputs and boundaries. Now:
benchmark_modelsfeeds everycompared model the identical preloaded sample sequence with identical
warmup/boundary/sync; old records quarantined as
legacy_v0_2.Also in this PR
CalibrationResult(enablement, exitdistribution, sample count, honest
greedy-coordinate-gridmethod name).format_version: 2+ deterministic v1 migration, testedagainst a genuine v0.2-written fixture;
factory=loading for custommodels.
estimated_backbone_flops_fraction(alias kept): lazyFlopsEstimatewith provenance, reused-module detection, exit-head cost disclosed.
integrated energy kept distinct; missing telemetry is
None.custom_ee: example args/kwargs, device inference, feature extractors,exactly-once/in-order exit validation.
proven equivalent to eager routing and proven to skip later stages.
sdist no longer ships a partial test tree.
static MobileNetV2) with disjoint train/temperature/calibration/test
splits — results in
docs/evidence/.guide, security review, honest README repositioning
("deployment-oriented", legacy labels, batch-1 scope).
Compatibility
Old checkpoints load (migrated, warned).
computation_used,fitted_temperatureand the single-model benchmark helpers remain ascompatible aliases/wrappers. Breaking edges are listed in
docs/MIGRATION.md.Test plan
pytestfull suite (CPU + CUDA host): all greenmypy earlyonstrict,ruff,black --check,isort --check-onlytwine check, clean-venv install + out-of-reposmoke (
scripts/smoke_test.py)scripts/evidence_run.py)scripts/run_benchmarks.pyontarget hardware; Jetson procedure from
docs/STAGED_DEPLOYMENT.md