Repository navigation

Add adaptive metadata upload batch sizing - #659

Merged

kbolashev merged 30 commits into

mainfrom

feature/adaptive-upload

Mar 23, 2026

Merged

Add adaptive metadata upload batch sizing#659
kbolashev merged 30 commits into
mainfrom
feature/adaptive-upload

Commits on Mar 3, 2026
- Add adaptive metadata upload batch sizing
  guysmoilov
  committed
  84632f2
- Handle non-retryable metadata upload failures
  guysmoilov
  committed
  ded5007
- Allow retry shrink on partial metadata batches
  guysmoilov
  committed
  567c830
- Add retry backoff for metadata upload failures
  guysmoilov
  committed
  4a767ab
- Honor min batch size on retry failures
  guysmoilov
  committed
  0f5e9d7
- Mock retry backoff sleep in upload tests
  guysmoilov
  committed
  82e4503
- Avoid reusing known-bad upload batch size
  guysmoilov
  committed
  996b7fc
Commits on Mar 4, 2026
- Extract metadata upload retry backoff constants
  guysmoilov
  committed
  2338c94
- Align metadata upload backoff cap with schedule
  guysmoilov
  committed
  b884e9e
Commits on Mar 5, 2026
- Refactor adaptive metadata upload sizing and retries
  guysmoilov
  committed
  4c1132c
- Fix adaptive upload expected batch sequence test
  guysmoilov
  committed
  6ae0a58
Commits on Mar 10, 2026
- Extract AdaptiveBatcher into dagshub.common for reuse
  guysmoilov
  committed
  1b6356b
- Support unbounded iterables in AdaptiveBatcher
  guysmoilov
  committed
  3a944ae
- Format config.py line length (black)
  guysmoilov
  committed
  6cea20f
- Fix review issues in AdaptiveBatcher
  guysmoilov
  committed
  a52a0aa
- Make batch growth factor and retry backoff configurable, add tests
  guysmoilov
  committed
  f6dfec9
- Fix batch size stall at bad_batch_size - 1, clear stale bounds
  guysmoilov
  committed
  349c25c
- Simplify batch sizing functions with clear strategy comments
  guysmoilov
  committed
  3e0ae68
- Clear stale good/bad bounds when they become incoherent
  guysmoilov
  committed
  3c1c817
- Improve failure fallback convergence, add bad-bound clearing test
  guysmoilov
  committed
  232ebbf
- Fix retryable error check for wrapped exceptions in DataEngineGqlError
  guysmoilov
  committed
  2c44d83
- Align metadata upload tests with adaptive batching behavior
  guysmoilov
  committed
  0453a33
- Handle tail-batch retry edge case and docstring mismatch
  guysmoilov
  committed
  00294fe
Commits on Mar 11, 2026
Commits on Mar 18, 2026
- Review fixes
  guysmoilov
  committed
  946134f
- removed LEGACY_ nonsense
  guysmoilov
  committed
  841972c
Commits on Mar 22, 2026
- Refine adaptive batching search behavior
  guysmoilov
  committed
  ebde20c
Commits on Mar 23, 2026
- Show adaptive batch size progress again
  guysmoilov
  committed
  e07343c

Commits

Commits on Mar 3, 2026

Commits on Mar 4, 2026

Commits on Mar 5, 2026

Commits on Mar 10, 2026

Commits on Mar 11, 2026

Commits on Mar 18, 2026

Commits on Mar 22, 2026

Commits on Mar 23, 2026