Skip to content

Fix mixed bucket key types during metadata cache refresh - #3213

Merged
bghira merged 1 commit into
mainfrom
fix/3206-bucket-key-types
Sep 6, 2026
Merged

bghira merged 1 commit into
mainfrom
fix/3206-bucket-key-types

Conversation

@bghira

@bghira bghira commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Cache loading and dataset initialization converted aspect-ratio keys to floats while discovery appended string keys. This created separate 1.0 and "1.0" buckets and caused empty-bucket warnings during training.

Keep bucket keys as strings across discovery, cache loading, and sampler restore. Merge equivalent keys without losing scheduled sample occurrences, preserve audio/video/caption bucket names, and remove five duplicate float-conversion helpers plus the sampler's mixed-type lookup.

Closes #3206.

Validation:

  • Reproduced failing regressions before the fix for cache reload followed by discovery, all four metadata cache loaders, mixed-key assignment, and Webshart metadata preparation.
  • 302 tests passed, covering metadata backends, sample occurrence counts, unchanged-layout checkpoint round trips, factory setup, conditioning alignment, and VAE caching:
    .venv/bin/python -m unittest -v -f tests.test_metadata_backend tests.test_webshart_backend tests.test_huggingface_metadata_backend tests.test_sampler tests.test_dataset tests.helpers.metadata.test_video_bucket_strategy tests.helpers.metadata.test_audio_backends tests.helpers.metadata.test_caption_backend tests.test_factory_edge_cases tests.test_conditioning_split_alignment tests.test_vae tests.test_audio_sampler.TestAudioSampler.test_missing_reloaded_metadata_does_not_require_image_crop_coordinates
  • All 3 audio sampler tests passed separately: .venv/bin/python -m unittest -v -f tests.test_audio_sampler.
  • Black, isort, and git diff --check passed.

The combined sampler/factory/conditioning/audio test order triggers an existing StopIteration failure in TestAudioSampler.test_sampler_bucket_exhaustion; the same failure was reproduced against an unmodified export of main. The audio sampler suite passes in isolation.

@bghira
bghira merged commit c22737e into main Sep 6, 2026
2 checks passed
@bghira
bghira deleted the fix/3206-bucket-key-types branch September 6, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] BUCKET KEY MISMATCH

1 participant