Repository navigation
Add a lakeFS + Backblaze B2 standalone sample - #295
Merged
Merged
Conversation
Runs lakeFS Enterprise with Backblaze B2 as its blockstore, and uses lakeFS Datasets to publish a curated, versioned slice of the data stored there. Object data, lakeFS commit metadata and the dataset's own backing storage all live in the B2 bucket. B2 speaks the S3 API, so the blockstore type is s3, but three settings are not the S3 defaults and it will not connect without them: a regional endpoint, force_path_style, and discover_bucket_region disabled. Datasets was chosen over Multiple Storage Backends deliberately. MSB would need a second cloud to be meaningful, and standing MinIO in for it would demo the cross-cloud claim against a local container. With Datasets the whole sample runs on B2 and nothing is simulated. The Iceberg catalog and Metadata Search were ruled out: they depend on S3 conditional writes, which B2 has in limited preview only, with broader rollout expected from Q1 2027. The README says so, including that B2 closes the connection rather than returning a clean error, which makes it easy to misdiagnose as a network fault. The license is mounted read-only from an absolute path given in a gitignored .env, so it never needs to sit inside the repository. Missing .env values fail compose immediately with a readable message instead of interpolating an empty string and failing later inside lakeFS. Verified against a real B2 account: from an empty bucket and a fresh stack the notebook runs top to bottom with zero cell errors, leaving 19 objects in B2 and a dataset at v2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Removed Docker Hub login instructions from README. This step is not needed any more.
Added back examples for NetApp ONTAP and OneBucket storage.
Contributor
|
@iddoavn I made 2 small fixes in the README. |
kesarwam
approved these changes
Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs lakeFS Enterprise with Backblaze B2 as its blockstore, and uses lakeFS Datasets to publish a curated, versioned slice of the data stored there. Object data, lakeFS commit metadata and the dataset's own backing storage all live in the B2 bucket — nothing is stored locally.
Configuring lakeFS for B2
B2 speaks the S3 API, so the blockstore type is
s3, but three settings are not the S3 defaults and it will not connect without them:Why Datasets, and not Iceberg or Multiple Storage Backends
Iceberg catalog and Metadata Search are not usable on B2 today. They depend on S3 conditional writes (
If-None-Match), which B2 has in limited preview only, with broader rollout expected from Q1 2027. Worth knowing for anyone who tries: B2 does not return a clean501— it closes the connection mid-response, so clients report a transport error and it looks like a network problem. The README says this explicitly.Multiple Storage Backends would need a second cloud to mean anything, and standing MinIO in for it would demo the cross-cloud claim against a container on your laptop. With Datasets the whole sample runs on B2 and nothing is simulated.
Handling the license
Mounted read-only from an absolute path given in a gitignored
.env, so it never needs to sit inside the repository. Pairs with #294, which stops license files being committable at all.Missing
.envvalues faildocker composeimmediately with a readable message rather than interpolating an empty string and failing later inside lakeFS:Verified
Against a real Backblaze B2 account. From an empty bucket and a fresh stack, the notebook runs top to bottom with zero cell errors, leaving:
Note this sample sits under
01_standalone_examples/, which CI'scheck-notebooksworkflow ignores — it needs B2 credentials, so it cannot run there.🤖 Generated with Claude Code