Skip to content

Add a lakeFS + Backblaze B2 standalone sample - #295

Merged
kesarwam merged 3 commits into
mainfrom
backblaze-b2-sample
Sep 24, 2026
Merged

kesarwam merged 3 commits into
mainfrom
backblaze-b2-sample

Conversation

@iddoavn

@iddoavn iddoavn commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Runs lakeFS Enterprise with Backblaze B2 as its blockstore, and uses lakeFS Datasets to publish a curated, versioned slice of the data stored there. Object data, lakeFS commit metadata and the dataset's own backing storage all live in the B2 bucket — nothing is stored locally.

Configuring lakeFS for B2

B2 speaks the S3 API, so the blockstore type is s3, but three settings are not the S3 defaults and it will not connect without them:

blockstore:
  type: "s3"
  s3:
    endpoint: "https://s3.<region>.backblazeb2.com"
    region: "<region>"
    force_path_style: true          # B2 does not serve virtual-host style buckets
    discover_bucket_region: false   # the bucket-region probe is an AWS-ism

Why Datasets, and not Iceberg or Multiple Storage Backends

Iceberg catalog and Metadata Search are not usable on B2 today. They depend on S3 conditional writes (If-None-Match), which B2 has in limited preview only, with broader rollout expected from Q1 2027. Worth knowing for anyone who tries: B2 does not return a clean 501 — it closes the connection mid-response, so clients report a transport error and it looks like a network problem. The README says this explicitly.

Multiple Storage Backends would need a second cloud to mean anything, and standing MinIO in for it would demo the cross-cloud claim against a container on your laptop. With Datasets the whole sample runs on B2 and nothing is simulated.

Handling the license

Mounted read-only from an absolute path given in a gitignored .env, so it never needs to sit inside the repository. Pairs with #294, which stops license files being committable at all.

Missing .env values fail docker compose immediately with a readable message rather than interpolating an empty string and failing later inside lakeFS:

required variable B2_REGION is missing a value: set B2_REGION in .env - e.g. us-east-005 (see .env.example)

Verified

Against a real Backblaze B2 account. From an empty bucket and a fresh stack, the notebook runs top to bottom with zero cell errors, leaving:

repo:    backblaze-b2-demo -> s3://lakefs-samples/backblaze-b2-demo
dataset: chihuahua-images v2

backblaze-b2-demo/      13 objects
datasets/                6 objects
TOTAL                   19 objects, 311 KB

Note this sample sits under 01_standalone_examples/, which CI's check-notebooks workflow ignores — it needs B2 credentials, so it cannot run there.

🤖 Generated with Claude Code

Runs lakeFS Enterprise with Backblaze B2 as its blockstore, and uses lakeFS
Datasets to publish a curated, versioned slice of the data stored there. Object
data, lakeFS commit metadata and the dataset's own backing storage all live in
the B2 bucket.

B2 speaks the S3 API, so the blockstore type is s3, but three settings are not
the S3 defaults and it will not connect without them: a regional endpoint,
force_path_style, and discover_bucket_region disabled.

Datasets was chosen over Multiple Storage Backends deliberately. MSB would need
a second cloud to be meaningful, and standing MinIO in for it would demo the
cross-cloud claim against a local container. With Datasets the whole sample runs
on B2 and nothing is simulated. The Iceberg catalog and Metadata Search were
ruled out: they depend on S3 conditional writes, which B2 has in limited preview
only, with broader rollout expected from Q1 2027. The README says so, including
that B2 closes the connection rather than returning a clean error, which makes
it easy to misdiagnose as a network fault.

The license is mounted read-only from an absolute path given in a gitignored
.env, so it never needs to sit inside the repository. Missing .env values fail
compose immediately with a readable message instead of interpolating an empty
string and failing later inside lakeFS.

Verified against a real B2 account: from an empty bucket and a fresh stack the
notebook runs top to bottom with zero cell errors, leaving 19 objects in B2 and
a dataset at v2.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@iddoavn
iddoavn requested a review from kesarwam September 23, 2026 18:29
Removed Docker Hub login instructions from README. This step is not needed any more.
Added back examples for NetApp ONTAP and OneBucket storage.
@kesarwam

Copy link
Copy Markdown
Contributor

@iddoavn I made 2 small fixes in the README.

@kesarwam
kesarwam merged commit 6ad0684 into main Sep 24, 2026
1 check failed
@kesarwam
kesarwam deleted the backblaze-b2-sample branch September 24, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants