Skip to content

[SeaweedFS] Migrate from MinIO to SeaweedFS as the S3-compatible storage backend #245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shay79il wants to merge 1 commit into
base: development
Choose a base branch
from
Open

[SeaweedFS] Migrate from MinIO to SeaweedFS as the S3-compatible storage backend #245

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,17 @@

This repo contains the Helm charts for the MLRun Community Edition (CE) - a full open source MLOps stack.

The Open source MLRun ce chart includes the following stack:
The Open source MLRun CE chart includes the following stack:

* Nuclio - https://github.com/nuclio/nuclio
* MLRun - https://github.com/mlrun/mlrun
* Jupyter - https://github.com/jupyter/notebook (+MLRun integrated)
* MPI Operator - https://github.com/kubeflow/mpi-operator
* Minio - https://github.com/minio/minio/tree/master/helm/minio
* SeaweedFS - https://github.com/seaweedfs/seaweedfs (S3-compatible storage)
* Spark Operator - https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
* Pipelines - https://github.com/kubeflow/pipelines
* Prometheus stack - https://github.com/prometheus-community/helm-charts

## Installation

## Installation
Refer to the installation instructions in the [README](charts/mlrun-ce/README.md) of the `mlrun-ce` chart.
Refer to the installation instructions in the [README](charts/mlrun-ce/README.md) of the `mlrun-ce` chart.
2 changes: 1 addition & 1 deletion charts/mlrun-ce/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apiVersion: v1
name: mlrun-ce
version: 0.11.0-rc4
version: 0.11.0-rc5
description: MLRun Open Source Stack
home: https://iguazio.com
icon: https://www.iguazio.com/wp-content/uploads/2019/10/Iguazio-Logo.png
Expand Down
51 changes: 33 additions & 18 deletions charts/mlrun-ce/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ The Open source MLRun ce chart includes the following stack:
* MLRun - https://github.com/mlrun/mlrun
* Jupyter - https://github.com/jupyter/notebook (+MLRun integrated)
* MPI Operator - https://github.com/kubeflow/mpi-operator
* Minio - https://github.com/minio/minio/tree/master/helm/minio
* SeaweedFS - https://github.com/seaweedfs/seaweedfs (S3-compatible storage)
* Spark Operator - https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
* Pipelines - https://github.com/kubeflow/pipelines
* Prometheus stack - https://github.com/prometheus-community/helm-charts
Expand Down Expand Up @@ -52,15 +52,14 @@ kubectl --namespace mlrun create secret docker-registry registry-credentials \
```

To install the chart with the release name `my-mlrun` use the following command,
note the reference to the pre-created `registry-credentials` secret in `global.registry.secretName`,
and a `global.registry.url` with an appropriate registry URL which can be authenticated by this secret:
note the reference to the pre-created `registry-credentials` secret in `global.containerRegistry.secretName`, and a `global.containerRegistry.url` with an appropriate registry URL which can be authenticated by this secret:

```bash
helm --namespace mlrun \
install my-mlrun \
--wait \
--set global.registry.url=<registry URL e.g. index.docker.io/iguazio > \
--set global.registry.secretName=registry-credentials \
--set global.containerRegistry.url=<registry URL e.g. index.docker.io/iguazio > \
--set global.containerRegistry.secretName=registry-credentials \
mlrun/mlrun-ce
```

Expand Down Expand Up @@ -161,10 +160,10 @@ helm --namespace mlrun \
install my-mlrun \
--wait \
... other overrides ... \
--set global.registry.url=${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com \
--set global.containerRegistry.url=${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com \
--set global.containerRegistry.secretName=ecr-registry-credentials \
--set nuclio.dashboard.kaniko.registryProviderSecretName=aws-credentials \
--set mlrun.defaultDockerRegistrySecretName=aws-credentials \
--set global.registry.secretName=ecr-registry-credentials \
mlrun/mlrun-ce
```

Expand All @@ -173,19 +172,35 @@ helm --namespace mlrun \
## Usage

Your applications are now available in your local browser:
- jupyter-notebook - http://nodeipaddress:30040
- nuclio - http://nodeipaddress:30050
- mlrun UI - http://nodeipaddress:30060
- mlrun API (external) - http://nodeipaddress:30070
- minio API - http://nodeipaddress:30080
- minio UI - http://nodeipaddress:30090
- pipeline UI - http://nodeipaddress:30100
- grafana UI - http://nodeipaddress:30110
- Jupyter Notebook - http://nodeipaddress:30040
- Nuclio - http://nodeipaddress:30050
- MLRun UI - http://nodeipaddress:30060
- MLRun API (external) - http://nodeipaddress:30070
- SeaweedFS Master UI (cluster status) - http://nodeipaddress:30091
- SeaweedFS Filer UI (file browser) - http://nodeipaddress:30092
- SeaweedFS Admin UI (user/policy management) - http://nodeipaddress:30093
- Pipeline UI - http://nodeipaddress:30100
- Grafana UI - http://nodeipaddress:30010
- Prometheus UI - http://nodeipaddress:30020



**S3 API Access** (programmatic, for tools like aws-cli):
```bash
kubectl port-forward svc/seaweedfs-s3 8333:8333 -n mlrun
aws --endpoint-url=http://localhost:8333 s3 ls s3://mlrun/
```

**With Ingress enabled**, the UIs are available at:
- `https://seaweedfs-master.<namespace>.<cluster>.lab.iguazeng.com`
- `https://seaweedfs-filer.<namespace>.<cluster>.lab.iguazeng.com`
- `https://seaweedfs-admin.<namespace>.<cluster>.lab.iguazeng.com`

> **Note:**
> The above links assume your Kubernetes cluster is exposed on localhost.
> If that's not the case, the different components will be available on `externalHostAddress`
>
> For production deployments, consider enabling ingress for each service instead of using NodePorts.

## Start Working

Expand Down Expand Up @@ -278,6 +293,6 @@ Refer to the [**Kubeflow documentation**](https://www.kubeflow.org/docs/started/

This table shows the versions of the main components in the MLRun CE chart:

| MLRun CE | MLRun | Nuclio | Jupyter | MPI Operator | Minio | Spark Operator | Pipelines | Kube-Prometheus-Stack | Prometheus | Grafana |
|-----------|-------|--------|---------|--------------|------------------------------|----------------|-----------|-----------------------|------------|---------|
| **0.6.3** | 1.6.4 | 1.13.5 | 6.4.0 | 0.2.3 | RELEASE.2022-05-08T23-50-31Z | v1beta2-1.1.25 | 1.8.1 | 17.0.0 | 2.39.1 | 6.40.4 |
| MLRun CE | MLRun | Nuclio | Jupyter | MPI Operator | SeaweedFS | Spark Operator | Pipelines | Kube-Prometheus-Stack |
Copy link
Collaborator

@GiladShapira94 GiladShapira94 Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the rest of the components

|-----------|-------|--------|---------|--------------|-----------|----------------|-----------|-----------------------|
| **0.6.3** | 1.6.4 | 1.13.5 | 6.4.0 | 0.2.3 | 4.0.407 | 2.1.0 | 2.14.3 | 72.1.1 |
2 changes: 1 addition & 1 deletion charts/mlrun-ce/admin_installation_values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ mpi-operator:
deployment:
create: false

minio:
seaweedfs:
enabled: false

spark-operator:
Expand Down
13 changes: 2 additions & 11 deletions charts/mlrun-ce/non_admin_cluster_ip_installation_values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,8 @@ mpi-operator:
clusterResources:
create: false

minio:
service:
type: ClusterIP
port: 9000
nodePort: ""
consoleService:
type: ClusterIP
port: 9001
nodePort: ""
mode: standalone
replicas: 1
seaweedfs:
enabled: true

spark-operator:
enabled: false
Expand Down
13 changes: 2 additions & 11 deletions charts/mlrun-ce/non_admin_installation_values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,17 +34,8 @@ mpi-operator:
clusterResources:
create: false

minio:
service:
type: NodePort
port: 9000
nodePort: 30080
consoleService:
type: NodePort
port: 9001
nodePort: 30090
mode: standalone
replicas: 1
seaweedfs:
enabled: true

spark-operator:
enabled: false
Expand Down
6 changes: 3 additions & 3 deletions charts/mlrun-ce/requirements.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ dependencies:
- name: mpi-operator
repository: https://v3io.github.io/helm-charts/stable
version: 0.6.0
- name: minio
repository: https://charts.min.io/
version: 5.2.0
- name: seaweedfs
repository: https://seaweedfs.github.io/seaweedfs/helm
version: 4.0.407
- name: spark-operator
repository: https://kubeflow.github.io/spark-operator
version: 2.1.0
Expand Down
8 changes: 4 additions & 4 deletions charts/mlrun-ce/requirements.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ dependencies:
- name: mpi-operator
version: "0.6.0"
repository: "https://v3io.github.io/helm-charts/stable"
- name: minio
repository: "https://charts.min.io/"
version: "5.2.0"
condition: minio.enabled
- name: seaweedfs
repository: "https://seaweedfs.github.io/seaweedfs/helm"
version: "4.0.407"
condition: seaweedfs.enabled
- name: spark-operator
repository: "https://kubeflow.github.io/spark-operator"
version: "2.1.0"
Expand Down
32 changes: 10 additions & 22 deletions charts/mlrun-ce/templates/NOTES.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{- if or (.Values.jupyterNotebook.enabled) (.Values.mlrun.enabled) (.Values.minio.enabled) (.Values.pipelines.enabled) (index .Values "kube-prometheus-stack" "enabled") }}
{{- if or (.Values.jupyterNotebook.enabled) (.Values.mlrun.enabled) (.Values.seaweedfs.enabled) (.Values.pipelines.enabled) (index .Values "kube-prometheus-stack" "enabled") }}
You're up and running!

{{- if .Values.jupyterNotebook.enabled }}
Expand Down Expand Up @@ -57,41 +57,29 @@ mlrun-api.{{ .Values.global.externalHostAddress }}
{{- end }}
{{- end }}

{{- if .Values.minio.enabled }}
{{- if ne .Values.global.infrastructure.kind "aws" }}
{{- "\n" }}
Minio UI is available at:
{{- if .Values.minio.ingress.enabled }}
{{ (index .Values.minio.consoleIngress.hosts 0) }}
{{- else }}
{{ .Values.global.externalHostAddress }}:{{ .Values.minio.consoleService.nodePort }}
{{- end }}
- username: {{ .Values.minio.rootUser }}
- password: {{ .Values.minio.rootPassword }}
{{- end }}
{{- end }}

{{- if .Values.minio.enabled }}
{{- if .Values.seaweedfs.enabled }}
{{- if ne .Values.global.infrastructure.kind "aws" }}
{{- if and .Values.seaweedfs.admin .Values.seaweedfs.admin.enabled }}
{{- "\n" }}
Minio API is available at:
{{- if .Values.minio.ingress.enabled }}
{{ (index .Values.minio.ingress.hosts 0) }}
SeaweedFS Admin UI is available at:
{{- if and .Values.seaweedfs.adminService .Values.seaweedfs.adminService.ingress .Values.seaweedfs.adminService.ingress.enabled }}
{{ .Values.seaweedfs.adminService.ingress.host }}
{{- else }}
{{ .Values.global.externalHostAddress }}:{{ .Values.minio.service.nodePort }}
{{ .Values.global.externalHostAddress }}:{{ .Values.seaweedfs.adminService.nodePort | default 30093 }}
{{- end }}
- S3 credentials: minio / minio123
{{- end }}
{{- end }}
{{- end }}

{{- if and .Values.pipelines.enabled .Values.pipelines.ui.enabled -}} }}
{{- if and .Values.pipelines.enabled .Values.pipelines.ui.enabled }}
{{- if ne .Values.global.infrastructure.kind "aws" }}
{{- if not .Values.minio.ingress.enabled }}
{{- "\n" }}
Pipelines UI is available at:
{{ .Values.global.externalHostAddress }}:{{ .Values.pipelines.service.nodePort }}
{{- end }}
{{- end }}
{{- end }}

{{- if index .Values "kube-prometheus-stack" "enabled" }}
{{- "\n" }}
Expand Down
66 changes: 62 additions & 4 deletions charts/mlrun-ce/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,25 +123,83 @@ Create chart name and version as used by the chart label.
{{- end -}}

{{/*
Minio Service URL
=============================================================================
S3 Storage Backend Helpers
Supports both MinIO and SeaweedFS as S3-compatible storage backends
=============================================================================
*/}}

{{/*
Validate S3 storage backend configuration
Fails if seaweedfs is not enabled
*/}}
{{- define "mlrun-ce.s3.validateConfig" -}}
{{- if not .Values.seaweedfs.enabled -}}
{{- fail "seaweedfs.enabled must be set to true for S3 storage backend." -}}
{{- end -}}
{{- end -}}

{{/*
S3 Service URL - returns the endpoint URL for SeaweedFS
*/}}
{{- define "mlrun-ce.s3.service.url" -}}
{{- include "mlrun-ce.s3.validateConfig" . -}}
http://seaweedfs-s3.{{.Release.Namespace}}.svc.cluster.local:{{ .Values.seaweedfs.s3.port }}
{{- end -}}

{{/*
S3 Service Host - returns just the hostname for pipeline config
*/}}
{{- define "mlrun-ce.s3.service.host" -}}
{{- include "mlrun-ce.s3.validateConfig" . -}}
seaweedfs-s3.{{.Release.Namespace}}.svc.cluster.local
{{- end -}}

{{/*
S3 Service Port - returns the port for pipeline config
*/}}
{{- define "mlrun-ce.s3.service.port" -}}
{{- include "mlrun-ce.s3.validateConfig" . -}}
{{- .Values.seaweedfs.s3.port | toString -}}
{{- end -}}

{{/*
S3 Access Key
*/}}
{{- define "mlrun-ce.s3.accessKey" -}}
{{- include "mlrun-ce.s3.validateConfig" . -}}
{{- .Values.pipelines.s3.accessKey -}}
{{- end -}}

{{/*
S3 Secret Key
*/}}
{{- define "mlrun-ce.s3.secretKey" -}}
{{- include "mlrun-ce.s3.validateConfig" . -}}
{{- .Values.pipelines.s3.secretKey -}}
{{- end -}}

{{/*
Legacy Minio Service URL - kept for backward compatibility
*/}}
{{- define "mlrun-ce.minio.service.url" -}}
http://minio.{{.Release.Namespace}}.svc.cluster.local:{{ .Values.minio.service.port }}
{{ include "mlrun-ce.s3.service.url" . }}
{{- end -}}
{{- define "mlrun-ce.minio-pipeline.service.url" -}}
minio.{{.Release.Namespace}}.svc.cluster.local
{{ include "mlrun-ce.s3.service.host" . }}
{{- end -}}

{{/*
MLRun storage auto mount params
Global toggle is for fast toggling between on-prem/standalone and s3 cases
Can be overriden if params are explicitly specified
Uses SeaweedFS as the storage backend
*/}}
{{- define "mlrun.storage.auto.mount.params" -}}
{{- if hasKey .Values.mlrun "storageAutoMountParams" -}}
{{ .Values.mlrun.storageAutoMountParams }}
{{- else if not .Values.global.infrastructure.aws.s3NonAnonymous -}}
"aws_access_key={{ .Values.minio.rootUser }},aws_secret_key={{ .Values.minio.rootPassword }},endpoint_url={{ include "mlrun-ce.minio.service.url" . }}"
"aws_access_key={{ include "mlrun-ce.s3.accessKey" . }},aws_secret_key={{ include "mlrun-ce.s3.secretKey" . }},endpoint_url={{ include "mlrun-ce.s3.service.url" . }}"
{{- else -}}
"non_anonymous=True"
{{- end -}}
Expand Down
6 changes: 3 additions & 3 deletions charts/mlrun-ce/templates/config/jupyter-env-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ metadata:
name: jupyter-common-env
data:
{{- if not .Values.global.infrastructure.aws.s3NonAnonymous }}
AWS_ENDPOINT_URL_S3: {{ include "mlrun-ce.minio.service.url" . }}
AWS_SECRET_ACCESS_KEY: {{ .Values.minio.rootPassword }}
AWS_ACCESS_KEY_ID: {{ .Values.minio.rootUser }}
AWS_ENDPOINT_URL_S3: {{ include "mlrun-ce.s3.service.url" . }}
AWS_SECRET_ACCESS_KEY: {{ include "mlrun-ce.s3.secretKey" . }}
AWS_ACCESS_KEY_ID: {{ include "mlrun-ce.s3.accessKey" . }}
{{- end }}
MLRUN_STORAGE__AUTO_MOUNT_TYPE: {{ default "" .Values.mlrun.storageAutoMountType }}
S3_NON_ANONYMOUS: {{ .Values.global.infrastructure.aws.s3NonAnonymous | toString | title | quote | default "\"True\"" }}
Expand Down
6 changes: 3 additions & 3 deletions charts/mlrun-ce/templates/config/mlrun-env-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ data:
MLRUN_HTTPDB__PROJECTS__LEADER: mlrun
MLRUN_HTTPDB__PROJECTS__FOLLOWERS: nuclio
{{- if not .Values.global.infrastructure.aws.s3NonAnonymous }}
AWS_SECRET_ACCESS_KEY: {{ .Values.minio.rootPassword }}
AWS_ACCESS_KEY_ID: {{ .Values.minio.rootUser }}
AWS_ENDPOINT_URL_S3: {{ include "mlrun-ce.minio.service.url" . }}
AWS_SECRET_ACCESS_KEY: {{ include "mlrun-ce.s3.secretKey" . }}
AWS_ACCESS_KEY_ID: {{ include "mlrun-ce.s3.accessKey" . }}
AWS_ENDPOINT_URL_S3: {{ include "mlrun-ce.s3.service.url" . }}
{{- end }}
MLRUN_FUNCTION__SPEC__SERVICE_ACCOUNT__DEFAULT: {{ .Values.mlrun.api.functionSpecServiceAccountDefault | default "" | quote }}
MLRUN_HTTPDB__REAL_PATH: s3://
Expand Down
6 changes: 3 additions & 3 deletions charts/mlrun-ce/templates/pipelines/configmaps/pipeline-install-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
{{- if .Values.pipelines.enabled -}}
apiVersion: v1
data:
minioServiceHost: {{ include "mlrun-ce.minio-pipeline.service.url" . }}
minioServicePort: "{{ .Values.pipelines.minio.endpointPort }}"
objectStoreServiceHost: {{ include "mlrun-ce.s3.service.host" . }}
objectStoreServicePort: {{ include "mlrun-ce.s3.service.port" . | quote }}
ConMaxLifeTime: 120s
appName: pipeline
appVersion: {{ .Values.pipelines.images.apiServer.tag }}
autoUpdatePipelineDefaultVersion: "false"
bucketName: "{{ .Values.pipelines.minio.bucket }}"
bucketName: "{{ .Values.pipelines.s3.bucket }}"
cacheDb: cachedb
cacheImage: {{ .Values.pipelines.images.cacheImage.repository }}:{{ .Values.pipelines.images.cacheImage.tag }}
cacheNodeRestrictions: "false"
Expand Down
Loading
Loading