Helm ignores some of custom values

[Andryyyha]

Checks

• I have checked for existing issues.
• This report is about the User-Community Airflow Helm Chart.

Chart Version

8.6.0

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:38:26Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.9-eks-14c7a48", GitCommit:"717bfb2b8ceb809a42a6c0baabde59fae28637ef", GitTreeState:"clean", BuildDate:"2022-04-01T03:17:28Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Helm Version

version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.18.1"}

Description

I am trying to deploy airflow with KubernetesExecutor on EKS cluster. I have provisioned 2 persistent volumes based on AWS EFS and 2 claims based on that volumes. Also I have enabled persistence in dags: and logs: sections and specified claim names. But after deployment of chart volumes didn't attach to any of pods. Output of helm get values shown that persistence disabled for both dags and logs.

Relevant Logs

Output of `helm get values` command after deploy
USER-SUPPLIED VALUES:
airflow:
  config:
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
  connections: []
  executor: KubernetesExecutor
  extraEnv: []
  extraPipPackages: []
  extraVolumeMounts: []
  extraVolumes: []
  fernetKey: 7T512UXSSmBOkpWimFHIVb8jK6lfmSAvx4mO6Arehnc=
  image:
    repository: apache/airflow
    tag: 2.2.5-python3.8
  kubernetesPodTemplate:
    extraPipPackages: []
    extraVolumeMounts: []
    extraVolumes: []
    resources: {}
    stringOverride: ""
  legacyCommands: false
  pools: []
  users:
  - email: admin@example.com
    firstName: admin
    lastName: admin
    password: admin
    role: Admin
    username: admin
  variables: []
  webserverSecretKey: THIS IS UNSAFE!
dags:
  gitSync:
    enabled: false
  path: /opt/airflow/dags
  persistence:
    enabled: false
externalDatabase:
  database: airflow
  host: airflow-metadata.cgxqlq4ayveu.us-east-2.rds.amazonaws.com
  password:
  port: 5432
  properties: ?sslmode=disable
  type: postgres
  user: airflow
extraManifests: []
flower:
  enabled: false
ingress:
  enabled: false
logs:
  path: /opt/airflow/logs
  persistence:
    enabled: false
pgbouncer:
  authType: md5
  enabled: true
  resources: {}
postgresql:
  enabled: false
redis:
  enabled: false
scheduler:
  livenessProbe:
    enabled: true
    taskCreationCheck:
      enabled: false
      thresholdSeconds: 300
  logCleanup:
    enabled: true
    retentionMinutes: 21600
  replicas: 1
  resources: {}
serviceAccount:
  annotations: {}
  create: true
  name: airflow-master
triggerer:
  capacity: 1000
  enabled: true
  replicas: 1
  resources: {}
web:
  replicas: 1
  resources: {}
  service:
    externalPort: 8080
    type: ClusterIP
  webserverConfig:
    existingSecret: ""
    stringOverride: |
      from airflow import configuration as conf
      from flask_appbuilder.security.manager import AUTH_DB

      # the SQLAlchemy connection string
      SQLALCHEMY_DATABASE_URI = conf.get("core", "SQL_ALCHEMY_CONN")

      # use embedded DB for auth
      AUTH_TYPE = AUTH_DB
workers:
  enabled: false

Custom Helm Values

########################################
## CONFIG | Airflow Configs
########################################
airflow:
  ## if we use legacy 1.10 airflow commands
  legacyCommands: false

  ## configs for the airflow container image
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/airflow-version.md
  image:
    repository: apache/airflow
    tag: 2.2.5-python3.8

  ## the airflow executor type to use
  executor: KubernetesExecutor

  ## the fernet encryption key (sets `AIRFLOW__CORE__FERNET_KEY`)
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/set-fernet-key.md
  ## [WARNING] change from default value to ensure security
  fernetKey: "7T512UXSSmBOkpWimFHIVb8jK6lfmSAvx4mO6Arehnc="

  ## the secret_key for flask (sets `AIRFLOW__WEBSERVER__SECRET_KEY`)
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/set-webserver-secret-key.md
  ## [WARNING] change from default value to ensure security
  webserverSecretKey: "THIS IS UNSAFE!"

  ## environment variables for airflow configs
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/airflow-configs.md
  config:
    AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
    AIRFLOW__CORE__LOAD_EXAMPLES: "False"

  ## a list of users to create
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/security/airflow-users.md
  users:
    - username: admin
      password: admin
      role: Admin
      email: admin@example.com
      firstName: admin
      lastName: admin

  ## a list airflow connections to create
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/airflow-connections.md
  connections: []

  ## a list airflow variables to create
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/airflow-variables.md
  variables: []

  ## a list airflow pools to create
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/dags/airflow-pools.md
  pools: []

  ## extra pip packages to install in airflow Pods
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/extra-python-packages.md
  ## [WARNING] this feature is not recommended for production use, see docs
  extraPipPackages: []

  ## extra environment variables for the airflow Pods
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-environment-variables.md
  extraEnv: []

  ## extra VolumeMounts for the airflow Pods
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
#  extraVolumeMounts:
#    - name: airflow-logs
#      mountPath: /opt/airflow/logs
#    - name: airflow-dags
#      mountPath: /opt/airflow/dags

  ## extra Volumes for the airflow Pods
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
#  extraVolumes:
#    - name: airflow-logs
#      persistentVolumeClaim:
#        name: logs-pvc
#    - name: airflow-dags
#      persistentVolumeClaim:
#        name: dags-pvc

  ## configs generating the `pod_template.yaml` file for `AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE`
  ## [NOTE] the `dags.gitSync` values will create a git-sync init-container in the pod
  ## [NOTE] the `airflow.extraPipPackages` will NOT be installed
  kubernetesPodTemplate:

    ## the full content of the pod-template file (as a string)
    ## [NOTE] all other `kubernetesPodTemplate.*` are disabled when this is set
    stringOverride: ""

    ## resource requests/limits for the Pod template "base" container
    ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
    resources: {}

    ## extra pip packages to install in the Pod template
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/extra-python-packages.md
    ## [WARNING] this feature is not recommended for production use, see docs
    extraPipPackages: []

    ## extra VolumeMounts for the Pod template
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
    extraVolumeMounts: []

    ## extra Volumes for the Pod template
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
    ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
    extraVolumes: []

###################################
## COMPONENT | Airflow Scheduler
###################################
scheduler:
  ## the number of scheduler Pods to run
  replicas: 1

  ## resource requests/limits for the scheduler Pods
  ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
  resources: {}

  logCleanup:
    enabled: false

  ## configs for the scheduler Pods' liveness probe
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md
  livenessProbe:
    enabled: false

  ## configs for an additional check that ensures tasks are being created by the scheduler
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/monitoring/scheduler-liveness-probe.md
  taskCreationCheck:
    enabled: false
    thresholdSeconds: 300

###################################
## COMPONENT | Airflow Webserver
###################################
web:
  ## the number of web Pods to run
  replicas: 1

  ## resource requests/limits for the web Pods
  ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
  resources: {}

  ## configs for the Service of the web Pods
  service:
    type: ClusterIP
    externalPort: 8080

  ## configs generating the `webserver_config.py` file
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/configuration/airflow-configs.md#webserver_configpy
  webserverConfig:
    ## the full content of the `webserver_config.py` file (as a string)
    stringOverride: |
      from airflow import configuration as conf
      from flask_appbuilder.security.manager import AUTH_DB
      
      # the SQLAlchemy connection string
      
      # use embedded DB for auth
      AUTH_TYPE = AUTH_DB

    ## the name of a Secret containing a `webserver_config.py` key
    existingSecret: ""

  livenessProbe:
    enabled: false

#  extraVolumeMounts:
#    - name: airflow-logs
#      mountPath: /opt/airflow/logs
#    - name: airflow-dags
#      mountPath: /opt/airflow/dags

  ## extra Volumes for the airflow Pods
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-persistent-volumes.md
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/mount-files.md
#  extraVolumes:
#    - name: airflow-logs
#      persistentVolumeClaim:
#        name: logs-pvc
#    - name: airflow-dags
#      persistentVolumeClaim:
#        name: dags-pvc

###################################
## COMPONENT | Airflow Workers
###################################
workers:
  ## if the airflow workers StatefulSet should be deployed
  enabled: false

###################################
## COMPONENT | Triggerer
###################################
triggerer:
  ## if the airflow triggerer should be deployed
  enabled: false

  ## the number of triggerer Pods to run
  replicas: 1

  ## resource requests/limits for the triggerer Pods
  ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
  resources: {}

  ## maximum number of triggers each triggerer will run at once (sets `AIRFLOW__TRIGGERER__DEFAULT_CAPACITY`)
  capacity: 1000

###################################
## COMPONENT | Flower
###################################
flower:
  ## if the airflow flower UI should be deployed
  enabled: false

###################################
## CONFIG | Airflow Logs
###################################
logs:
  ## NOTE: this is the default value
  path: /opt/airflow/logs

  persistence:
    enabled: true

    ## the name of your existing PersistentVolumeClaim
    existingClaim: logs-pvc

    ## WARNING: as multiple pods will write logs, this MUST be ReadWriteMany
    accessMode: ReadWriteMany

###################################
## CONFIG | Airflow DAGs
###################################
dags:
  ## NOTE: this is the default value
  path: /opt/airflow/dags

  persistence:
    enabled: true

    ## NOTE: this is name of your existing volume
    existingClaim: dags-pvc

    ## NOTE: as multiple Pods read the DAGs concurrently this MUST be ReadOnlyMany or ReadWriteMany
    accessMode: ReadOnlyMany

###################################
## CONFIG | Kubernetes Ingress
###################################
ingress:
  ## if we should deploy Ingress resources
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/ingress.md
  enabled: false

###################################
## CONFIG | Kubernetes ServiceAccount
###################################
serviceAccount:
  ## if a Kubernetes ServiceAccount is created
  create: true

  ## the name of the ServiceAccount
  name: "airflow-master"

  ## annotations for the ServiceAccount
  annotations: {}

###################################
## CONFIG | Kubernetes Extra Manifests
###################################

## a list of extra Kubernetes manifests that will be deployed alongside the chart
## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/kubernetes/extra-manifests.md
extraManifests: []

###################################
## DATABASE | PgBouncer
###################################
pgbouncer:
  ## if the pgbouncer Deployment is created
  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/pgbouncer.md
  enabled: true

  ## resource requests/limits for the pgbouncer Pods
  ## [SPEC] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.20/#resourcerequirements-v1-core
  resources: {}

  ## sets pgbouncer config: `auth_type`
  authType: md5

###################################
## DATABASE | Embedded Postgres
###################################
#postgresql:
#  ## if the `stable/postgresql` chart is used
#  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/embedded-database.md
#  ## [WARNING] the embedded Postgres is NOT SUITABLE for production deployments of Airflow
#  ## [WARNING] consider using an external database with `externalDatabase.*`
#  enabled: true
#
#  ## configs for the PVC of postgresql
#  persistence:
#    enabled: true
#    storageClass: ""
#    size: 8Gi

###################################
## DATABASE | External Database
###################################
externalDatabase:
  ## the type of external database
  ## - allowed values: "mysql", "postgres"
  ##
  type: "postgres"

  ## the host of the external database
  ##
  host: "airflow-metadata.cgxqlq4ayveu.us-east-2.rds.amazonaws.com"

  ## the port of the external database
  ##
  port: 5432

  ## the database/scheme to use within the external database
  ##
  database: "airflow"

  ## the username for the external database
  user: "airflow"

  ## the name of a pre-created secret containing the external database user
  ## - if set, this overrides `externalDatabase.user`
  ##
  #  userSecret: "postgres-creds"

  ## the key within `externalDatabase.userSecret` containing the user string
  ##
  #  userSecretKey: "username"

  ## the password for the external database
  ## - [WARNING] to avoid storing the password in plain-text within your values,
  ##   create a Kubernetes secret and use `externalDatabase.passwordSecret`
  ##
#  password: 

  ## the name of a pre-created secret containing the external database password
  ## - if set, this overrides `externalDatabase.password`
  ##
    passwordSecret: "postgres-creds"

  ## the key within `externalDatabase.passwordSecret` containing the password string
  ##
    passwordSecretKey: "password"

  ## extra connection-string properties for the external database
  ##
  ## ____ EXAMPLE _______________
  ##   # require SSL (only for Postgres)
  ##   properties: "?sslmode=require"
  ##
  properties: "?sslmode=disable"

postgresql:
  enabled: false

###################################
## DATABASE | Embedded Redis
###################################
redis:
  ## if the `stable/redis` chart is used
  enabled: false

###################################
## DATABASE | External Redis
###################################
#externalRedis:
#  ## the host of the external redis
#  ## [FAQ] https://github.com/airflow-helm/charts/blob/main/charts/airflow/docs/faq/database/external-redis.md
#  host: localhost

[thesuperzapper]

@Andryyyha are you 100% sure that you are applying the custom-values.yaml file you are intending to?

Because you have said yourself that the helm get values ... is saying that logs.persistence.enabled is false when you have set it to true in the provided custom values.

Can you tell me the commands you are running to:

1. Install/Update the chart (for example, helm upgrade --namespace ....)
2. Get your values (for example, helm get values --namespace ...)

PS: I wonder if you are accidentally installing the chart into two different namespaces (forgetting to set --namespace)

[Andryyyha]

I am using commands from Quickstart Guide

export AIRFLOW_NAME="airflow-cluster"
export AIRFLOW_NAMESPACE="airflow-cluster"
kubectl create ns "$AIRFLOW_NAMESPACE"

helm install   "$AIRFLOW_NAME"   airflow-stable/airflow   --namespace "$AIRFLOW_NAMESPACE"   --version "8.X.X"   --values custom-values-test.yaml

After successful deployment, I run helm get values airflow-cluster -n airflow-cluster
Funny thing that some values from my file are applied(e.g. values from externalDatabase section). I tried to add variable through Airflow UI and it's appeared in backend DB

[thesuperzapper]

@Andryyyha did you figure out what the problem is?

I am still unclear what would cause helm to not apply the requested --values file.