doc: Align k8s docs and add 2.x.x upgrade notes

doc/content/enterprise/kubernetes/azure/_index.md

@@ -13,9 +13,9 @@ This guide gives an overview of the architecture and shows how to deploy your ow
 <!--more-->
 
 {{< warning >}}
-Until the release of v1.0.0, all versions are considered to be in the Alpha stage and are not suitable for production use. The Things Industries does not offer any guarantees on compatibility between the Alpha versions.
+All versions before v1.0.0 are considered to be in the Alpha stage and are not suitable for production use. The Things Industries does not offer any guarantees on compatibility between the Alpha versions.
 {{</ warning >}}
 
-{{< warning >}}
-Until Azure releases a new version of their managed Redis service ([Redis 7.0 support in Azure](https://learn.microsoft.com/en-us/answers/questions/1191155/redis-7-0-support-in-azure)) this deployment uses Bitnami in-cluster Redis Helm charts. They are not recommended for production use.
-{{</ warning >}}
+{{< note >}}
+This deployment uses Bitnami in-cluster Redis Helm charts. They are not recommended for production use. Use the [Azure Managed Redis](https://azure.microsoft.com/en-us/products/managed-redis) instead with the NoCluster option.
+{{</ note >}}

doc/content/enterprise/kubernetes/azure/configuration/_index.md

@@ -82,3 +82,35 @@ global:
       adminUserID: <initial_tenant_admin_id>
       adminPassword: <initial_tenant_admin_password>
 ```
+
+{{< note "The remaining mandatory values (cluster ID, blob storage, database and Redis addresses) are supplied by the Terraform `2-kubernetes` layer. The fields shown below document the Azure-specific chart values that the Terraform layer sets, so you can understand or override them." />}}
+
+## Azure Workload Identity
+
+{{% tts %}} authenticates to Azure services (such as Blob Storage) using [Azure Workload Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview). The user-assigned managed identity created during the infrastructure deployment is wired into the {{% tts %}} components through two chart values:
+
+- `global.serviceAccount.annotations` attaches the managed identity's client ID to every component ServiceAccount.
+- `global.podLabels` opts the component pods into workload identity token injection.
+
+```yaml
+global:
+  serviceAccount:
+    annotations:
+      azure.workload.identity/client-id: <managed_identity_client_id>
+  podLabels:
+    azure.workload.identity/use: "true"
+```
+
+## Blob Storage
+
+{{% tts %}} stores blobs (for example device and profile pictures) in an Azure Storage Account. Set the blob provider to `azure` and provide the storage account name. Authentication is handled by [Azure Workload Identity](#azure-workload-identity), so no account key is required in the values file.
+
+```yaml
+global:
+  blob:
+    provider: azure
+    azure:
+      accountName: <storage_account_name>
+```
+
+{{< note "Azure Key Vault is not a supported key vault provider. The chart's `global.keyVault.provider` accepts only `static` or `aws`." />}}

doc/content/enterprise/kubernetes/azure/troubleshooting/_index.md

@@ -15,7 +15,7 @@ This guide contains general troubleshooting information.
 
 ## Error: Get "http://localhost/api/v1/namespaces/...": dial tcp [::1]:80: connect: connection refused
 
-Set `KUBECONFIG` and `KUBE_CONFIG_PATH` variables to the location of the `kubeconfig` file. The default location is `~/.local/kubeconfig`.
+Set `KUBECONFIG` and `KUBE_CONFIG_PATH` variables to the location of the `kubeconfig` file. The default location is `~/.kube/config`.
 
 ## pkg/util/store:driver (driver error)
 

doc/content/enterprise/kubernetes/azure/upgrading/_index.md

@@ -36,3 +36,26 @@ $ terraform apply -var='is-db-migrate=true' -var='noc-db-migrate=true'
 ## Kubernetes version upgrade
 
 AKS Kubernetes version upgrades have to be performed sequentially for each minor version. For example to upgrade from `1.25` to `1.27` it's necessary to upgrade to version `1.26` first as an intermediate step. For more information refer to the [official documentation](https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions).
+
+## Upgrading from Helm chart 1.x.x to 2.x.x
+
+{{< new-in-version "2.0.0" >}} Workload Identity is no longer wired up automatically from `global.blob.provider: azure`. 
+Users must now set the annotation and pod label explicitly in values:
+```yaml
+global:
+  serviceAccount:
+    annotations:
+      azure.workload.identity/client-id: "<client-id>"
+    podLabels:
+      azure.workload.identity/use: "true"
+```
+The pod label has also moved from Deployment selector labels to pod template labels. Existing Azure deployments must delete their Deployments before upgrading. Use `--cascade=orphan` to keep pods running during the migration:
+```bash
+RELEASE="<your-release-name>"
+NAMESPACE="<your-namespace>"
+for component in is as ns gs js dcs gcs console noc pba; do # Drop pba and/or noc from the list if not enabled.
+  kubectl delete deployment "${RELEASE}-${component}" -n "${NAMESPACE}" --cascade=orphan
+done
+helm upgrade "${RELEASE}" . -n "${NAMESPACE}" -f <your-values-file>
+```
+This is required because Kubernetes does not allow changes to `spec.selector.matchLabels` on existing Deployments. The `--cascade=orphan` flag ensures pods continue running while the Deployment objects are recreated by Helm.

doc/content/enterprise/kubernetes/generic/_index.md

@@ -14,8 +14,6 @@ This guide helps the user to install and configure {{% tts %}} on a Kubernetes c
 
 {{< warning "Operating The Things Stack on a Kubernetes cluster is only meant for advanced users with sufficient experience with the chosen infrastructure platform and with Kubernetes." />}}
 
-{{< warning >}} Until the release of v1.0.0, all versions are considered to be in the Alpha stage and are not suitable for production use. The Things Industries does not offer any guarantees on compatibility between the Alpha versions. {{</ warning >}}
-
 {{< note  "Make sure to follow each section of this guide in the same order as it is written without skipping any steps." />}}
 
 ## UDP Gateway Support
@@ -26,6 +24,6 @@ Please check the documentation of the Load Balancer and run tests to ensure that
 
 ## Kubernetes Version Support
 
-|                                | Kubernetes v1.21 and above |
-| ------------------------------ | -------------------------- |
-| Chart v1.0.0 and below (alpha) | [x]                        |
+{{% tts %}} Helm chart versions v1.x and below support **Kubernetes v1.21 and above**.
+
+Starting with {{% tts %}} Helm chart v2.0.0, **Kubernetes v1.27 or above** is required, because the chart's PodDisruptionBudgets use the `unhealthyPodEvictionPolicy` field, which is only available from Kubernetes v1.27 onwards.

doc/content/enterprise/kubernetes/generic/configuration/_index.md

@@ -101,7 +101,6 @@ global:
       secretAccessKey: # AWS access key secret
     azure: # Set only if provider is "azure".
       accountName: # account name
-      clientID: # client ID
     gcp: # Set only if provider is "gcp".
       # Base64 encoded GCP credentials.json file.
       # One option is to run `$ cat <credentials>.json | base64`.
@@ -160,6 +159,49 @@ dcs:
       bucket: # End Device Claiming Server bucket from "Section 4. Blob Storage"
 ```
 
+## Optional features
+
+Beyond the mandatory minimum above, the chart supports a number of optional features. Each is configured through additional `values.yaml` keys; see the full `values.yaml` for the complete list and defaults.
+
+- **High availability**: per-component PodDisruptionBudgets (`<component>.podDisruptionBudget.*`) and horizontal pod autoscaling.
+- **Redis high availability**: Redis Sentinel failover (`global.redis.failover.*`) and separate cache/events Redis endpoints (`global.cache.redis.*`, `global.events.redis.*`).
+- **OpenTelemetry tracing**: `global.tracing.*`.
+- **OIDC login**: `is.oidcProvider.*`.
+- **Email delivery**: `is.email.provider` with `smtp`, `sendgrid` or `dir` backends.
+
+## Security context, scheduling and service accounts
+
+The Helm chart exposes a set of `global` values that apply to every component (`as`, `console`, `dcs`, `gcs`, `gs`, `is`, `js`, `noc`, `ns`, `pba`). Each sub-chart exposes the same keys so that the global defaults can be overridden or augmented per component.
+
+**Field name**                       | **Default**                       | **Description**
+--------------------------------------|-----------------------------------|----------------------------------------------------------------
+`global.podSecurityContext`           | `runAsUser`/`runAsGroup: 886`, `runAsNonRoot: true` | Pod-level security context shared by every component.
+`global.containerSecurityContext`     | `allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]` | Container-level security context shared by every component.
+`global.nodeSelector`                 | `{}`                              | Node labels used to schedule all component pods.
+`global.podLabels`                    | `{}`                              | Labels applied to the pod template of all components.
+`global.serviceAccount.create`        | `true`                            | Whether to create ServiceAccounts for the components. Disable to bring your own pre-provisioned ServiceAccounts.
+`global.serviceAccount.automountServiceAccountToken` | `false`            | Whether pods should mount the ServiceAccount token. {{% tts %}} components do not call the Kubernetes API, so this is `false` for hardening.
+`global.serviceAccount.annotations`   | `{}`                              | Annotations applied to all component ServiceAccounts. Use this to attach cloud-provider workload-identity annotations.
+
+The override semantics differ per value type:
+
+- **Scalar keys** (for example `serviceAccount.create`, `serviceAccount.automountServiceAccountToken`): if the component value is set, it replaces the global value. Otherwise the global applies.
+- **Map keys** (for example `podSecurityContext`, `containerSecurityContext`, `nodeSelector`, `serviceAccount.annotations`): the component map is merged on top of the global map. Keys present in both are won by the component.
+
+For example, to run all pods on the `lorawan` node pool except the NOC Grafana pod which must land on the `monitoring` pool:
+
+```yaml
+global:
+  nodeSelector:
+    agentpool: lorawan
+noc:
+  grafana:
+    nodeSelector:
+      agentpool: monitoring
+```
+
+{{< note "`global.serviceAccount.name` has no global equivalent because each component needs a unique ServiceAccount name. Set `<component>.serviceAccount.name` per component to bring your own ServiceAccount." />}}
+
 ## {{% ttigpro %}} configuration
 
 The Helm chart does not support {{% ttigpro %}} by default. To enable it in the Helm chart, The Things Gateway Controller must be enabled and the ingress controller of the Kubernetes cluster must have mTLS configured.
@@ -229,17 +271,21 @@ For more info check the [Traefik docs on the PassTLSClientCert middleware](https
 
 For more info check the [Traefik docs on TLS options](https://doc.traefik.io/traefik/https/tls/#tls-options).
 
-3. Set the protocol annotations for {{% ttigpro %}}, middleware annotations and serviceAnnotations in `values.yaml` 
+3. Set the protocol annotations for {{% ttigpro %}}, middleware annotations and service annotations in `values.yaml`
 (in addition to the existing annotations):
 
 ```yaml
-annotations:
-  ttigw:
-    traefik.ingress.kubernetes.io/router.entrypoints: ttigw,ttigwsecure
-    traefik.ingress.kubernetes.io/router.middlewares: traefik-passtlsclientcert@kubernetescrd
-    traefik.ingress.kubernetes.io/router.tls: "true"
-serviceAnnotations:
-  traefik.ingress.kubernetes.io/service.serversscheme: h2c
+global:
+  ingress:
+    annotations:
+      ttigw:
+        traefik.ingress.kubernetes.io/router.entrypoints: ttigw,ttigwsecure
+        traefik.ingress.kubernetes.io/router.middlewares: traefik-passtlsclientcert@kubernetescrd
+        traefik.ingress.kubernetes.io/router.tls: "true"
+  services:
+    annotations:
+      grpc:
+        traefik.ingress.kubernetes.io/service.serversscheme: h2c
 ```
 
 4. Install (or upgrade) the helm chart.

doc/content/enterprise/kubernetes/generic/database-migrations/_index.md

@@ -82,4 +82,4 @@ kubectl apply -f <name>-<component>-migrate.yaml
 kubectl delete -f <name>-<component>-migrate.yaml
 ```
 
-Repeat the above process for all the components required. Currently only the IS and NOC components require migrations.
+Repeat the above process for all the components required. Currently the IS, NOC (when the NOC is enabled) and AS (when the storage integration is enabled) components require migrations.

doc/content/enterprise/kubernetes/generic/monitoring/_index.md

@@ -9,12 +9,16 @@ aliases:
   ]
 ---
 
-{{% tts %}} exposes Prometheus metrics at the `/metrics` endpoint. This route is protected by Basic Auth with username `metrics` and password `global.http.metrics.password`.
+{{% tts %}} exposes Prometheus metrics at the `/metrics` endpoint. This route is protected by Basic Auth with username `metrics` and password `global.http.metrics.password`. Exposure of the endpoint is controlled by `global.http.metrics.enable` (default `true`).
 
 <!--more-->
 
 To scrape these metrics, you can install Prometheus in the cluster. The installation details are outside the scope of this document.
 
+The chart does not ship a `ServiceMonitor` or Prometheus scrape annotations. If you use the Prometheus Operator, add your own `ServiceMonitor`, or attach scrape annotations to the component services through `global.services.annotations` (and `global.services.annotations.grafana` for the NOC Grafana service).
+
+The metrics endpoint listens on `global.http.port` (default `1885`). The examples below use `1885`; adjust the port if you have changed `global.http.port`.
+
 The following scrape config can be used to scrape metrics from each of the components of The Things Stack.
 
 ```yaml
@@ -46,3 +50,61 @@ scrape_configs:
     static_configs:
       - targets: ["mytts-is.tts.svc.cluster.local:1885"]
 ```
+
+## Network Operations Center (NOC)
+
+The Network Operations Center (NOC) provides network insights through a bundled Grafana instance. It is disabled by default. Enabling it requires a dedicated TimescaleDB database and event storage in Redis.
+
+To enable the NOC:
+
+1. Enable the NOC and turn on event storage in Redis (the NOC consumes events from Redis):
+
+    ```yaml
+    global:
+      noc:
+        enabled: true
+      events:
+        redis:
+          storageEnabled: true
+    ```
+
+2. Configure the NOC TimescaleDB database. `migrate` runs the database migration during the Helm upgrade (see [Database Migrations]({{< ref "enterprise/kubernetes/generic/database-migrations" >}})).
+
+    ```yaml
+    noc:
+      store:
+        database:
+          uri: # postgres://<username>:<password>@<host>:<port>/<database>?<options>
+          readURI: # (Optional) Read replica connection string.
+          migrate: true
+    ```
+
+3. Configure the bundled Grafana instance. The Grafana store points at the same TimescaleDB server as the NOC.
+
+    ```yaml
+    noc:
+      grafana:
+        adminPassword: # See preparation section.
+        store:
+          host: # Hostname of the NOC TimescaleDB database (without the port).
+          username:
+          password:
+          name:
+    ```
+
+    The OAuth client secret (`global.noc.oauth.clientSecret`) is also required and is covered in the [preparation section]({{< ref "enterprise/kubernetes/generic/preparation" >}}).
+
+{{< note "The NOC Grafana image is pinned via `noc.grafana.image.tag` and does not follow the {{% tts %}} release version cycle. Override it only when instructed." />}}
+
+### Tuning the NOC data store
+
+The following optional values tune the NOC data store. The defaults are suitable for most deployments.
+
+**Field name**                          | **Default**   | **Description**
+-----------------------------------------|---------------|----------------------------------------------------------------
+`noc.store.maxIdleConnections`           | `2`           | Maximum number of idle database connections.
+`noc.store.maxOpenConnections`           | `10`          | Maximum number of open database connections.
+`noc.store.rawDataRetention`             | `72h0m0s`     | Retention duration for raw data.
+`noc.store.targetInsertBatchSize`        | `8192`        | Target batch size for `INSERT` commands.
+`noc.store.targetInsertBatchWindow`      | `1m`          | Target batch window for `INSERT` commands.
+`noc.store.pagination.defaultLimit`      | `1000`        | Default limit for NOC pagination.

doc/content/enterprise/kubernetes/generic/preparation/_index.md

@@ -26,7 +26,7 @@ Copy each generated secret to a local file and then transfer those values later
 | PProf Password                        | Password to access profiling (pprof)                                  | 16   | `global.http.pprof.password`        |
 | Console Oauth Secret                  | Secret used by the Console to connect to The Things Stack             | 16   | `global.console.oauth.clientSecret` |
 | Tenant Administration Key             | Secret used to administer cluster tenants                             | 32   | `global.tenancy.adminKey`           |
-| NOC Grafana Admin Password (optional) | Grafana admin password for The Things Stack Network Operations Center | 24   | `global.noc.grafana.adminPassword`  |
+| NOC Grafana Admin Password (optional) | Grafana admin password for The Things Stack Network Operations Center | 24   | `noc.grafana.adminPassword`         |
 | NOC Oauth Client Secret (optional)    | OAuth client secret for The Things Stack Network Operations Center    | 24   | `global.noc.oauth.clientSecret`     |
 
 The simplest option is to use `openssl`.

doc/content/enterprise/kubernetes/generic/prerequisites/_index.md

@@ -53,7 +53,7 @@ The following is a guide of the general principles involved in setting up the in
 
 #### 1. Kubernetes Cluster
 
-{{% tts %}} requires a minimum kubernetes version of v1.21. We recommend using the highest available version.
+{{% tts %}} requires a minimum kubernetes version of v1.21. We recommend using the highest available version. From {{% tts %}} Helm chart v2.0.0, a minimum kubernetes version of v1.27 is required.
 
 #### 2. Postgres Compatible Database
 
@@ -127,7 +127,7 @@ Although we do support UDP Packet Forwarder as a gateway connection option, it r
 
 To configure the ingress controller for {{% tts %}}:
 1. Specify the ingress controller by setting the `global.ingress.controller` to the class name of the ingress controller deployed in the cluster. This will be used to set the ingress class name in the ingress routes that handle {{% tts %}} traffic.
-2. Specify the TLS secret by setting the `global.ingress.controller.tls.secretName`. The secret has to be accessible from the namespace where the {{% tts %}} Helm Chart is deployed. This will be used to terminate TLS for {{% tts %}} traffic
+2. Specify the TLS secret by setting the `global.ingress.tls.secretName`. The secret has to be accessible from the namespace where the {{% tts %}} Helm Chart is deployed. This will be used to terminate TLS for {{% tts %}} traffic
 3. Add annotations for the ingress routes if needed by setting `global.ingress.annotations.http`, `global.ingress.annotations.grpc`, `global.ingress.annotations.semtechws` or `global.ingress.annotations.ttigw`.
 4. Add service annotations for {{% tts %}} services by setting `global.services.annotations` if needed.
 5. Expose the ports used by {{% tts %}} in your ingress controller. A list of all the ports can be found [here]({{< ref "/concepts/networking/#port-allocations" >}}). For production environments, make sure to expose only TLS ports.

doc/content/enterprise/kubernetes/generic/troubleshooting/_index.md

@@ -29,7 +29,7 @@ For this error, check that the value set in `global.tls.rootCA` is base64 encode
 
 ## x509_certificate_invalid
 
-For this error, make sure that the value set in `ingress.traefik.tls.secretName` is a valid TLS certificate.
+For this error, make sure that the value set in `global.ingress.tls.secretName` is a valid TLS certificate.
 
 ## pkg/util/store:driver (driver error)