Skip to content

[FLINK-39021][Deployment/Kubernetes] Support native PersistentVolumeClaim (PVC) mounting for Kubernetes deployments #27523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Myracle wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[FLINK-39021][Deployment/Kubernetes] Support native PersistentVolumeClaim (PVC) mounting for Kubernetes deployments #27523

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 110 additions & 0 deletions docs/content.zh/docs/deployment/resource-providers/native_kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,116 @@ $ ./bin/kubernetes-session.sh -Dkubernetes.env.secretKeyRef=\
The env variable `SECRET_USERNAME` contains the username and the env variable `SECRET_PASSWORD` contains the password of the secret `mysecret`.
For more details see the [official Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables).

### 挂载持久卷声明（PVC）

[Kubernetes PersistentVolumeClaims (PVCs)](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) 提供了一种在 Kubernetes 中请求和使用持久存储的方式。
Flink on Kubernetes 支持通过配置选项直接将 PVC 挂载到 JobManager 和 TaskManager Pod 中。

#### 配置选项

Flink 提供了两个用于挂载 PVC 的配置选项：

<table class="table table-bordered">
<thead>
<tr>
<th class="text-left" style="width: 30%">配置选项</th>
<th class="text-left" style="width: 15%">类型</th>
<th class="text-left" style="width: 15%">默认值</th>
<th class="text-left" style="width: 40%">描述</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>kubernetes.persistent-volume-claims</code></td>
<td>Map&lt;String, String&gt;</td>
<td>(无)</td>
<td>用户指定的 PersistentVolumeClaims，将被挂载到 Flink 容器中。
值的格式为 <code>pvc-name:/mount/path</code>。
多个 PVC 可以用逗号分隔。</td>
</tr>
<tr>
<td><code>kubernetes.persistent-volume-claims.read-only</code></td>
<td>Boolean</td>
<td>false</td>
<td>是否以只读模式挂载 PersistentVolumeClaims。
设置为 true 时，所有 PVC 将以只读方式挂载。</td>
</tr>
</tbody>
</table>

#### 使用示例

以下命令将 PVC `checkpoint-pvc` 挂载到启动 Pod 的 `/opt/flink/checkpoints` 路径：

```bash
$ ./bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=my-session-cluster \
-Dkubernetes.persistent-volume-claims=checkpoint-pvc:/opt/flink/checkpoints
```

您可以通过逗号分隔来挂载多个 PVC：

```bash
$ ./bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=my-session-cluster \
-Dkubernetes.persistent-volume-claims=checkpoint-pvc:/opt/flink/checkpoints,data-pvc:/opt/flink/data
```

以只读模式挂载 PVC（适用于共享参考数据）：

```bash
$ ./bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=my-session-cluster \
-Dkubernetes.persistent-volume-claims=shared-data:/opt/flink/shared \
-Dkubernetes.persistent-volume-claims.read-only=true
```

#### 示例：使用 PVC 进行 Checkpoint 存储

一个常见的用例是使用 PVC 存储 checkpoint。首先，在 Kubernetes 集群中创建一个 PVC：

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: flink-checkpoints-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: standard
```

然后启动一个挂载 PVC 用于 checkpoint 存储的 Flink 集群：

```bash
$ ./bin/flink run \
--target kubernetes-application \
-Dkubernetes.cluster-id=my-checkpoint-cluster \
-Dkubernetes.container.image=flink:latest \
-Dkubernetes.persistent-volume-claims=flink-checkpoints-pvc:/opt/flink/checkpoints \
-Dstate.checkpoints.dir=file:///opt/flink/checkpoints \
local:///opt/flink/usrlib/my-flink-job.jar
```

#### 前提条件

- PVC 必须在 Flink 集群部署之前存在于同一命名空间中。
- PVC 必须具有适当的访问模式：
- **ReadWriteOnce (RWO)**：单个 Pod 访问。适用于独立的 JobManager 部署。
- **ReadWriteMany (RWX)**：多个 Pod 访问同一存储。推荐用于高可用（HA）配置或当 JobManager 和 TaskManager 都需要写入同一存储时。
- **ReadOnlyMany (ROX)**：多个 Pod 只读访问。适用于共享参考数据。

{{< hint warning >}}
对于具有多个 JobManager 的高可用（HA）配置，或当 JobManager 和 TaskManager 都需要对同一存储进行写入访问时，请确保您的 PVC 支持 ReadWriteMany (RWX) 访问模式。在此类场景下使用 ReadWriteOnce (RWO) 可能会导致挂载失败或 I/O 错误。
{{< /hint >}}

{{< hint info >}}
`kubernetes.persistent-volume-claims.read-only` 选项会全局应用于所有配置的 PVC。如果您需要为不同的 PVC 设置不同的访问模式（例如：checkpoint 使用读写模式，而参考数据使用只读模式），请使用 [Pod 模板]({{< ref "docs/deployment/resource-providers/native_kubernetes" >}}#pod-template) 来定义细粒度的卷配置。
{{< /hint >}}

### High-Availability on Kubernetes

For high availability on Kubernetes, you can use the [existing high availability services]({{< ref "docs/deployment/ha/overview" >}}).
Expand Down
110 changes: 110 additions & 0 deletions docs/content/docs/deployment/resource-providers/native_kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,116 @@ $ ./bin/kubernetes-session.sh -Dkubernetes.env.secretKeyRef=\
The env variable `SECRET_USERNAME` contains the username and the env variable `SECRET_PASSWORD` contains the password of the secret `mysecret`.
For more details see the [official Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables).

### Mounting Persistent Volume Claims

[Kubernetes PersistentVolumeClaims (PVCs)](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) provide a way to request and use persistent storage in Kubernetes.
Flink on Kubernetes supports mounting PVCs directly to JobManager and TaskManager pods via configuration options.

#### Configuration Options

Flink provides two configuration options for mounting PVCs:

<table class="table table-bordered">
<thead>
<tr>
<th class="text-left" style="width: 30%">Configuration Option</th>
<th class="text-left" style="width: 15%">Type</th>
<th class="text-left" style="width: 15%">Default</th>
<th class="text-left" style="width: 40%">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>kubernetes.persistent-volume-claims</code></td>
<td>Map&lt;String, String&gt;</td>
<td>(none)</td>
<td>The user-specified PersistentVolumeClaims that will be mounted into Flink containers.
The value should be in the form of <code>pvc-name:/mount/path</code>.
Multiple PVCs can be specified by separating them with commas.</td>
</tr>
<tr>
<td><code>kubernetes.persistent-volume-claims.read-only</code></td>
<td>Boolean</td>
<td>false</td>
<td>Whether to mount PersistentVolumeClaims as read-only.
When set to true, all PVCs will be mounted as read-only.</td>
</tr>
</tbody>
</table>

#### Usage Examples

The following command will mount the PVC `checkpoint-pvc` to the path `/opt/flink/checkpoints` in the started pods:

```bash
$ ./bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=my-session-cluster \
-Dkubernetes.persistent-volume-claims=checkpoint-pvc:/opt/flink/checkpoints
```

You can mount multiple PVCs by separating them with commas:

```bash
$ ./bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=my-session-cluster \
-Dkubernetes.persistent-volume-claims=checkpoint-pvc:/opt/flink/checkpoints,data-pvc:/opt/flink/data
```

To mount PVCs as read-only (useful for shared reference data):

```bash
$ ./bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=my-session-cluster \
-Dkubernetes.persistent-volume-claims=shared-data:/opt/flink/shared \
-Dkubernetes.persistent-volume-claims.read-only=true
```

#### Example: Using PVC for Checkpoint Storage

A common use case is to use PVC for storing checkpoints. First, create a PVC in your Kubernetes cluster:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: flink-checkpoints-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: standard
```

Then start a Flink cluster with the PVC mounted for checkpoint storage:

```bash
$ ./bin/flink run \
--target kubernetes-application \
-Dkubernetes.cluster-id=my-checkpoint-cluster \
-Dkubernetes.container.image=flink:latest \
-Dkubernetes.persistent-volume-claims=flink-checkpoints-pvc:/opt/flink/checkpoints \
-Dstate.checkpoints.dir=file:///opt/flink/checkpoints \
local:///opt/flink/usrlib/my-flink-job.jar
```

#### Prerequisites

- The PVCs must exist in the same namespace as the Flink cluster before deployment.
- The PVCs must have appropriate access modes:
- **ReadWriteOnce (RWO)**: For single pod access. Suitable for standalone JobManager deployments.
- **ReadWriteMany (RWX)**: For multiple pods accessing the same storage. Recommended for HA setups or when both JobManager and TaskManagers need to write to the same storage.
- **ReadOnlyMany (ROX)**: For read-only access from multiple pods. Suitable for shared reference data.

{{< hint warning >}}
For high availability (HA) setups with multiple JobManagers or when both JobManager and TaskManagers need write access to the same storage, ensure your PVC supports ReadWriteMany (RWX) access mode. Using ReadWriteOnce (RWO) in such scenarios may cause mount failures or I/O errors.
{{< /hint >}}

{{< hint info >}}
The `kubernetes.persistent-volume-claims.read-only` option applies globally to all configured PVCs. If you need different access modes for different PVCs (e.g., read-write for checkpoints but read-only for reference data), use [Pod Templates]({{< ref "docs/deployment/resource-providers/native_kubernetes" >}}#pod-template) instead to define fine-grained volume configurations.
{{< /hint >}}

### High-Availability on Kubernetes

For high availability on Kubernetes, you can use the [existing high availability services]({{< ref "docs/deployment/ha/overview" >}}).
Expand Down
78 changes: 78 additions & 0 deletions ...etes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,84 @@ public class KubernetesConfigOptions {
"The node label whose value is the same as the node name. "
+ "Currently, this will only be used to set the node affinity of TM pods to avoid being scheduled on blocked nodes.");

/**
* The user-specified PersistentVolumeClaims (PVCs) that will be mounted into Flink containers.
*
* <p>The value should be in the form of {@code pvc-name:/mount/path} separated by commas.
* Multiple PVCs can be specified by separating them with commas.
*
* <p>Example: {@code checkpoint-pvc:/opt/flink/checkpoints,data-pvc:/opt/flink/data}
*
* <p>Prerequisites:
*
* <ul>
* <li>The PVCs must exist in the same namespace as the Flink cluster before deployment
* <li>The PVCs must have appropriate access modes:
* <ul>
* <li>ReadWriteOnce (RWO): For single pod access
* <li>ReadWriteMany (RWX): For multiple pods (recommended for HA setups)
* <li>ReadOnlyMany (ROX): For read-only access from multiple pods
* </ul>
* </ul>
*
* <p>Common use cases:
*
* <ul>
* <li>Checkpoint storage: Mount a shared PVC for storing checkpoints
* <li>Savepoint storage: Mount a PVC for savepoint data
* <li>Shared data: Mount read-only PVCs containing reference data
* <li>Job artifacts: Mount PVCs containing job JARs or dependencies
* </ul>
*/
public static final ConfigOption<Map<String, String>> KUBERNETES_PERSISTENT_VOLUME_CLAIMS =
key("kubernetes.persistent-volume-claims")
.mapType()
.noDefaultValue()
.withDescription(
Description.builder()
.text(
"The user-specified %s that will be mounted into Flink containers. "
+ "The value should be in the form of %s. "
+ "Multiple PVCs can be specified, for example: %s. "
+ "The PVCs must exist in the same namespace as the Flink cluster before deployment. "
+ "For HA setups with multiple JobManagers or TaskManagers accessing the same storage, "
+ "use PVCs with ReadWriteMany (RWX) or ReadOnlyMany (ROX) access modes.",
link(
"https://kubernetes.io/docs/concepts/storage/persistent-volumes/",
"PersistentVolumeClaims (PVCs)"),
code("pvc-name:/mount/path"),
code(
"checkpoint-pvc:/opt/flink/checkpoints,data-pvc:/opt/flink/data"))
.build());

/**
* Whether to mount PersistentVolumeClaims (PVCs) as read-only.
*
* <p>When set to true, all PVCs configured via {@link #KUBERNETES_PERSISTENT_VOLUME_CLAIMS}
* will be mounted as read-only. This is useful when the PVC contains shared data that should
* not be modified by Flink, such as reference datasets or pre-trained models.
*
* <p>Note: This setting applies globally to all PVCs configured via {@link
* #KUBERNETES_PERSISTENT_VOLUME_CLAIMS}. If you need different access modes for different PVCs,
* consider using pod templates instead.
*
* <p>Default: false (read-write mode)
*/
public static final ConfigOption<Boolean> KUBERNETES_PERSISTENT_VOLUME_CLAIM_READ_ONLY =
key("kubernetes.persistent-volume-claims.read-only")
.booleanType()
.defaultValue(false)
.withDescription(
Description.builder()
.text(
"Whether to mount PersistentVolumeClaims (PVCs) as read-only. "
+ "When set to true, all PVCs configured via '%s' will be mounted as read-only. "
+ "This is useful for shared data that should not be modified by Flink, "
+ "such as reference datasets or pre-trained models. "
+ "Note: This setting applies globally to all configured PVCs.",
text(KUBERNETES_PERSISTENT_VOLUME_CLAIMS.key()))
.build());

private static String getDefaultFlinkImage() {
// The default container image that ties to the exact needed versions of both Flink and
// Scala.
Expand Down
Loading