From 3748a12cc9f87d368714da0eb949cf8945252999 Mon Sep 17 00:00:00 2001 From: richardmilles Date: Fri, 7 Aug 2026 11:13:51 +0200 Subject: [PATCH] [Doc-16963] Document AWS credentials for Amazon EMR tasks Classic EMR docs omitted conf/aws.yaml / aws.emr setup, which leads to UnrecognizedClientException when adding steps. --- docs/docs/en/guide/task/emr.md | 37 ++++++++++++++++++++++++++++++++++ docs/docs/zh/guide/task/emr.md | 37 ++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/docs/docs/en/guide/task/emr.md b/docs/docs/en/guide/task/emr.md index de6c0d309665..7e322f512ecd 100644 --- a/docs/docs/en/guide/task/emr.md +++ b/docs/docs/en/guide/task/emr.md @@ -100,6 +100,43 @@ stepsDefineJson example } ``` +## AWS Authentication Configuration + +The Amazon EMR task (EMR on EC2) reads AWS credentials from the DolphinScheduler Worker `aws.yaml` configuration file, under the `aws.emr` section at `conf/aws.yaml`. + +Without valid credentials, task submission fails with errors such as: + +```text +AmazonElasticMapReduceException: The security token included in the request is invalid +(Error Code: UnrecognizedClientException) +``` + +### Using IAM Role (Recommended) + +If the DolphinScheduler Worker node runs on an EC2 instance with an attached IAM Role that can call EMR APIs: + +```yaml +aws: + emr: + credentials.provider.type: InstanceProfileCredentialsProvider + region: us-east-1 +``` + +### Using Access Key + +If you need to authenticate using AK/SK: + +```yaml +aws: + emr: + credentials.provider.type: AWSStaticCredentialsProvider + access.key.id: your-access-key-id + access.key.secret: your-secret-access-key + region: us-east-1 +``` + +> **Note**: The `aws.emr` section is shared by both EMR on EC2 and EMR Serverless task types. See also [Amazon EMR Serverless](emr-serverless.md). + ## Notice - Failover on EMR Task type has not been implemented. In this time, DolphinScheduler only supports failover on yarn task type . Other task type, such as EMR task, k8s task not ready yet. diff --git a/docs/docs/zh/guide/task/emr.md b/docs/docs/zh/guide/task/emr.md index 5e29e58cc168..4582279f2e98 100644 --- a/docs/docs/zh/guide/task/emr.md +++ b/docs/docs/zh/guide/task/emr.md @@ -95,6 +95,43 @@ stepsDefineJson 参数样例 } ``` +## AWS 认证配置 + +Amazon EMR(EMR on EC2)任务通过 DolphinScheduler Worker 的 `aws.yaml` 配置文件读取 AWS 认证信息,配置路径为 `conf/aws.yaml` 中的 `aws.emr` 段。 + +若凭证缺失或无效,任务提交会失败,常见错误类似: + +```text +AmazonElasticMapReduceException: The security token included in the request is invalid +(Error Code: UnrecognizedClientException) +``` + +### 使用 IAM Role(推荐) + +如果 DolphinScheduler Worker 节点运行在 EC2 实例上,并已绑定可调用 EMR API 的 IAM Role,配置如下: + +```yaml +aws: + emr: + credentials.provider.type: InstanceProfileCredentialsProvider + region: us-east-1 +``` + +### 使用 Access Key + +如果需要使用 AK/SK 方式认证: + +```yaml +aws: + emr: + credentials.provider.type: AWSStaticCredentialsProvider + access.key.id: your-access-key-id + access.key.secret: your-secret-access-key + region: us-east-1 +``` + +> **注意**:`aws.emr` 段的配置同时被 EMR on EC2 和 EMR Serverless 任务类型共享。也可参考 [Amazon EMR Serverless](emr-serverless.md)。 + ## 注意事项: - EMR 任务类型的故障转移尚未实现。目前,DolphinScheduler 仅支持对 yarn task type 进行故障转移。其他任务类型,如 EMR 任务、k8s 任务尚未准备好。