[DSIP-109][Storage] Add Git repository as a resource storage backend

[eye-gu]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

DolphinScheduler currently supports LOCAL, HDFS, S3, OSS, GCS, ABS, OBS, COS as resource storage backends. These are all object/file storage systems. In many scenarios, users want to store and version-control workflow resources (shell scripts, SQL files, Python files, UDFs) directly in a Git repository, gaining:


1. Version history — every resource change is a Git commit with a diffable audit trail
2. GitOps workflow — resources managed through pull requests / code review
3. Git-native CI/CD integration — resources can trigger CI pipelines natively

This DSIP proposes adding a new storage plugin module dolphinscheduler-storage-git that implements the existing StorageOperator SPI, supporting GitHub, Gitea, and GitLab as storage backends via their REST Contents/Files APIs.

Design Detail

Implements StorageOperator. Maps DS file operations (create/exists/delete/copy/upload/download/list) to Git API calls via REST.

github:
https://docs.github.com/en/rest/authentication/authenticating-to-the-rest-api?apiVersion=2026-03-10
https://docs.github.com/en/rest/repos/contents?apiVersion=2026-03-10
gitlab:
https://docs.gitlab.com/api/rest/#authentication
https://docs.gitlab.com/api/repository_files/
gitea:
https://docs.gitea.com/development/api-usage
https://docs.gitea.com/api/1.25/#tag/repository/operation/repoMigrate

I can submit a PR!

Compatibility, Deprecation, and Migration Plan

This is a purely additive feature

Test Plan

GitLab/gitea can use container test.

Code of Conduct

• I agree to follow this project's Code of Conduct

[SbloodyS]

According to my experience, the reading and writing performance of git is very low compared with the existing s3, hdfs and other object storage. Frequent concurrent pulling of files on git during the task running will make git a bottleneck of the task running back pressure. How did you solve this problem?

[eye-gu]

@SbloodyS Thanks for the feedback. Writing is a low-frequency operation, and it is usually not written through DS. The problem you mentioned mainly occurs in the download process during task startup. In terms of performance, Git downloads may be slower, but resource files are usually small (KB-level scripts), it mainly depends on the network, and the overall impact is relatively small compared to the time consumption of ETL tasks. Another issue is GitHub's 5000/h traffic limit, which is sufficient for middle-scale use. If it is a self built Git repository, there is no such problem.

[SbloodyS]

There is no way to solve the back pressure in your reply, which will lead to a sharp decline in the performance of DS task parallelism.

[eye-gu]

For small-to-medium clusters where Git concurrency is not an issue, real-time download (via the REST API) works fine. If concurrency does become a bottleneck, we can introduce a config switch that lets each worker maintain a local clone refreshed by periodic git fetch, so task downloads just copy local files instead of hitting Git. The trade-off is that users need to configure a sync interval, and within that interval resources may be inconsistent.

[SbloodyS]

Git doesn't support single file or multiple file downloading. The same task instance may refer to multiple files in different folders. For git, it needs to pull extra contents of multiple projects or folders, which will cause waste of resources and performance degradation. And if a worker local cache is introduced, it needs careful design, which I haven't seen in the current issue. So before the above problems are solved, I'm -1 on this.

[eye-gu]

Git doesn't support single file or multiple file downloading. The same task instance may refer to multiple files in different folders. For git, it needs to pull extra contents of multiple projects or folders, which will cause waste of resources and performance degradation

Git can download single file through the REST API. Such as GitHub, you can check the documentation: https://docs.github.com/en/rest/repos/contents?apiVersion=2026-03-10

The design of local cache is roughly as follows:

1. Configure to open cache+interval time+local cache directory
2. When creating a GitStorageOperator, determine whether a git clone is needed, and then start a scheduled task for git fetch
3. When the worker calls download and cache enable, it reads from the local directory. Otherwise download through the REST API.

[SbloodyS]

Git can download single file through the REST API. Such as GitHub, you can check the documentation: https://docs.github.com/en/rest/repos/contents?apiVersion=2026-03-10 

It's ok to use rest api to download file.

The design of local cache is roughly as follows:

1. Configure to open cache+interval time+local cache directory
2. When creating a GitStorageOperator, determine whether a git clone is needed, and then start a scheduled task for git fetch
3. When the worker calls download and cache enable, it reads from the local directory. Otherwise download through the REST API.

This is too simple, please refer to #17703

[eye-gu]

Git local cache

flowchart TB
    subgraph DS["DolphinScheduler"]
        subgraph API["API Server (cache disabled)"]
            API_OP["GitStorageOperator<br/>upload / download / delete / copy / list"]
            API_CC["GitApiClient<br/>(REST API)"]
            API_OP -->|"A1"| API_CC
        end

        subgraph WK["Worker(s)"]
            TASK["Task"]
            WK_OP["GitStorageOperator<br/>cache.enabled?<br/>(all reads: download/fetch/list/exists/get)"]
            CACHE["GitLocalCacheManager<br/>R3: checkFreshness()<br/>R4: read local clone"]
            LOCAL["Local Git Clone<br/>(disk: cacheDir/)"]
            BG["Background Refresh<br/>B1: ls-remote → fetch --depth 1<br/>+ reset --hard (if SHA changed)<br/>every refreshIntervalSec"]

            TASK -->|"R1 read"| WK_OP
            WK_OP -->|"R2 cache enabled"| CACHE
            WK_OP -.->|"cache disabled → REST API<br/>"| API_CC2["GitApiClient<br/>(REST API)"]
            CACHE --> LOCAL
            BG -->|"reset --hard"| LOCAL
        end
    end

    REMOTE["Git Remote Repository<br/>(GitHub / Gitea / GitLab)<br/>Single Source of Truth"]

    API_CC -->|"A2 Git REST API<br/>GET / PUT / DELETE<br/>/repos/{owner}/{repo}/contents/{path}"| REMOTE
    BG -->|"B1 git fetch"| REMOTE
    BG -.->|"B0 git clone (initial)"| REMOTE
    API_CC2 -.->|"REST API"| REMOTE

Loading

Three paths:

Path A — API Server（cache disabled, read/write through by REST API）:

A1  GitStorageOperator(upload/download/delete/copy/list) → GitApiClient
A2  GitApiClient → Git Remote, via Git Rest API:
    GET / PUT / DELETE  /repos/{owner}/{repo}/contents/{path}?ref={branch}

Path R — Worker （cache enabled）:

R1  Task read request(download/fetchFileContent/listStorageEntity/exists/getStorageEntity)
    → GitStorageOperator(cache.enabled?)
R2  cache enabled  → GitLocalCacheManager
R3  checkFreshness(): now - lastRefreshTime > maxStaleness?
    ├── YES → throw exception (fail-fast, reject stale data)
    └── NO  → continue
R4  read from Local Git Clone → return content (zero network cost)
    ─────────────────────────────────────────────────────────────
    cache disabled → GitApiClient(REST API) → Git Remote

Path B — Worker （Background Refresh）:

B0  initial: git clone --depth 1  (Worker startup and local cache not exist)
B1  every refreshIntervalSec:
    ls-remote → SHA equal? skip : fetch --depth 1 + reset --hard
    success → lastRefreshTime = now
    failure → no update, timestamp ages, eventually triggers R3 stale guard

Core idea: API Server reads and writes all go through REST API (ensuring write-then-immediately-visible); Worker has only read operations, served from local cache (background periodic refresh + maxStaleness expiry protection).

• API Server cache disabled: upload, download, delete, copy, listStorageEntity, etc. all go through REST API, bypassing the cache
• Worker cache enabled: read-only; all read operations go through the local clone, zero network overhead
• Refresh (Worker): a background periodic task runs git fetch + git reset --hard, the sole refresh path

Design Decisions

Decision	Choice	Rationale
API Server cache disabled	Always REST API	Ensures user writes are immediately visible, avoids cache-induced user confusion
Worker cache enabled	Local cache	High-frequency reads during task execution; local I/O eliminates network overhead
Read-only mode	git.read-only switch, default off	Remote repo may be managed by CI, DS is read-only; the configured token may lack write permission, so writes must be rejected fast at the entry point
Cache granularity	Entire repo local clone	Git manages at the repo unit; avoids fragmentation and consistency issues of per-file caching
Refresh strategy	Background periodic refresh (sole path)	Read path triggers no network operations, deterministic performance; refresh fully decoupled from read
Staleness protection	maxStaleness threshold check	When background refresh fails consecutively beyond the threshold, reject cache reads and error out, avoiding use of stale files (likely the remote repo has issues; falling back to REST API is meaningless)
Git operation library	JGit	Pure Java, no need to install git CLI on Worker

[eye-gu]

@SbloodyS Is this plan acceptable? If there are no issues, I would be more than happy to submit the PR.

[SbloodyS]

This implementation and maintenance costs are high, but the benefits are low, which will also lead to the problem of low performance. So I'm -1 on this. Waiting for others opinion.

cc @ruanwenjun @zhongjiajie @Gallardot

[eye-gu]

Maybe can skip cache for now, so only the rest API will be called. For most scenarios, the performance of the rest API is sufficient and the maintenance cost is not high. Only when the performance bottleneck of the rest API is really reported, cache can be considered.