development : idea for large feature branches

[ggerganov]

I was thinking recently how to optimize the development process a bit. What I noticed is that some important PRs end-up in long periods of time waiting for a review to get merged into master and blocking some follow-up work. A few examples:

• Speculative decoding (eagle, mtp, d-flash)
• Deepseek v3, v4 support
• Metal backend refactoring for per-device optimizations
• NVFP4 support
• RPC over RDMA

Such big features often involve multiple steps to complete. Merging the individual steps into master, without having the complete final picture is not easy, because we want the intermediate steps to be stable (i.e. not break other unrelated stuff) and also compatible with what will follow when working towards the final feature. At the same time, we also want the maintainers to provide feedback and guidance during the development of the features, so we don't want to wait for contributors to implement the full feature before they open the first PR.

I think we can improve the current state by creating feature branches that are parallel to master and managed by a dedicated maintainer. Something like this:

• The feature branches are called dev-[feature]
• Each feature branch has a dedicated owner who is responsible to keep it up-to date with master by resolving conflicts and to merge the intermediate steps from incoming PRs
• Each feature branch has a dedicated discussion where user feedback is going to be provided
• Each PR into a feature branch can be reviewed by the related maintainers as usual, but ultimately the feature owner makes the decision when to merge changes. (we can relax the "2 approvals" setting for the feature branches)
• Once the feature is complete, a final review is performed for merging into master - same as we do now. If needed, the final implementation can additionally be split into multiple PRs when going into master, but these would be much simpler to create and review as we will have the complete picture by that point of time.
• If things work as I imagine, at some point we can consider doing release-candidate builds from the feature branches for more broader user feedback before final merge

graph TD
    subgraph "branch: master"
        M1["Base"] --> M2["Unrelated A"] --> M3["Unrelated B"] --> M4["Unrelated C"] --> M5["Final Merge"]
    end
    subgraph "branch: dev-feature"
        D1["Feature PR #1"] --> D2["Feature PR #2"] --> D3["Rebase onto master"] --> D4["Feature PR #3"] --> D5["Feature PR #4"] --> D6["Rebase onto master"] --> D7["Final feature"]
    end

    M1 -.-> D1
    M2 -.-> D3
    M3 -.-> D6
    D7 --> M5

Loading

Note that this process only makes-sense for large features that we know in advance will take multiple , non-trivial steps to complete. I.e. we don't want to have too many active features, so basic stuff will still go directly into master as it does now.

---

I will be giving this a try with the work in #24021 (comment) (branch dev-metal) and see if it makes sense. I think the DSv4 work is also a good candidate to experiment with this idea (in case @am17an or @fairydreaming want to give it a go as feature owners).

Open to suggestions and opinions. cc @ggml-org/maintainers

---

Current features: https://github.com/ggml-org/llama.cpp-dev

[CISC]

This definitely sounds like it's worth a go, should significantly reduce the slog.

Would make it a lot easier to manage PRs that require new ops and backend support as well.

[CISC]

When it's out of preview it should be worth looking into Stacked PRs as well.

[ggerganov]

Yes, I already signed up for the waitlist, but haven't received an invite.

[ORippler]

+1, the OP largely read to me like a description of what stacked PRs are supposed to be (with the absence of RCs)

[pwilkin]

Yeah, this is basically what I'm trying to do already for #21160 . I think it's a good idea and having RC builds for those branches might make it easier to debug "frontier" code changes without breaking the main branch too quickly.

[ngxson]

Generally I think the idea is worth trying. The only concern is that the dev branches should also be synced regularly with master, which can potentially add a bit more works.

[am17an]

Doesn't the author need to be able to make the feature branch in ggml-org? I tried doing this earlier but I didn't have the correct permissions.

[CISC]

Everyone with write access should be able to create a branch, what did you try?

[am17an]

I'm not sure but we ended up creating PRs on my fork with a base on the main feature branch. From what I understand this should happen on the main repo using consistent naming?