[DO-NOT-MERGE] [incubator-kie-6854] Parallel CI Jobs by tkobayas · Pull Request #6860 · apache/incubator-kie

tkobayas · 2026-07-31T01:19:30Z

Issue : Parallel run for PR check CI #6854

tkobayas · 2026-08-13T05:17:11Z

(deleted)

tkobayas · 2026-08-14T04:23:20Z

Confirmed sequential CI (original ci.yaml) and parallel CI have the same test coverage.

tkobayas · 2026-08-14T12:49:51Z

(deleted)

tkobayas · 2026-08-17T05:12:06Z

Split PR CI into 4 parallel test jobs

Problem

The monorepo PR build ran all ~850 modules sequentially in a single job, taking ~2-3h.

Solution

Split the PR test step into 4 concurrent jobs by project category: drools, optaplanner, kogito-runtimes, kogito-apps. Wall-clock time drops to ~1h15m (limited by the longest job, kogito-apps).

Key changes

script/ci/CiComputeBuildScopes.java -- Added parseModuleCategories() which reads  markers in the root pom.xml to categorize modules. Writes per-category affected-*.txt files (e.g., affected-drools.txt, affected-kogito-runtimes.txt) alongside the existing combined output.

.github/actions/ci-setup/action.yaml -- New composite action extracting shared CI setup: free disk space, checkout-pr, Java, JBang, Maven settings, changed-file collection, and build scope computation.

.github/workflows/ci.yaml -- Push-to-main remains a single full-build job. PRs fan out to 4 parallel test jobs. Each job:

Checks if its category has affected modules -- if not, skips entirely
Installs upstream dependencies without tests (-T 1C -Dquarkus.build.skip=true for speed)
Runs tests only for its own category's affected modules

.rat-excludes -- Replaced 48 individual golden file entries with a single glob script/ci/tests/scenarios-compute-build-scopes/**.

Golden files -- 48 new per-category expected files added across 12 test scenarios. Existing golden files updated for new modules on main.

Cross-category dependency handling

Each test job installs all upstream categories' affected modules (without tests) before running its own:

Job	Upstream install (no tests)
test-drools	upstream only
test-optaplanner	upstream + affected-drools
test-kogito-runtimes	upstream + affected-drools + affected-optaplanner
test-kogito-apps	upstream + affected-drools + affected-optaplanner + affected-kogito-runtimes

Build time comparison

Sequential CI (run 85920644738):

Job	Duration
Java 17	3h 05m 04s
Java 21	2h 14m 10s

Parallel CI (run 86233087403):

Job	Java 17	Java 21
Drools	44m 01s	41m 55s
OptaPlanner	15m 27s	19m 06s
Kogito runtimes	1h 12m 45s	1h 06m 30s
Kogito apps	1h 16m 13s	1h 05m 33s

tkobayas · 2026-08-17T05:59:37Z

-          echo "3c9fb9ac5823ae5ab9d136dec08b896e2a0f6f6b313689b21f43c511b7c34b85  jbang.zip" | sha256sum -c -
-          unzip -q jbang.zip
-          echo "${{ github.workspace }}/jbang-0.138.0/bin" >> $GITHUB_PATH
+        run: mvn --batch-mode --no-transfer-progress -fae -Dsurefire.redirectTestOutputToFile=true -Dfull -Dreproducible install


On push, we still use the non-parallel full build.

tkobayas · 2026-08-17T06:04:34Z

+      matrix:
+        os:   [ubuntu-latest]
+        java: [17, 21]


In each parallel job, I don't use exclude, but exact ubuntu-latest and java 17 21 for conciseness. This should be enhanced when we accept mac and windows.

tkobayas · 2026-08-17T06:05:34Z

          pl=$(paste -sd, "$MAVEN_PL_UPSTREAM_FILE")
          [ -z "$pl" ] && echo "No upstream modules. Skipping." && exit 0
-          mvn -T 1C --batch-mode --no-transfer-progress -fae -DskipTests -DskipITs -Denforcer.skip=true -Dcheckstyle.skip=true -Dformatter.skip=true -Darchunit.skip=true -Dsurefire.redirectTestOutputToFile=true -pl "$pl" install
+          mvn --batch-mode --no-transfer-progress -T 1C -fae -DskipTests -DskipITs -Dquarkus.build.skip=true -Denforcer.skip=true -Dcheckstyle.skip=true -Dformatter.skip=true -Darchunit.skip=true -Dsurefire.redirectTestOutputToFile=true -pl "$pl" install


I added -Dquarkus.build.skip=true to solve the quarkus build ordering issue with -T 1C.

tkobayas · 2026-08-17T06:08:04Z

-          [ -z "$pl" ] && echo "No affected modules. Skipping." && exit 0
+          pl=$(paste -sd, "$MAVEN_PL_AFFECTED_DROOLS_FILE")
+          [ -z "$pl" ] && echo "No affected drools modules. Skipping." && exit 0
          mvn --batch-mode --no-transfer-progress -fae -Dsurefire.redirectTestOutputToFile=true -Dfull -Dreproducible -pl "$pl" install


I don't use -T 2 for the test phase because it will raise many issues, and they will likely require a long time to fix (See #6864).

tkobayas · 2026-08-17T06:20:37Z

+    <!-- DO NOT remove the BEGIN/END comments. They are used by CiComputeBuildScopes for parallel CI job split
+         When you add a new module, put it in one of the 4 categories (drools, optaplanner, kogito-runtimes, kogito-apps) -->
+    <!-- BEGIN drools modules (auto) -->


CiComputeBuildScopes depends on these BEGIN END markers for 4 categories split. It's a little fragile, but when broken, CiComputeBuildScopes makes sure to exit(1) with an error, so we will be able to detect.

I see you're putting some guardrails in place, but I think we can do better. The "partitions" files listing leaf modules which bring a larger dependency hierarchy with them will give us a lot more flexibility, and will allow us to not couple the pom.xml with a partitioning solution for PR checks and CI.

tkobayas · 2026-08-17T07:18:08Z

@tiagobento @yesamer @Rikkola @athirakm94 @kbowers-ibm

Hi, this PR splits the PR check CI into four parallel jobs to reduce the overall CI time.

Please have a look at the above summary.

The Build time comparison is the "big impact" case where almost all modules are tested (e.g., root pom.xml change, drools-core change, etc.). But other use cases like only kogito-runtimes change would also benefit from the parallel CI.

The CI logic (CiComputeBuildScopes, ci.yaml) gets a little more complicated, but I think the gain is larger

Please review, thanks!

Rikkola · 2026-08-17T09:03:17Z

Looks like this is also making the scripts a bit easier to read, but is there an easy way to run this locally? First to test this, but also to run my own nighties on my WIP topic branches.

tkobayas · 2026-08-17T09:27:40Z

@Rikkola

is there an easy way to run this locally?

I haven't tried, but this seems to be for the purpose > https://github.com/nektos/act

Btw, sometimes I use my fork repo to run GHA jobs (e.g., tkobayas#17) So it doesn't consume my local PC resource :)

Copilot

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR updates the CI scope computation so pull requests can run tests in parallel per module category (drools/optaplanner/kogito-runtimes/kogito-apps), and updates the workflow plus golden tests to match.

Changes:

Partition “affected” modules by category in CiComputeBuildScopes and emit category-specific affected lists.
Split PR CI into four parallel test jobs (category-based) while keeping a full build on push events.
Update/extend compute-build-scopes golden scenarios and test harness to validate category partitioning.

Reviewed changes

Copilot reviewed 48 out of 61 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
script/ci/tests/scenarios-compute-build-scopes/12-quarkus-extension-pairing/expected-affected-kogito-runtimes.txt	Add golden output for kogito-runtimes affected modules in scenario 12
script/ci/tests/scenarios-compute-build-scopes/12-quarkus-extension-pairing/expected-affected-kogito-apps.txt	Add golden output for kogito-apps affected modules in scenario 12
script/ci/tests/scenarios-compute-build-scopes/12-quarkus-extension-pairing/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 12
script/ci/tests/scenarios-compute-build-scopes/11-fully-removed-subtree/expected-affected-optaplanner.txt	Add golden output for optaplanner affected modules in scenario 11
script/ci/tests/scenarios-compute-build-scopes/11-fully-removed-subtree/expected-affected-kogito-runtimes.txt	Add golden output for kogito-runtimes affected modules in scenario 11
script/ci/tests/scenarios-compute-build-scopes/11-fully-removed-subtree/expected-affected-kogito-apps.txt	Add golden output for kogito-apps affected modules in scenario 11
script/ci/tests/scenarios-compute-build-scopes/11-fully-removed-subtree/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 11
script/ci/tests/scenarios-compute-build-scopes/10-plugin-dep-as-dependency/expected-upstream.txt	Update golden upstream ordering for scenario 10
script/ci/tests/scenarios-compute-build-scopes/10-plugin-dep-as-dependency/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 10
script/ci/tests/scenarios-compute-build-scopes/09-nested-pom-under-src/expected-upstream.txt	Update golden upstream ordering for scenario 09
script/ci/tests/scenarios-compute-build-scopes/09-nested-pom-under-src/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 09
script/ci/tests/scenarios-compute-build-scopes/08-combined-mixed-changes/expected-affected.txt	Adjust golden affected ordering for scenario 08
script/ci/tests/scenarios-compute-build-scopes/08-combined-mixed-changes/expected-affected-optaplanner.txt	Add golden output for optaplanner affected modules in scenario 08
script/ci/tests/scenarios-compute-build-scopes/08-combined-mixed-changes/expected-affected-kogito-runtimes.txt	Add golden output for kogito-runtimes affected modules in scenario 08
script/ci/tests/scenarios-compute-build-scopes/08-combined-mixed-changes/expected-affected-kogito-apps.txt	Add golden output for kogito-apps affected modules in scenario 08
script/ci/tests/scenarios-compute-build-scopes/08-combined-mixed-changes/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 08
script/ci/tests/scenarios-compute-build-scopes/07-removed-module/expected-affected.txt	Adjust golden affected ordering for scenario 07
script/ci/tests/scenarios-compute-build-scopes/07-removed-module/expected-affected-optaplanner.txt	Add golden output for optaplanner affected modules in scenario 07
script/ci/tests/scenarios-compute-build-scopes/07-removed-module/expected-affected-kogito-runtimes.txt	Add golden output for kogito-runtimes affected modules in scenario 07
script/ci/tests/scenarios-compute-build-scopes/07-removed-module/expected-affected-kogito-apps.txt	Add golden output for kogito-apps affected modules in scenario 07
script/ci/tests/scenarios-compute-build-scopes/07-removed-module/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 07
script/ci/tests/scenarios-compute-build-scopes/06-aggregator-pom/expected-affected.txt	Adjust golden affected ordering for scenario 06
script/ci/tests/scenarios-compute-build-scopes/06-aggregator-pom/expected-affected-optaplanner.txt	Add golden output for optaplanner affected modules in scenario 06
script/ci/tests/scenarios-compute-build-scopes/06-aggregator-pom/expected-affected-kogito-runtimes.txt	Add golden output for kogito-runtimes affected modules in scenario 06
script/ci/tests/scenarios-compute-build-scopes/06-aggregator-pom/expected-affected-kogito-apps.txt	Add golden output for kogito-apps affected modules in scenario 06
script/ci/tests/scenarios-compute-build-scopes/06-aggregator-pom/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 06
script/ci/tests/scenarios-compute-build-scopes/05-upstream-downstream-pair/expected-affected.txt	Adjust golden affected ordering for scenario 05
script/ci/tests/scenarios-compute-build-scopes/05-upstream-downstream-pair/expected-affected-optaplanner.txt	Add golden output for optaplanner affected modules in scenario 05
script/ci/tests/scenarios-compute-build-scopes/05-upstream-downstream-pair/expected-affected-kogito-runtimes.txt	Add golden output for kogito-runtimes affected modules in scenario 05
script/ci/tests/scenarios-compute-build-scopes/05-upstream-downstream-pair/expected-affected-kogito-apps.txt	Add golden output for kogito-apps affected modules in scenario 05
script/ci/tests/scenarios-compute-build-scopes/05-upstream-downstream-pair/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 05
script/ci/tests/scenarios-compute-build-scopes/04-two-independent-leaves/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 04
script/ci/tests/scenarios-compute-build-scopes/03-root-pom/expected-affected-optaplanner.txt	Add golden output for optaplanner affected modules in scenario 03
script/ci/tests/scenarios-compute-build-scopes/03-root-pom/expected-affected-kogito-runtimes.txt	Add golden output for kogito-runtimes affected modules in scenario 03
script/ci/tests/scenarios-compute-build-scopes/03-root-pom/expected-affected-kogito-apps.txt	Add golden output for kogito-apps affected modules in scenario 03
script/ci/tests/scenarios-compute-build-scopes/03-root-pom/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 03
script/ci/tests/scenarios-compute-build-scopes/02-hub-module-source/expected-affected.txt	Adjust golden affected ordering for scenario 02
script/ci/tests/scenarios-compute-build-scopes/02-hub-module-source/expected-affected-optaplanner.txt	Add golden output for optaplanner affected modules in scenario 02
script/ci/tests/scenarios-compute-build-scopes/02-hub-module-source/expected-affected-kogito-runtimes.txt	Add golden output for kogito-runtimes affected modules in scenario 02
script/ci/tests/scenarios-compute-build-scopes/02-hub-module-source/expected-affected-kogito-apps.txt	Add golden output for kogito-apps affected modules in scenario 02
script/ci/tests/scenarios-compute-build-scopes/02-hub-module-source/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 02
script/ci/tests/scenarios-compute-build-scopes/01-leaf-module-source/expected-affected-drools.txt	Add golden output for drools affected modules in scenario 01
script/ci/tests/CiComputeBuildScopesTest.java	Teach scenario runner / golden updater to validate category-specific affected outputs
script/ci/CiComputeBuildScopes.java	Parse module category markers from root POM and emit per-category affected outputs
pom.xml	Add BEGIN/END markers around drools modules (auto) to support category partitioning
.rat-excludes	Simplify RAT exclusions for CI test scenarios via directory glob
.github/workflows/ci.yaml	Split PR checks into four parallel category jobs; keep full build on push; reuse ci-setup composite action
.github/actions/ci-setup/action.yaml	Add reusable composite action to standardize CI setup and build-scope computation

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    private static String categorizeGa(String ga, Map<String, Path> gaToDir,
+                                        Map<String, Set<Path>> categories) {
+        Path dir = gaToDir.get(ga);
+        if (dir == null) return "drools";
+        dir = dir.toAbsolutePath().normalize();
+        for (var entry : categories.entrySet()) {
+            for (Path catPath : entry.getValue()) {
+                if (dir.equals(catPath) || dir.startsWith(catPath + "/")) {
+                    return entry.getKey();
+                }
+            }
+        }
+        return "drools";
+    }


+                currentCategory = beginMatcher.group(1);
+                if (!categories.containsKey(currentCategory)) {
+                    System.err.println("ERROR: unknown module category '" + currentCategory
+                            + "' in pom.xml marker. Expected one of: " + EXPECTED_CATEGORIES);


+        for (String cat : categories.keySet()) {
+            Set<String> catAffected = affected.stream()
+                    .filter(ga -> categorizeGa(ga, gaToDir, categories).equals(cat))
+                    .collect(Collectors.toCollection(LinkedHashSet::new));
+            writeLines(partitionedPath(affectedOut, cat), catAffected);
+        }


+        for (String cat : categories.keySet()) {
+            long catCount = affected.stream()
+                    .filter(ga -> categorizeGa(ga, gaToDir, categories).equals(cat))
+                    .count();
+            sb.append(" affected-").append(cat).append("=").append(catCount);
+        }


+    <!-- DO NOT remove the BEGIN/END comments. They are used by CiComputeBuildScopes for parallel CI job split
+         When you add a new module, put it in one of the 4 categories (drools, optaplanner, kogito-runtimes, kogito-apps) -->


tiagobento

Hi @tkobayas. First of all I would like to thank you immensely for the effort and time you're putting into making our infrastructure better. It's great to see such great initiatives coming together recently.

I left a few comments which I believe will steer the direction of this partitioning mechanism you're introducing here for the better. And I'd also like to bring a thinking point for the total time spent on GitHub Actions that such parallelization introduces.

One additional, general topic:

Total GitHub Actions resource usage

Recently, we had our own PR checks and CI jobs taking a lot of time to start. We reached out to Apache Infra and were told some projects sometimes end up using too much of the GitHub Actions quota. Since we share that with all of Apache, we need to be mindful of our growth too. Sequential builds are slower from start to finish, but they take on less resources too, as there's less repeated effort compared to Parallel builds.

Probably something for us to keep in mind is to have an easy-enough mechanism to revert PR checks to sequential builds if we ever reach a GitHub Actions usage scale that will be called out by Apache Infra.

Lastly, but not least important: maybe we should try to combine partitioning and a more aggressive -T configuration? Taking advantage of multi-core builds will probably help even more, now that we're aiming to have less scope to build on each partition.

Thanks @tkobayas!

tiagobento · 2026-08-18T17:20:10Z

+        Pattern beginPattern = Pattern.compile("<!--\\s*BEGIN\\s+(\\S+)\\s+modules\\s+\\(auto\\)\\s*-->");
+        Pattern endPattern = Pattern.compile("<!--\\s*END\\s+(\\S+)\\s+modules\\s+\\(auto\\)\\s*-->");
+        Pattern modulePattern = Pattern.compile("<module>(.+)</module>");


I'm a concerned with this parsing strategy. As it relies on comments being maintained in the top-level pom.xml file. This simply mimics the old structure we had with the repos, and doesn't to separate things logically per domain. I don't think we should carry forward the old, artificial grouping we had.

Can we try to make this partitioned by domain? We can have even more partitions, like Kogito Data-Index, Kogito Jobs Service, for example.

In kie-tools, when we needed to do a similar partitioning effort in order to keep PR checks build times in check, what I did was I created partitioning files which listed the leaf modules which represented good vertical partitions of the dependency DAG.

We don't have this anymore since the 10.3.x+ stream effort began, but you can see how it used to look like in 10.2.

https://github.com/apache/incubator-kie-tools/tree/10.2.0/.github/supporting-files/ci/partitions

Note that there are 2 partition files only, with an implicit 3rd partition which contained all the modules that were not listed explicitly in neither of those 2 partition files.

tiagobento · 2026-08-18T17:24:13Z

+    <!-- DO NOT remove the BEGIN/END comments. They are used by CiComputeBuildScopes for parallel CI job split
+         When you add a new module, put it in one of the 4 categories (drools, optaplanner, kogito-runtimes, kogito-apps) -->
+    <!-- BEGIN drools modules (auto) -->


I see you're putting some guardrails in place, but I think we can do better. The "partitions" files listing leaf modules which bring a larger dependency hierarchy with them will give us a lot more flexibility, and will allow us to not couple the pom.xml with a partitioning solution for PR checks and CI.

tiagobento · 2026-08-18T17:26:38Z

+        touch "${{ runner.temp }}/maven-pl-affected-drools.txt"
+        touch "${{ runner.temp }}/maven-pl-affected-optaplanner.txt"
+        touch "${{ runner.temp }}/maven-pl-affected-kogito-runtimes.txt"
+        touch "${{ runner.temp }}/maven-pl-affected-kogito-apps.txt"


Declaring 4 fixes partitions I believe will limit the flexibility of the whole mechanism, and further constrain us into fitting new modules into these 4 boxes. Can we try to make partitions more fluid? Adding/removing a new partition should be easy enough. We can use file names to name them, and list the leaf modules which belong in each partition inside the files. E.g.,

drools.txt optaplanner.txt kogito-runtimes.txt kogito-data-index.txt kogito-jobs-service.txt

and so on.

tiagobento · 2026-08-18T17:28:28Z

+  test-kogito-runtimes:
+    if: github.event.action != 'closed' && github.event_name == 'pull_request'
+    name: 'kogito-runtimes :: ${{ matrix.os }}, Java ${{ matrix.java }}'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os:   [ubuntu-latest]
+        java: [17, 21]
+    steps:
+      - name: "SETUP :: Checkout to load local actions"
+        uses: actions/checkout@v7
+
+      - uses: ./.github/actions/ci-setup
+        with:
+          ref: ${{ github.base_ref }}
+          java-version: ${{ matrix.java }}
+
+      - name: "PR CHECK :: BUILD :: Upstream + drools + optaplanner modules (tests skipped)"
+        shell: bash
+        run: |
+          [ ! -s "$MAVEN_PL_AFFECTED_KOGITO_RUNTIMES_FILE" ] && echo "No affected kogito-runtimes modules. Skipping upstream build." && exit 0
+          upstream=$(paste -sd, "$MAVEN_PL_UPSTREAM_FILE")
+          drools=$(paste -sd, "$MAVEN_PL_AFFECTED_DROOLS_FILE")
+          optaplanner=$(paste -sd, "$MAVEN_PL_AFFECTED_OPTAPLANNER_FILE")
+          pl=$(echo "$upstream,$drools,$optaplanner" | sed 's/^,//;s/,,*/,/g;s/,$//')
+          [ -z "$pl" ] && echo "No modules to build. Skipping." && exit 0
+          mvn --batch-mode --no-transfer-progress -T 1C -fae -DskipTests -DskipITs -Dquarkus.build.skip=true -Denforcer.skip=true -Dcheckstyle.skip=true -Dformatter.skip=true -Darchunit.skip=true -Dsurefire.redirectTestOutputToFile=true -pl "$pl" install


Commenting on this one, but it applies to each of the fixed 4 partitions. There's a lot of duplication here. If we make partitions dynamic (and thus more flexible), we remove the duplication and keep things simpler to change as we evolve our codebase. We really need a CI system that's not rigid for us to have room to grow / adapt.

tkobayas · 2026-08-19T09:06:54Z

@tiagobento Thank you for the feedback. I'll work on the suggestions.

One thing:

Lastly, but not least important: maybe we should try to combine partitioning and a more aggressive -T configuration? Taking advantage of multi-core builds will probably help even more, now that we're aiming to have less scope to build on each partition.

I experimented with the -T 4 approach in #6864 and achieved a 34% speedup, but at the cost of disabling 10 flaky tests, with one hang issue still unresolved. In theory, we should fix these issues anyway, but I'm afraid it could take a long time. Also, it would be hard to keep all tests reliable under parallel Maven builds.

tiagobento · 2026-08-19T23:49:14Z

@tkobayas 34% is a HUGE gain! That combined with the parallelization this PR brings will put our PR checks into another level of speed.

The monorepo PR build ran all ~857 modules sequentially in a single job (~3h). This splits the test step into parallel jobs partitioned by project area, cutting wall-clock time to ~1h. Partition files (.github/supporting-files/ci/partitions/) list leaf modules per project area. CiComputeBuildScopes computes transitive closures and assigns each affected module exclusively: a module goes to a partition only if it is in exactly one partition's closure. Shared modules and everything else go to an implicit default partition. - 3 partition files: partition0 (optaplanner), partition1 (kogito-runtimes), partition2 (kogito-apps). Drools + shared core modules form the implicit default (no file needed). - ci-parallel.yaml: compute-partitions job outputs JSON partition array, test job uses dynamic matrix (partition × java × os cross-product). Activated by 'parallel ci' PR label. Per-partition upstream files make each job fully independent. - ci.yaml: sequential build, unchanged. Skips when 'parallel ci' label is attached. - ci-setup: new optional inputs partitions-dir and skip-build-scopes. Partition logic gated on CI_PARTITIONS_DIR env var. - Upstream builds use -T 1C and -Dquarkus.build.skip=true for speed. - Per-partition + default golden files for all 12 test scenarios. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

tkobayas · 2026-08-27T05:15:49Z

Parallel CI Test Run: Partition Files Approach

Run 89495155157 — all modules affected (root pom change), 857 modules total.

Module distribution

Partition	Affected modules	Tests
partition0 (optaplanner)	28	1,535
partition1 (kogito-runtimes)	179	2,260
partition2 (kogito-apps)	194	2,083
default (drools + shared)	452	29,432
Total	853	35,310

Job elapsed time

Job	Java 17			Java 21
	upstream	test	total	upstream	test	total
compute-partitions	—	—	2m 24s	—	—	—
partition0 (optaplanner)	3m 26s	9m 17s	15m 21s	3m 06s	7m 52s	13m 28s
partition1 (kogito-runtimes)	9m 32s	40m 21s	52m 57s	7m 26s	39m 34s	49m 28s
partition2 (kogito-apps)	14m 07s	1h 03m	1h 20m 31s	13m 12s	59m 06s	1h 14m 20s
default (drools + shared)	8m 41s	53m 32s	1h 05m 15s	8m 28s	51m 53s	1h 03m 29s

Wall-clock time = compute-partitions (2m 24s) + slowest test job (1h 20m 31s) = ~1h 23m

Note:

The upstream doesn't have -T 1C to avoid quarkus build ordering issue. (will investigate next)
default partition contains "unspecified modules and shared modules between partitions" e.g., kie-addons-springboot-persistence-jdbc. Anyway, we can tailor the partition files. This is just an example.

tkobayas · 2026-08-28T00:13:15Z

Parallel CI Test Run: Parallel Upstream with Quarkus Application Rebuild

Run 89559962903 — all jobs completed successfully.

Job elapsed time

Job	Java 17			Java 21
	upstream	test	total	upstream	test	total
compute-partitions	—	—	2m 31s	—	—	—
partition0 (optaplanner)	4m 55s	9m 52s	16m 47s	4m 30s	9m 06s	16m 00s
partition1 (kogito-runtimes)	7m 06s	43m 29s	52m 37s	7m 29s	46m 53s	57m 15s
partition2 (kogito-apps)	10m 18s	1h 04m	1h 17m 16s	10m 05s	1h 01m	1h 14m 28s
default (drools + shared)	8m 09s	56m 39s	1h 07m 18s	7m 51s	51m 20s	1h 02m 12s

The partition2 upstream figures include both upstream phases:

Java 17: parallel upstream 8m 39s + required Quarkus applications 1m 39s = 10m 18s
Java 21: parallel upstream 8m 30s + required Quarkus applications 1m 35s = 10m 05s

The workflow wall-clock time, measured from the start of compute-partitions to the end of the slowest matrix job, was approximately 1h 24m 45s. This includes an approximately five-minute scheduling gap between compute-partitions and the matrix jobs.

tkobayas · 2026-08-28T08:13:16Z

Parallel CI Test Run: Computed Image-Producer Rebuild

Run 89788266903 — all jobs completed successfully.

Job elapsed time

Job	Java 17			Java 21
	upstream	test	total	upstream	test	total
compute-partitions	—	—	3m 06s	—	—	—
partition0 (optaplanner)	1m 52s	9m 42s	14m 18s	1m 50s	9m 13s	13m 36s
partition1 (kogito-runtimes)	5m 09s	42m 09s	51m 56s	6m 21s	49m 36s	58m 50s
partition2 (kogito-apps)	8m 23s	1h 03m	1h 14m 30s	7m 04s	1h 18m	1h 29m 02s
default (drools + shared)	5m 29s	52m 50s	1h 00m 33s	5m 35s	54m 54s	1h 02m 50s

The partition2 upstream figures include both upstream phases:

Java 17: parallel upstream 6m 36s + required image-producing modules 1m 47s = 8m 23s
Java 21: parallel upstream 5m 33s + required image-producing modules 1m 31s = 7m 04s

The workflow wall-clock time, measured from the start of compute-partitions to the end of the slowest matrix job, was approximately 1h 32m 16s. The matrix jobs started approximately five seconds after compute-partitions finished.

yesamer · 2026-08-28T08:17:21Z

@tkobayas Very nice work!! It's curious that JDK21 is slower than JDK17 :o

tkobayas force-pushed the parallel-ci branch from bf8a674 to 825db97 Compare August 12, 2026 07:20

tkobayas changed the title ~~[DO-NOT-MERGE] check tested scope~~ [DO-NOT-MERGE] Parallel CI Jobs Aug 12, 2026

tkobayas mentioned this pull request Aug 12, 2026

[DO-NOT-MERGE] Test parallel mvn T option #6864

Draft

tkobayas force-pushed the parallel-ci branch 2 times, most recently from c25cc5b to 1752375 Compare August 12, 2026 09:10

tkobayas force-pushed the parallel-ci branch from bd2cf56 to 72c09c8 Compare August 14, 2026 04:42

tkobayas force-pushed the parallel-ci branch 4 times, most recently from afa5fc2 to 344885e Compare August 17, 2026 02:03

tkobayas changed the title ~~[DO-NOT-MERGE] Parallel CI Jobs~~ [incubator-kie-6854] Parallel CI Jobs Aug 17, 2026

tkobayas marked this pull request as ready for review August 17, 2026 02:04

tkobayas force-pushed the parallel-ci branch from 344885e to 95fb55e Compare August 17, 2026 02:16

tkobayas force-pushed the parallel-ci branch from 95fb55e to 649bb04 Compare August 17, 2026 05:36

tkobayas commented Aug 17, 2026

View reviewed changes

tkobayas requested review from Rikkola, kbowers-ibm, tiagobento and yesamer August 17, 2026 07:19

tkobayas mentioned this pull request Aug 18, 2026

[NO-ISSUE]: Build only the modules your changes affect, with make dev #6901

Merged

Rikkola approved these changes Aug 18, 2026

View reviewed changes

yesamer requested a balanced review from Copilot August 18, 2026 08:49

yesamer added the area:CI label Aug 18, 2026

Copilot AI reviewed Aug 18, 2026

View reviewed changes

Copilot started reviewing on behalf of yesamer August 18, 2026 09:08 View session

tiagobento requested changes Aug 18, 2026

View reviewed changes

yesamer added the status:work-in-progress DO NOT MERGE! just testing PR label Aug 20, 2026

tkobayas added the parallel ci Triggers ci-parallel.yaml GHA workflow PR check label Aug 26, 2026

tkobayas changed the title ~~[incubator-kie-6854] Parallel CI Jobs~~ [DO-NOT-MERGE] [incubator-kie-6854] Parallel CI Jobs Aug 26, 2026

tkobayas marked this pull request as draft August 26, 2026 07:20

tkobayas force-pushed the parallel-ci branch 2 times, most recently from cb5a28e to 44d0f54 Compare August 26, 2026 08:47

tkobayas force-pushed the parallel-ci branch from 44d0f54 to 70ca8c2 Compare August 27, 2026 02:30

tkobayas added 2 commits August 27, 2026 15:39

enable -T 1C

45ab8f8

2 phase upstream build to avoid -T C1 issue

533c4db

move hard-coded image-producer modules to image-producers.txt

0730096

		<!-- DO NOT remove the BEGIN/END comments. They are used by CiComputeBuildScopes for parallel CI job split
		When you add a new module, put it in one of the 4 categories (drools, optaplanner, kogito-runtimes, kogito-apps) -->

Uh oh!

Conversation

tkobayas commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tkobayas commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tkobayas commented Aug 14, 2026

Uh oh!

tkobayas commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tkobayas commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Split PR CI into 4 parallel test jobs

Problem

Solution

Key changes

Cross-category dependency handling

Build time comparison

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tkobayas Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tkobayas commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rikkola commented Aug 17, 2026

Uh oh!

tkobayas commented Aug 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

tiagobento left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tkobayas commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tiagobento commented Aug 19, 2026

Uh oh!

tkobayas commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Parallel CI Test Run: Partition Files Approach

Module distribution

Job elapsed time

Uh oh!

tkobayas commented Aug 28, 2026

Parallel CI Test Run: Parallel Upstream with Quarkus Application Rebuild

Job elapsed time

Uh oh!

tkobayas commented Aug 28, 2026

Parallel CI Test Run: Computed Image-Producer Rebuild

Job elapsed time

Uh oh!

yesamer commented Aug 28, 2026

Uh oh!

tkobayas commented Jul 31, 2026 •

edited

Loading

tkobayas commented Aug 13, 2026 •

edited

Loading

tkobayas commented Aug 14, 2026 •

edited

Loading

tkobayas commented Aug 17, 2026 •

edited

Loading

tkobayas Aug 17, 2026 •

edited

Loading

tkobayas commented Aug 17, 2026 •

edited

Loading

tkobayas commented Aug 19, 2026 •

edited

Loading

tkobayas commented Aug 27, 2026 •

edited

Loading