Skip to content

Enable more tests with embedded backend - #1290

Open
havogt wants to merge 16 commits into
mainfrom
try-gt4py-premap-fix
Open

Enable more tests with embedded backend#1290
havogt wants to merge 16 commits into
mainfrom
try-gt4py-premap-fix

Conversation

@havogt

@havogt havogt commented May 28, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@havogt havogt changed the title GT4Py embedded: test premap fix GT4Py embedded: test premap fix, concat_where, as_offset May 29, 2026
@havogt havogt changed the title GT4Py embedded: test premap fix, concat_where, as_offset Test gt4py/main (to become 1.1.11) Jun 2, 2026
@philip-paul-mueller philip-paul-mueller mentioned this pull request Jun 2, 2026
Draft
@philip-paul-mueller

philip-paul-mueller commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

I found out that 56c6b04 introduced changes that made the CI fail.
Furthermore, I saw that this changed the concat_where conditions, since that part is rather delicate, I would also propose to run blueline to see if there are regression before merging.

@edopao edopao mentioned this pull request Jun 12, 2026
Closed
@edopao

edopao commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

cscs-ci run dace

@edopao edopao mentioned this pull request Jun 12, 2026
Closed
@edopao

edopao commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

cscs-ci run default

@edopao edopao changed the title Test gt4py/main (to become 1.1.11) Test premap fix from gt4py/main (to become 1.1.11) Jun 17, 2026
@edopao
edopao force-pushed the try-gt4py-premap-fix branch from 88575d3 to 6a14776 Compare June 17, 2026 10:11
@edopao

edopao commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

I have moved a minimal set of changes to #1327 for the upgrade to gt4py/main. I have left in this PR the changes related to testing the premax fix.

@edopao
edopao changed the base branch from main to update_to_gt4py_1.1.11 June 17, 2026 10:14
@edopao
edopao deleted the branch main June 19, 2026 06:03
@edopao edopao closed this Jun 19, 2026
@edopao edopao reopened this Jun 19, 2026
@edopao
edopao changed the base branch from update_to_gt4py_1.1.11 to main June 19, 2026 07:03
@edopao edopao changed the title Test premap fix from gt4py/main (to become 1.1.11) Test premap fix from gt4py v1.1.11 Jun 19, 2026
@msimberg

msimberg commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

@edopao @havogt can we close this since we're at gt4py 1.2.1 now?

@edopao edopao changed the title Test premap fix from gt4py v1.1.11 Test premap fix Sep 2, 2026
@edopao

edopao commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

@edopao @havogt can we close this since we're at gt4py 1.2.1 now?

I have rebased this PR on latest main, but @havogt knows more about it.

@edopao

edopao commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

@edopao @havogt can we close this since we're at gt4py 1.2.1 now?

I have rebased this PR on latest main, but @havogt knows more about it.

The concat_where expressions touched by this PR might also be touched by the rewrites with multiple output domain planned for this cycle.

@edopao

edopao commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

@edopao @havogt can we close this since we're at gt4py 1.2.1 now?

I have rebased this PR on latest main, but @havogt knows more about it.

The concat_where expressions touched by this PR might also be touched by the rewrites with multiple output domain planned for this cycle.

At the same time, enabling the tests on the embedded backend is beneficial to detect boundary issues while we introduce multiple output domains. This is an argument for proceeding with this change.

edopao and others added 5 commits September 3, 2026 09:55
@edopao
Merge branch 'main' into try-gt4py-premap-fix
09e0967
@edopao
undo extra change
630c336
@havogt @claude
Merge remote-tracking branch 'upstream/main' into premap-helpers
3bb0ce7 
Resolves the modify/delete conflict on test_compute_tangential_wind.py:
#1452 moved it to common, so the marker removal is applied there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BgWWd2xeKd9JnJK3rHtegH
@havogt @claude
Simplify the embedded workarounds at the PR's own sites
ac6a05a 
Plain typed scalars instead of broadcast on the bounded side of the
mask, natural argument order where both branches are fields, and the
horizontal window applied once as the outer selection in the diffusion
of w.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BgWWd2xeKd9JnJK3rHtegH
@havogt @claude
Give scalar concat_where branches the domain of a field, and re-triag…
662de5a 
…e the embedded markers

value_of_size_*(value, field) is value on the domain of field: the embedded
backend cannot bound a scalar (or broadcast) branch of concat_where on an open
mask side (gt4py 1.2.2). Applied at every such site; with_boundaries_on_half_levels_on_cells
no longer needs a dummy scalar at all.

Markers, measured on embedded with the marker filters disabled: every
uses_concat_where whose test now passes is removed; the remaining ones are the
vertically implicit solver's per-output tuple domain with a scan (icon4py-f27),
the unverified mpi and muphys tests. embedded_remap_error is reinstated for the
tests that fail in a shift or neighbor gather (diffusion time step, tracer
advection single step, the driver), with the cause at each marker.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BgWWd2xeKd9JnJK3rHtegH
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown

When developing, you can test your changes on CSCS CI before merge with the default pipeline: cscs-ci run default. This will run a default subset of tests.

You can pass options to override pipeline variables, for example:

  • cscs-ci run default;BACKENDS=gtfn_cpu;LEVELS=unit
  • cscs-ci run default;MODEL_SUBPACKAGES=common:driver;SESSIONS=model

Avoid running the pipeline for all tests when you are developing.

Available options are:

  • SESSIONS: model, model_mpi, or tools (correspond to nox sessions)
  • MODEL_SUBSETS: datatest, basic, or stencils (correspond to nox session selections)
  • MODEL_SUBPACKAGES: subpackages for non-MPI tests (last component, e.g. diffusion, driver)
  • MODEL_MPI_SUBPACKAGES: subpackages for MPI tests (as above)
  • BACKENDS: backends
  • GRIDS: grids for stencil tests (simple, icon_regional, or icon_global)
  • LEVELS: testing level for non-stencil tests (unit or integration)

For each option, all can be used as a shorthand for all possible values of that variable, e.g. LEVELS=all.

Multiple values can be given to each option with : used as the separator (; separates options and , separates pipelines).

See scripts/python/generate_ci_pipeline.py and noxfile.py for available values for each option.

The all pipeline can be run with cscs-ci run all. This will run all icon4py tests in CSCS CI which can be expensive. This pipeline runs on a schedule on main, and can be run when extensive validation is needed (e.g. before releases).

Merging

Once your PR is approved and ready for merging, add it to the merge queue. The merge CSCS CI pipeline will run automatically on the merge-queue branch and must pass before the PR is merged. A dummy merge check will be triggered on the PR itself since it's required to add a PR to the merge queue.

Optional Tests

To run benchmarks you can use:

  • cscs-ci run benchmark-bencher

For more detailed information please look at CI in the EXCLAIM universe.

@havogt havogt changed the title Test premap fix Enable more tests with embedded backend Sep 11, 2026
This was referenced Sep 11, 2026
Draft
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@havogt @philip-paul-mueller @edopao @msimberg