Enhance subfiling environment checking and update usage notes by Yushan-Wang · Pull Request #654 · pdidev/pdi

Yushan-Wang · 2026-03-20T09:08:15Z

Added important notes for using subfiling correctly.

List of things to check before making a PR

Before merging your code, please check the following:

jbigot

Looks great.

Side question, but do you check the level of thread support with MPI_Query_thread to give the user a clean error if the required level is not available.

Yushan-Wang · 2026-03-25T09:10:45Z

Looks great.

Side question, but do you check the level of thread support with MPI_Query_thread to give the user a clean error if the required level is not available.

OK. The subfiling will be ignored/deactivated if the required level is not available and the simulation can go on. In this case, we can explicitly return the message of the thread level. I feel like this is more user-friendly.
Or do you prefer to stop the simulation and throw an error?

jbigot · 2026-03-25T09:32:10Z

I feel in general, going on when the user request can not be fulfilled is indeed less "user friendly" in the short run. But in the long run, if you launch lots of simulations, they behave differently on different machines, you don't understand why, and you have to dig in the log to understand, it can be dangerous. I would be in favor of sending an error by default (errors can be ignored BTW) and maybe adding an additional option in the yaml. Something like SUBFILING_IF_AVAILABLE so that the user can explicitly specify that they want it only if possible. Thoughts?

Yushan-Wang · 2026-03-25T09:38:35Z

I feel in general, going on when the user request can not be fulfilled is indeed less "user friendly" in the short run. But in the long run, if you launch lots of simulations, they behave differently on different machines, you don't understand why, and you have to dig in the log to understand, it can be dangerous. I would be in favor of sending an error by default (errors can be ignored BTW) and maybe adding an additional option in the yaml. Something like SUBFILING_IF_AVAILABLE so that the user can explicitly specify that they want it only if possible. Thoughts?

Sure! You have a point. We shall not waste computing hours without generating the desired output. Let me patch it!

jbigot

Looks great

jbigot

looks great!

jmorice91 · 2026-04-01T21:23:07Z

The shorthand configuration:

subfiling:2

is not tested in the CI. Is it a choice?

Yushan-Wang · 2026-04-02T07:19:10Z

The shorthand configuration:
subfiling:2
is not tested in the CI. Is it a choice?

Well, I think I can add a test for the short version.

benoitmartin88

I just have a little question concerning the expected result if subfiling_stripe_count is <0

jbigot

I believe this is nearly done, just a few minor comments

jmorice91

few questions

iole-bolognesi

Seems good

Yushan-Wang · 2026-04-09T07:25:04Z

@jbigot @benoitmartin88 PR is ready for the final review. Thanks

benoitmartin88

I have 2 comments.

jbigot

Loogs great!

Yushan-Wang · 2026-04-15T10:15:51Z

PR rebased. one more review to go! @benoitmartin88 @jbigot

jbigot

Looks great! Thanks

@jbigot

* Datasets as regex (pdidev#583) * First version with regex * fix pdidev#582, update authors README and indent * update changelog * adding class for test * fix indent * fix indent * fix spdlog * adding test in parallel * fix error on file * Update plugins/decl_hdf5/README.md Co-authored-by: Benoit Martin <benoitmartin88.pro@gmail.com> * Update plugins/decl_hdf5/README.md Co-authored-by: Benoit Martin <benoitmartin88.pro@gmail.com> * fix pdidev#582, fix request change --------- Co-authored-by: Benoit Martin <benoitmartin88.pro@gmail.com> * Add macOS CI (pdidev#576) * Add macOS CI * Remove chunking check on a non-chunked dataset * Use MPI oversubscription * Disable unqualified std cast call * Use float type instead of integer * Use storage instead of computation * Disable zsh test * use `H5Pget_layout` to check if dataset is chunked * remove warning : cast hsize_t to int before `printf` * Re-enable zsh test with a new env.bash path * add call to `build_and_run_all_tests` and add workaround for macos * add netcdf * initialize CMAKE_FLAGS * disable PYTHON for macos * install hdf5-mpi on macos 14 and 15 * build hdf5 from source + add -j4 * Update copyright headers * Update AUTHORS * Update CHANGELOG.md fixes pdidev#556, fixes pdidev#565, fixes pdidev#587, fixes pdidev#588 --------- Co-authored-by: Julien Bigot <jbigot@users.noreply.github.com> Co-authored-by: yushan wang <yushan.wang@cea.fr> Co-authored-by: Benoit Martin <bmartin@cea.fr> Co-authored-by: Julien Bigot <julien.bigot@cea.fr> * Fix libyaml cmake policy version (pdidev#594) * add CMAKE_POLICY_VERSION_MINIMUM to 3.5 when building libyaml * fix pdidev#593 * update changelog and add comment * Fix github action (CI) access issue of tests.xml (pdidev#584) * Fix github action (CI) access issue of tests.xml * Fix pdidev#585 github action (CI) access issue of tests.xml, and go through PR checklist * Fix pdidev#585 github action (CI) access issue of tests.xml, using an other folder 'tmp_dir_test' containing the tests output, rather than docker's tmpfs * Update AUTHORS --------- Co-authored-by: Benoit Martin <bmartin@cea.fr> * Add CODEOWNERS (pdidev#591) * Add CODEOWNERS --------- Co-authored-by: Benoit Martin <bmartin@cea.fr> Co-authored-by: yushan wang <yushan.wang@cea.fr> * Check of the HDF5 API version to use in error handler (pdidev#600) * Ensuring the use of a compatible HDF5 API versions during errors handling Fix pdidev#567 --------- Co-authored-by: Anida Khizar <anida.khizar@cea.fr> Co-authored-by: Julien Bigot <julien.bigot@cea.fr> Co-authored-by: yushan wang <yushan.wang@cea.fr> * 592 improvement hdf5 dataset as regex (pdidev#597) * fix#592, comment about dataset as regex * Fix pdidev#592, reduce the interior loop * fix copyright * add fmt/range * fix indent * sort name of dataset in case of multiple regex found * fix indent * Fix pdidev#592, dsets as vector, introduce struct for explicit dataset * change Dataset_explicit_type to class * Apply suggestion from @jbigot * Update plugins/decl_hdf5/dataset_explicit_type.h * Update plugins/decl_hdf5/dataset_explicit_type.h * Update plugins/decl_hdf5/dataset_explicit_type.h * Update plugins/decl_hdf5/dataset_explicit_type.h * add std:optional and remove unnecessary comment --------- Co-authored-by: Julien Bigot <jbigot@users.noreply.github.com> * Add workflow_dispatch in pages * Fix pdidev#598, add workflow_dispatch in pages * fix in PC parser * fix in PC parser * add test for catching possible error * catch config error in google test * Use GTest's 'ASSERT_NO_FATAL_FAILURE' to catch a write failure Co-authored-by: Julien Bigot <jbigot@users.noreply.github.com> * code cleaning * Using PDI_NULL_Handler for error catching. Co-authored-by: Benoit Martin <bmartin@cea.fr> * add rwx right to tmp_dir_test --------- Co-authored-by: Julien Bigot <jbigot@users.noreply.github.com> Co-authored-by: Benoit Martin <benoitmartin88.pro@gmail.com> * PDI Release 1.9.3 Update the embedded versions of paraconf, pybind11 & zpp to fix a build issue with recent cmake. (pdidev#632) Sort Doxyfile for easier update Update to be compatible with latest Doxygen (pdidev#629) * Make all pages subpages of the About page * Remove the hand-crafted module page and use the generated topics page instead * Update the layout file * Add missing options in the Doxyfile * Set MARKDOWN_STRICT=NO to prevent backquote errors * Set PAGE_OUTLINE_PANEL=NO to prevent generation of a useless index * Minor CSS update to handle the additional level in the generated HTML * Fix two minor doc bugs Fix pdidev#631 Fix pdidev#629 * Cleanup changelogs before release * Fix HDF5 compression test for MacOS 26 Fix pdidev#627 Co-authored-by: Julian Auriac <julian.auriac@cea.fr> * Indent * Remove Deisa plugin (pdidev#639) * Update the version of dependencies Update the version of dependencies according to our policy: oldest supported Debian, Fedora & Ubuntu, as well as spack 0.19. The new requirements are: CMake 3.22, Doxygen 1.9.1, HDF5 1.10.7, mpi4py 3.1.3, NetCDF 4.8.1, numpy 1.21.5, pyyaml 5.4.1 pybind11 2.9.1, Python 3.10.6, spdlog 1.9.2 * Rely on standard Find* from cmake as much as possible * Add policy(PUSH) in cmake to prevent leaks * Include FindHDF5 from CMake 4.1 and backport it to 3.22 to use its fix for HL Fix pdidev#613 * Fix default CMAKE_BUILD_TYPE value (pdidev#619) * Fix default CMAKE_BUILD_TYPE value * Mock PDI Fix pdidev#438 Co-authored-by: Julian Auriac <julian.auriac@cea.fr> Co-authored-by: Benoit Martin <benoit.martin2@cea.fr> * Update dependencies on googletest & benchmark * PDI Release 1.10.0 * Changelog minor fix * Adding missing LICENSE files and fix some minor indent * Support multiple consecutive calls to `find_package(PDI)` Fix pdidev#526 Co-authored-by: Benoit Martin <benoit.martin2@cea.fr> * add-compression-in-decl_netcdf (pdidev#604) Add deflate and chunking to Decl'NetCDF * Enable hdf5 subfiling2 (pdidev#650) enable HDF5 subfiling support, add test. note: to set the subfiling stripe size (64Mo for example), use `export H5FD_SUBFILING_STRIPE_SIZE= 67108864`. * add type check for reading data from netCDF file (pdidev#646) * add type and sign check for integer read * add size check for float and double variable read * Disable unit tests for JSON dependency (pdidev#651) * Update versions of the langagues standard we use (pdidev#659) Update the versions of the languages standard we use * We require versions of compilers that support more recent versions of the standard already. This is not a user facing change. * Fix for C++ 20 - some incorrect use of fmt:: are only detected now with the use of C++ 20 - C++ 20 introduced mirror operator == that lead to some abiguous calls - Bump embedded spdlog to 1.15 because 1.14 had issues with C++ 20 Fix pdidev#660 * fix pybind11 on macos (pdidev#655) * fix pybind11 linking error on macos and enable Python based plugins for macOS CI * enable netcdf in macos CI (pdidev#667) * enable netcdf in macos CI * Update clang-format to version 18 (pdidev#673) Match the version in test_env:v4 * Renamed some error codes and deprecated the old ones (pdidev#671) Fix pdidev#670 * Finish moving to clang-format 18 * Fully qualify `std::move` calls (pdidev#676) Prevent a compilation warning and incorrect usages Fix pdidev#675 Co-authored-by: yushan wang <yushan.wang@cea.fr> * Enhance subfiling environment checking and update usage notes (pdidev#654) * Enhance README with subfiling usage notes Added important notes for using subfiling correctly. * add MPI thread level check * update readme * fix * fix * update test for subfiling counting * add subfiling stripe size in test * make subfiling_stripe_size configurable in yaml * update readme * make subfiling a new class and add stripe size configuration * update readme * indent * Add subfiling.cxx to decl_hdf5 plugin sources * update upon review notes * enable subfiling check test only if supported * remove subfiling check count test * pass subfiling test from c to cxx * indent * update readme [skip ci] * remove unnecessary headers fropm test * add header in test * throw an error if stripe_size is negative * fix indent [skip ci] * use warning when stripe_size is 0 * Replace Config_error with Spectree_error * Move Spectree_error usage to a consistent location * Improved testing framework (pdidev#669) * Added `pdi/testing.h` that should be used t test plugins. * Ported `plugins/decl_hdf5/tests/decl_hdf5_tests.cxx` as an example. * Fixed a minor bug discovered along the way in Decl'HDF5 Fix pdidev#668 * Do some tests in sequential to not use all CI if everything fails (pdidev#674) * Do some tests in sequential to not use all CI if everything fails * Do not run benchmarks by default Fix pdidev#679 * Fix a bug that was introduced in mock PDI with the (unreleased) move to C++ 20 mock PDI now relies on C++ 20 features when compiling in C++ but did not advertize it * Fix target_compile_features syntax for PDI_C --------- Co-authored-by: jmorice91 <jacques.morice@cea.fr> Co-authored-by: Benoit Martin <benoitmartin88.pro@gmail.com> Co-authored-by: Thomas Padioleau <thomas.padioleau@cea.fr> Co-authored-by: Julien Bigot <jbigot@users.noreply.github.com> Co-authored-by: Benoit Martin <bmartin@cea.fr> Co-authored-by: Julien Bigot <julien.bigot@cea.fr> Co-authored-by: JAuriac <56091659+JAuriac@users.noreply.github.com> Co-authored-by: AnidaKhizar <anida-khizar@hotmail.fr> Co-authored-by: Anida Khizar <anida.khizar@cea.fr> Co-authored-by: Julian Auriac <julian.auriac@cea.fr> Co-authored-by: Benoit Martin <benoit.martin2@cea.fr>

Yushan-Wang requested a review from jbigot as a code owner March 20, 2026 09:08

jbigot previously approved these changes Mar 23, 2026

View reviewed changes

Yushan-Wang dismissed jbigot’s stale review via 5d3de33 March 25, 2026 11:40

Yushan-Wang changed the title ~~Enhance README with subfiling usage notes~~ Enhance subfiling environment checking and update usage notes Mar 25, 2026

Yushan-Wang requested review from benoitmartin88, jbigot and jmorice91 March 25, 2026 12:04

Yushan-Wang self-assigned this Mar 25, 2026

jbigot reviewed Mar 25, 2026

View reviewed changes

Comment thread plugins/decl_hdf5/file_op.cxx Outdated

jbigot previously approved these changes Mar 25, 2026

View reviewed changes

Yushan-Wang dismissed jbigot’s stale review via 12be1d2 March 26, 2026 15:20

Yushan-Wang requested a review from jbigot March 27, 2026 13:49

jbigot previously approved these changes Mar 27, 2026

View reviewed changes

benoitmartin88 reviewed Apr 2, 2026

View reviewed changes

Comment thread plugins/decl_hdf5/README.md Outdated

Comment thread plugins/decl_hdf5/file_op.cxx

Yushan-Wang dismissed jbigot’s stale review via 42534e6 April 2, 2026 08:40

jbigot reviewed Apr 2, 2026

View reviewed changes

Comment thread plugins/decl_hdf5/tests/CMakeLists.txt Outdated

Comment thread plugins/decl_hdf5/tests/CMakeLists.txt Outdated

Comment thread plugins/decl_hdf5/file_op.h Outdated

Comment thread plugins/decl_hdf5/subfiling.cxx Outdated

jmorice91 reviewed Apr 2, 2026

View reviewed changes

Comment thread plugins/decl_hdf5/file_op.cxx

Comment thread plugins/decl_hdf5/file_op.cxx

Yushan-Wang requested review from benoitmartin88 and jbigot April 2, 2026 12:37

jbigot previously approved these changes Apr 2, 2026

View reviewed changes

Comment thread plugins/decl_hdf5/tests/CMakeLists.txt

iole-bolognesi reviewed Apr 3, 2026

View reviewed changes

Comment thread plugins/decl_hdf5/subfiling.cxx

Comment thread plugins/decl_hdf5/README.md Outdated

Yushan-Wang dismissed jbigot’s stale review via b2fef1d April 3, 2026 09:41

Yushan-Wang requested a review from jbigot April 3, 2026 09:54

benoitmartin88 reviewed Apr 9, 2026

View reviewed changes

Comment thread plugins/decl_hdf5/subfiling.cxx

Comment thread plugins/decl_hdf5/README.md Outdated

Yushan-Wang added 17 commits April 15, 2026 09:40

make subfiling_stripe_size configurable in yaml

7a267df

update readme

9b5b4a8

make subfiling a new class and add stripe size configuration

1b931ba

update readme

297622a

indent

b3203bd

Add subfiling.cxx to decl_hdf5 plugin sources

be16cfb

update upon review notes

2ed55ba

enable subfiling check test only if supported

5ae58d8

remove subfiling check count test

228b202

pass subfiling test from c to cxx

a85c833

indent

80eb6a4

update readme [skip ci]

2c016ae

remove unnecessary headers fropm test

e465272

add header in test

47520c4

throw an error if stripe_size is negative

98c31e4

fix indent [skip ci]

027f4d4

use warning when stripe_size is 0

98c19f9

Yushan-Wang force-pushed the Yushan-Wang-patch-1 branch from 3339dab to 98c19f9 Compare April 15, 2026 07:40

jbigot previously approved these changes Apr 15, 2026

View reviewed changes

Merge branch 'main' into Yushan-Wang-patch-1

95cb25a

Yushan-Wang dismissed jbigot’s stale review via 95cb25a April 15, 2026 12:51

Yushan-Wang added 2 commits April 15, 2026 15:09

Replace Config_error with Spectree_error

b4fbf88

Move Spectree_error usage to a consistent location

7524577

Yushan-Wang requested review from iole-bolognesi and jbigot April 15, 2026 14:09

jbigot approved these changes Apr 15, 2026

View reviewed changes

jbigot merged commit fe1e553 into main Apr 15, 2026
37 checks passed

jbigot deleted the Yushan-Wang-patch-1 branch April 15, 2026 15:39

Conversation

Yushan-Wang commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

List of things to check before making a PR

Uh oh!

jbigot left a comment

Choose a reason for hiding this comment

Uh oh!

Yushan-Wang commented Mar 25, 2026

Uh oh!

jbigot commented Mar 25, 2026

Uh oh!

Yushan-Wang commented Mar 25, 2026

Uh oh!

Uh oh!

jbigot left a comment

Choose a reason for hiding this comment

Uh oh!

jbigot left a comment

Choose a reason for hiding this comment

Uh oh!

jmorice91 commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Yushan-Wang commented Apr 2, 2026

Uh oh!

benoitmartin88 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jbigot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jmorice91 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iole-bolognesi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Yushan-Wang commented Apr 9, 2026

Uh oh!

benoitmartin88 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jbigot left a comment

Choose a reason for hiding this comment

Uh oh!

Yushan-Wang commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbigot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Yushan-Wang commented Mar 20, 2026 •

edited

Loading

jmorice91 commented Apr 1, 2026 •

edited

Loading

Yushan-Wang commented Apr 15, 2026 •

edited

Loading