CAP-01 error for unet3d data writes to s3

[austingnanaraj]

'verification': None}^[[0m
^[[0m2026-06-26 20:40:49|INFO:dlio:154: --object mode: loading credentials from /root/storage/.env^[[0m
^[[0m2026-06-26 20:40:49|INFO:dlio:214: --object mode: injected storage params (storage_type=s3, storage_root=unet1, library=s3dlio, uri_scheme=s3, force_path_style=True)^[[0m
^[[0m2026-06-26 20:40:49|INFO:dlio:307: skip_listing enabled: 8,511,284 train files → validation_interval=1,000 (~8,513 HEAD checks at startup)^[[0m
^[[0m2026-06-26 20:40:49|DEBUG:dlio:494: Object storage (s3): skipping local directory creation for unet3d — path is an S3 key prefix, not a filesystem path.^[[0m
^[[0m2026-06-26 20:40:49|VERBOSER:mlps_logging:101: Instantiated the Training Benchmark...^[[0m
^[[0m2026-06-26 20:40:49|INFO:progress:181: Stage 1/4: Validating environment......^[[0m
^[[0m2026-06-26 20:40:49|INFO:progress:189: Stage 2/4: Collecting cluster info......^[[0m
^[[0m2026-06-26 20:40:49|DEBUG:base:653: Skipping start cluster collection (conditions not met)^[[0m
^[[0m2026-06-26 20:40:49|DEBUG:base:515: Skipping cluster info collection (conditions not met)^[[0m
^[[0m2026-06-26 20:40:49|DEBUG:dlio:97: Using CLI args for cluster info (MPI collection not available)^[[0m
^[[0m2026-06-26 20:40:49|INFO:dlio:614: CAP-01 deferred: unable to determine system memory ('Namespace' object has no attribute 'client_host_memory_in_gb'). Re-run with --client-host-memory-in-gb to enable the disk-capacity check.^[[0m
^[[1;34m2026-06-26 20:40:49|STATUS:mlps_logging:101: Writing metadata for benchmark to: /tmp/mlperf_results/closed/a/results/o/training/unet3d/datagen/20260626_204048/training_20260626_204048_metadata.json^[[0m
^[[1;31m2026-06-26 20:40:49|ERROR:main:501: [E401] CAP-01: cannot determine free space — no valid parent for unet3d
Details: Path: unet3d; Operation: cap01-check
Suggestion: Verify the path exists and is accessible^[[0m
^[[0m2026-06-26 20:40:49|INFO:main:503: Suggestion: Verify the path exists and is accessible^[[0m

mlpstorage closed training unet3d datagen object
--params storage.storage_type=s3 --debug
--data-dir unet3d
--results-dir /tmp/mlperf_results
--num-processes 64
--hosts
--params storage.storage_type=s3
--params storage.storage_root=
--params dataset.num_files_train=
--params dataset.num_samples_per_file=1
--params dataset.record_length=146600628
--params dataset.record_length_stdev=68341808
--params dataset.record_length_resize=0
--mpi-params "-x AWS_ACCESS_KEY_ID -x AWS_SECRET_ACCESS_KEY -x AWS_REGION -x S3_ENDPOINT_URIS -x DLIO_LOG_LEVEL -x RUST_LOG -x UCX_LOG_LEVEL -x MLPS_CLUSTER_COLLECTOR_SHARED_STAGING --map-by slot --mca pml ob1 --mca btl tcp,self --mca btl_tcp_if_exclude lo,docker0,virbr0 --mca btl_tcp_endpoint_timeout 1200 --mca btl_tcp_connect_timeout 600 --mca btl_tcp_sndbuf 131072 --mca btl_tcp_rcvbuf 131072 --mca oob_tcp_if_exclude lo,docker0,virbr0 --mca oob_tcp_connect_timeout 600 --mca btl_openib_allow_ib 0 --mca btl_openib_warn_no_device_params_found 0 --mca orte_tmpdir_base /tmp --mca plm_rsh_agent ssh --mca orte_base_help_aggregate 0"
--allow-run-as-root

[FileSystemGuy]

@austingnanaraj I'm not sure what you are asking, do mean to say that the benchmark didn't run? If so, please resolve all of the error messages by changing the way you run the benchmark, and then create a new PR.

There are several error messages in the output you included, could you please try to resolve those error messages?
For example:
2026-06-26 20:40:49|INFO:dlio:614: CAP-01 deferred: unable to determine system memory ('Namespace' object has no attribute 'client_host_memory_in_gb'). Re-run with --client-host-memory-in-gb to enable the disk-capacity check.
You need to give the size in GB of the amount of DRAM in the client system with the largest amount of DRAM via that CLI argument.

Have you read the ManPage.md file?

You're specifying --params storage.storage_type=s3 when the "object" as the 5th argument to the command is sufficient.

All of the following argument are inappropriate for a closed submission and will invalidate that submission (ie: it will be rejected):

--params dataset.num_files_train=
--params dataset.num_samples_per_file=1
--params dataset.record_length=146600628
--params dataset.record_length_stdev=68341808
--params dataset.record_length_resize=0

The --num-processes 64 tells the benchmark to create 64 virtual GPUs on each client system, which likely will not fit into your system's RAM capacity. Please consider starting with 8 or 12 and seeing the benchmark run before you try to stretch it to higher performance levels.

I believe, but cannot confirm because I don't have access to your lab, that all of the --mpi-params values are not required and will likely confuse things. Please try the benchmark without any of those parameters and only add the ones back that you truly need.

Please try this:
mlpstorage closed training unet3d datagen object --data-dir unet3d --results-dir /tmp/mlperf_results --num-processes 8 --hosts <_list-of-hosts_>