Skip to content

docs: describe workflow artifact downloads #395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
njzjz-bot wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs: describe workflow artifact downloads #395

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/quickcli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,36 @@ INFO:root:steps iter-000001--prep-run-explore--------------------- finished
The artifacts can be downloaded on-the-fly with `-d` flag. Note that the existing files are automatically skipped if one sets `dflow_config["archive_mode"] = None`.


## Download workflow results

The `download` command retrieves training, exploration, and labeling artifacts without requiring direct access to the workflow storage backend. List the supported artifact names first:

```bash
dpgen2 download input.json WFID --list-supported
```

Running without filters downloads every supported artifact from every successful iteration. Use iteration and artifact filters for a smaller result set:

```bash
dpgen2 download input.json WFID \
--iterations 0-2 \

@wanghan-iapcm wanghan-iapcm Sep 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0-2 is a half-open range: it expands to iterations 0 and 1, not 0, 1 and 2.

expand_idx in dpgen2/entrypoint/common.py does ret += range(int(range_str[0]), int(range_str[1]), step), and Python's range excludes the upper bound. This is also why the help text has to write the awkward -i 0-8 8 9 to cover iterations 0 through 9.

This page already knows how to say it - the resubmit section reads "0<=id<41, note that 41 is not included". The new section is the only range example on the page without that caveat, and a reader copying this command will believe they downloaded the last iteration when they did not.

--step-definitions \
prep-run-train/output/models \
prep-run-train/output/lcurves \
prep-run-train/output/logs \
prep-run-explore/output/trajs \
prep-run-explore/output/model_devis \
prep-run-fp/output/labeled_data \
--prefix results
```

Files are organized below `results/iter-000000/<step>/<input-or-output>/<artifact>`. Existing completed downloads are skipped by default; pass `--no-check-point` to request them again. The corresponding result groups are:

- training: models, learning curves, logs, and generated scripts;
- exploration: trajectories, model deviations, logs, and extra outputs;
- labeling: input configurations, labeled data, logs, and extra outputs.
Comment on lines +65 to +69

@wanghan-iapcm wanghan-iapcm Sep 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hand-written list does not match op_download_setting, and it will drift further.

  • collect-data is missing entirely. It is the fourth entry in the registry (.add_output("iter_data") - the accumulated training dataset, arguably the most useful thing to download), and --list-supported prints it. Line 50 says a no-filter run fetches "every supported artifact", so a reader will take these bullets as the inventory and never learn collect-data/output/iter_data exists.
  • The exploration bullet is already wrong against master. 1f21e7c (Add PLUMED CV-aware candidate filtering #372, 2026-09-06) added .add_output("plm_output") to prep-run-explore. That commit is not an ancestor of this branch's base, the files do not overlap, so git merges it silently and the bullet lands incomplete with nothing to flag it.
  • The training bullet lists outputs only, dropping init_models, init_data and iter_data - while the labeling bullet does include its input (confs -> "input configurations"). Same registry, two different conventions in adjacent lines. Note your own main.py example in this commit uses prep-run-train/input/init_data, an artifact these bullets never mention.

The section already tells the reader to run --list-supported first, which is the authoritative and self-maintaining answer. Consider naming the four step groups and leaving the artifact enumeration to that command, rather than copying a list that has to be re-checked on every registry change.



## Show the keys of steps

Each dpgen2 step is assigned a unique key. The keys of the finished steps can be checked with `showkey` command
Expand Down
6 changes: 3 additions & 3 deletions dpgen2/entrypoint/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,13 +149,13 @@ def main_parser() -> argparse.ArgumentParser:
1. list all supported steps and their input/output artifacts
$ dpgen2 download CONFIG ID -l

2. donwload all the input/output of all the steps.
2. download all supported input/output artifacts of all successful steps.
$ dpgen2 download CONFIG ID

3. donwload specified input/output artifacts of certain steps. For example
3. download specified input/output artifacts of certain steps. For example
$ dpgen2 download CONFIG ID -i 0-8 8 9 -d prep-run-train/input/init_data prep-run-explore/output/trajs

The command will download the init_data of prep-run-train's input and trajs of the prep-run-explore's output from iterations 0 to 9 (by -i 0-8 8 9).
The command downloads prep-run-train init_data and prep-run-explore trajectories from iterations 0 to 9 (selected by -i 0-8 8 9).
The supported step and the names of input/output can be checked by the -l flag.
"""
)
Expand Down
31 changes: 31 additions & 0 deletions tests/entrypoint/test_argparse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,37 @@ def test_dld(self):
self.assertEqual(parsed.keys, ["foo", "bar", "tar"])
self.assertEqual(parsed.prefix, "myprefix")

def test_download_by_definition(self):
parsed = self.parser.parse_args(
[
"download",
"input.json",
"workflow-id",
"--iterations",
"0-2",
"4",
"--step-definitions",
"prep-run-train/output/models",
"prep-run-explore/output/trajs",
"prep-run-fp/output/labeled_data",
"--prefix",
"results",
"--no-check-point",
]
)

self.assertEqual(parsed.iterations, ["0-2", "4"])
self.assertEqual(
parsed.step_definitions,
[
"prep-run-train/output/models",
"prep-run-explore/output/trajs",
"prep-run-fp/output/labeled_data",
],
)
self.assertEqual(parsed.prefix, "results")
self.assertFalse(parsed.no_check_point)

def test_resubmit(self):
parsed = self.parser.parse_args(
[
Expand Down