fix overrides dropped when --multirun flag appears between them

[Mr-Neutr0n]

When --multirun (or any optional flag) is placed between override arguments on the command line, argparse only sees the overrides before the flag and rejects the rest as unrecognized. For example:

python app.py task=1 --multirun db=mysql
# error: unrecognized arguments: db=mysql

This happens because nargs="*" positional args stop consuming tokens as soon as argparse encounters an optional flag. The overrides after the flag never get picked up.

Fixed by switching from parse_args() to parse_known_args() and merging any remaining tokens back into the overrides list. All orderings now work:

python app.py --multirun task=1 db=mysql    # works (already worked)
python app.py task=1 db=mysql --multirun    # works (already worked)
python app.py task=1 --multirun db=mysql    # works (was broken)

Fixes #3053

[Mr-Neutr0n]

Hi! Quick status: this still applies cleanly to the latest main as of today, and the PR is still mergeable. Re-requesting a review. If this looks good now, I would be happy to see it merged; if the repo has moved on, I am just as happy to close it.

[omry]

Thanks I'll get to it.

[omry]

Thanks for the fix. I agree this is a real CLI bug, and supporting overrides on both sides of --multirun is the right behavior.

Before merging, could you please make a few updates?

1. Add a news fragment for the user-visible bugfix, e.g. news/3053.bugfix.

2. Add an end-to-end regression test through an actual @hydra.main app invocation. The current parser-level test is useful, but the reported failure happens through the normal app path. It would be good to cover a command like:

   python app.py task=1 --multirun db=mysql

3. Please preserve the existing unknown-option behavior. The current implementation switches from parse_args() to parse_known_args() and appends all remaining tokens to overrides. That fixes misplaced overrides, but it also means an unrelated unknown CLI option can make it into the overrides list and probably fail later, instead of failing immediately as an unrecognized argument. For example:

   python app.py task=1 --bad-flag db=mysql

   In the current PR, --bad-flag can be appended to overrides. This should continue to fail as an unknown CLI option during argument parsing. Unknown CLI options should fail predictably; only valid Hydra overrides that appear after known flags should be accepted.

Once those are covered, this looks like a good 1.4.0 bugfix candidate.