compiletest: more granular `--nocapture` for `run-make` tests

[jieyouxu]

Would it be possible to add a way to revert back to the old behavior? For cg_clif this makes testing the rustc test suite very verbose. I have to use --nocapture to see any test failures in the first place as panic=abort means that without it compiletest aborts before it gets a chance to print the test failures, but I would like to not have my terminal getting spammed with the output of all passing tests.

Originally posted by @bjorn3 in #134111 (comment)

[jyn514]

I have to use --nocapture to see any test failures in the first place as panic=abort means that without it compiletest aborts before it gets a chance to print the test failures

can we fix this more directly in compiletest? for instance spawn every test in a process instead of a thread, so aborting doesn't break things? then you wouldn't need to mess with --nocapture at all.

[bjorn3]

That would be a great solution! In fact libtest already has builtin support for this through -Zpanic-abort-tests, however libtest only supports this for TestFn::StaticTestFn (as #[test] uses), but not for TestFn::DynTestFn (which compiletest needs to use as it dynamically discovers the set of tests), so compiletest probably needs custom logic to reimplement this.

[jieyouxu]

Unfortunately I don't think that's going to be very easy in the short-term...

[jieyouxu]

@bjorn3 so I don't think the "spawn every test in a process instead of the thread" is super easy atm because compiletest is married to libtest too hard atm. However, do you think that untying the --nocapture for run-make tests in favor of something more like --verbose-test-output (so that --nocapture, which is actually a libtest flag, continues to work more like the old behavior), would address your use case?