Tracking Issue for removing python as the main bootstrap entry point

[jyn514]

This is a tracking issue for removing the requirement on python for bootstrapping the compiler.
Note that bootstrap.py is never going to be removed altogether, since distros still need to get a bootstrap compiler from somewhere; this issue is about contributors who already have a rust toolchain installed.

The living document for this tracking issue is https://hackmd.io/j-XXBeYERuajJknd6zErPA?both.

In particular, this subsumes #71818.

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

• Implement cargo run -p bootstrap
• Move download-llvm to rustbuild: Move download-ci-llvm out of bootstrap.py #95170. This can just shell out to curl, the same way bootstrap.py does currently.
• Move download-rustc to rustbuild: Move download-rustc from python to rustbuild #96687
• Download rustfmt in rustbuild, not bootstrap.py: Using cargo run with bootstrap uses the host rustfmt, not nightly rustfmt #95136
• Make bootstrap downloadable from CI, like llvm: Distribute bootstrap in CI #98483, Download a pre-compiled bootstrap from CI #99989
• Add x.sh and x.ps1 entrypoints which don't go through bootstrap.py and download bootstrap from CI
  • Add the entrypoints Add x.sh and x.ps1 shell scripts #99992
  • Add a new bootstrap-shim binary that only loads the config info and downloads bootstrap from CI
  • Package that shim on each nightly
  • Call that shim from the shell scripts
  • Update documentation in the dev-guide to suggest x.sh / x.ps1
  • Change src/tools/x to use the new entrypoints: Make the x tool use the x and x.ps1 scripts #105844
• Manage submodules fully in rustbuild instead of bootstrap.py: Fully remove submodule handling from bootstrap.py #97513
  • Remove rustbuild from the workspace so it doesn't break when the submodules aren't cloned
• Move build locking to rustbuild: Move locking from bootstrap.py to rust bootstrap, using fd-lock #98373. See Lock bootstrap (x.py) build directory #88310 (comment) for some discussion about the current state of file locking in Rust.

If you'd like to help with this feature, feel free to join the Zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/122652-new-members/topic/Help.20with.20RRIR.20bootstrap.20entrypoint.20.2394829

Unresolved Questions

• Should we move the "Build completed successfully" output to rustbuild? If we keep it only in bootstrap.py, the new entry point will have less info; if we move it to bootstrap the info will be incomplete even for python users because it won't include the time to build rustbuild.

Implementation history

• De-couple Python and bootstrap slightly #76544
• Implement cargo run -p bootstrap: Move some more bootstrap logic from python to rust #92260
• Align rustbuild's submodule behavior with x.py's: Use shallow clones for submodules managed by rustbuild, not just bootstrap.py #93722
• Allow cargo run instead of cargo run -p bootstrap #94796
• Work-around lack of artifact dependencies when using cargo run -p bootstrap #94828
• Fix cargo run tidy #94806
• bootstrap: add rust-toolchain to the root directory #94830
• unify bootstrap and shim binaries #95164
• Move download-ci-llvm out of bootstrap.py #95170
• Make it possible to run cargo test for bootstrap #95253
• Fix cargo run on Windows #95254
• Move download-rustc from python to rustbuild #96687
• Move rustfmt downloads from bootstrap.py to rustbuild #97507
• Fully remove submodule handling from bootstrap.py #97513
• Distribute bootstrap in CI #98483
• Move locking from bootstrap.py to rust bootstrap, using fd-lock #98373
• Make all download functions need only Config, not Builder #104188
• Make the x tool use the x and x.ps1 scripts #105844

@rustbot label: +A-rustbuild +T-infra
cc @Mark-Simulacrum - I thought I'd write my plan down so you're not seeing PRs come in out of the blue.

[cuviper]

Note that bootstrap.py is never going to be removed altogether, since distros still need to get a bootstrap compiler from somewhere;

Distros generally need offline builds, so I'm not sure what role you see for bootstrap.py there. For most builds I'm just pointing to the /usr toolchain, or if that's broken or non-existent I'll use a manually-unpacked upstream toolchain.

[jyn514]

Distros generally need offline builds, so I'm not sure what role you see for bootstrap.py there. For most builds I'm just pointing to the /usr toolchain, or if that's broken or non-existent I'll use a manually-unpacked upstream toolchain.

Ah, interesting. I'm a little confused why we ever had a python entry point then - @Mark-Simulacrum do you remember the original goal? Was it intended for individual users trying to build from source? I'm ok with regressing that a little bit as long as people can still install rustup from source and then build bootstrap ...

[bjorn3]

bootstrap.py was introduced when rewriting the build system from make to rust in 046e687. Back then rustup (it was still called multirust at that point I believe) wasn't really common I believe.

[Mark-Simulacrum]

The python wrapper supports us in a number of ways (or did): submodule initialization (needed before first Cargo invocation due to workspace resolution), downloading and checking hashes on a specific beta/stable (sort of replaceable with rustup, but historically our needs are a little more complex such as with rustfmt being a separate toolchain), figuring out what the platform target triple is...

It also does more "mundane" things like permitting a single interface to rustbuild without cargo run -- ... prefixing every command, which to me feels at least not obviously better than x.py. (And configuring that initial Cargo invocation with our linting requirements etc)

[bjorn3]

Cargo allows adding aliases in .cargo/config.toml, so you can have eg cargo rustcbuild, cargo rustctest, ...

[jyn514]

It also does more "mundane" things like permitting a single interface to rustbuild without cargo run

We could have that unified interface with src/tools/x, which is also shorter than either x.py or cargo run. That seems nicer than having the interface tied to the language.

The other things are mostly done by rustup, or we could make them so - I guess the concern is for tooling in CI, which doesn't have rustup installed?

We could make submodules fully managed by rustbuild if we took it out of the workspace I think. That doesn't seem like a terribly bad regression; it mostly just means that it won't show up in cargo tree unless you run cd src/bootstrap first.

[jyn514]

I have divided this up into some smaller tasks that can be done in parallel; see https://rust-lang.zulipchat.com/#narrow/stream/122652-new-members/topic/Help.20with.20RRIR.20bootstrap.20entrypoint for more info.

@rustbot label: +E-mentor +E-help-wanted