Compiler doesn't work when using Docker

[jethrogb]

When I run this sequence of commands:

docker run --rm -it ubuntu:xenial
apt update
apt install wget
wget https://sh.rustup.rs -O rustup.sh
chmod ugo+x rustup.sh 
./rustup.sh -y --default-toolchain nightly
source $HOME/.cargo/env
rustc -vV

I get this error:

rustc 1.40.0-nightly (7979016af 2019-10-20)
binary: rustc
commit-hash: 7979016aff545f7b41cc517031026020b340989d
commit-date: 2019-10-20
host: x86_64-unknown-linux-gnu
release: 1.40.0-nightly
error: failed to find a `codegen-backends` folder in the sysroot candidates:
* /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu
* /root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu

This happens due to the statx syscall failing with EPERM. I believe Docker uses seccomp to limit which system calls may be made, and the statx call is too new, so it's not whitelisted. Because the syscall fails with EPERM instead of ENOSYS, the fallback to regular stat doesn't work.

Host kernel: 4.15.0-65-generic

docker version

Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.6.2
 Git commit:   f5ec1e2
 Built:        Thu Jul  5 23:07:48 2018
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.6.2
 Git commit:   f5ec1e2
 Built:        Thu Jul  5 23:07:48 2018
 OS/Arch:      linux/amd64
 Experimental: false

[Mark-Simulacrum]

Nominating for libs team as this is a recent regression caused by #65094 -- cc @alexcrichton

Personally this leads me to believe we should either be happier to fallback to the previous code (e.g., for any error) or revert the PR entirely since breaking usage of Rust code in Docker is not really feasible (even if this is arguably a Docker bug).

[jethrogb]

(e.g., for any error)

Alternatively, we can try statx on something that should always succeed (maybe /?)

[mati865]

Cannot it match ENOSYS or EPERM just like getrandom syscall does?

rust/src/libstd/sys/unix/rand.rs

Line 51 in ad7c55e

} else if err == libc::ENOSYS || err == libc::EPERM {

[jethrogb]

Unlike for getrandom, getting EPERM is not sufficient evidence that statx doesn't work.

[alexcrichton]

Thanks for investigating this down to statx being the culprit, definitely makes sense!

I was searching around for other instances of this, and definitely turns out we're not the only ones running into this

• statx syscalls inside a docker container docker/for-linux#208 - another docker user reporting the same issue. It's pointed out that Docker 18.04 apparently allows statx by default
• https://bugreports.qt.io/browse/QTBUG-70557?focusedCommentId=419518&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel - another user with qt's build system finding that seccomp disallows a new syscall
• Looks like musl uses statx for fstatat (called below) and only handles ENOSYS
• Seccomp itself only somewhat recently got support for statx as a sycall in general

Overaall I don't think there's a lot of prior art for this to draw from, it seems that everyone's working around the seccomp issue rather than addressing it directly.

I think a reasonable solution might be to do something like try to stat AT_CWD initially and if that fails with EPERM or ENOSYS we disable the syscall entirely, otherwise it's cached as always good to use. @oxalica would you be interested in helping to implement this?

[oxalica]

I think a reasonable solution might be to do something like try to stat AT_CWD initially and if that fails with EPERM or ENOSYS we disable the syscall entirely, otherwise it's cached as always good to use. @oxalica would you be interested in helping to implement this?

@alexcrichton
Do you mean to check current working directory instead of the actual argument before the first call, and regard EPERM and ENOSYS as statx not implemented?
I'm glad to fix it.

[alexcrichton]

@oxalica yeah that's what I'm thinking, the idea being that if we don't know whether statx works we invoke it with a path which we know should work almost all the time (like the cwd) and if it returns EPERM in that situation we can be pretty certain we're seccomp blocked.

[mguillemot-tel]

We are encountering a similar issue while building an Docker image with rustc & cargo on CircleCI. We are basically doing:

rustup-init -y --no-modify-path --default-toolchain nightly
rustup --version
cargo --version
rustc --version
cargo fmt --version

and are getting:

rustup 1.20.0 (a7f257941 2019-10-14)
cargo 1.40.0-nightly (3a9abe3f0 2019-10-15)
rustc 1.40.0-nightly (7979016af 2019-10-20)
error: no such subcommand: `fmt`

This errors happens since yesterday. Our last successful build with the same script was 2 days ago, with these versions:

rustup 1.20.0 (a7f257941 2019-10-14)
cargo 1.40.0-nightly (3a9abe3f0 2019-10-15)
rustc 1.40.0-nightly (c23a7aa77 2019-10-19)
rustfmt 1.4.9-nightly (33e3667 2019-10-07)

So basically between rustc c23a7aa77 2019-10-19 and rustc 7979016af 2019-10-20, something changed that makes cargo unable to figure out that rustfmt is installed.

After many tests on different Docker hosts, we haved narrowed the issue to the following: if the Docker host is using the overlay2 or btrfs storage drivers, everything works perfectly; if it is using aufs storage driver, then cargo fmt basically cannot see that the ~/.cargo/bin/cargo-fix file exists, and treats rustfmt as not installed.