Ability from the top-level of the compilation not to mark #[no_mangle] items exported from shared library

[hsivonen]

Firefox currently builds all its Rust code into a static artifact called gkrust which is then statically linked, with cross-language LTO, with C++ artifacts to form the shippable libxul.so/dylib/dll.

Upon examining the exported symbols of libxul, the FFI functions that are meant for internal glue between Rust and C++ show up.

This is a problem for two reasons:

1. It's a binary size issue, because cross-language LTO is supposed to inline the FFI functions into their callers. However, having them exported means also keeping those copies around. Also, unused FFI functions can't be eliminated as dead code.
2. It gives problematic third-party software more opportunities to hook into libxul in unsupported ways.

Since there's a variety of third-party crates that go into libxul, it wouldn't be practical to reannotate each FFI function. Therefore, please add a way from the top level of the Rust compilation to turn off the shared library export metadata generation for #[no_mangle] items.

[RReverser]

Ran into this today again, because I defined my own malloc-like allocator function and added extern "C" for the calling convention (it's meant to be passed as a callback over FFI), but since I called it malloc, it ended up being a linker error due to a clash with an actual "malloc".

For now, I can try to prefix it with something reasonably unique, but I'd really rather have private functions stay private and not have to worry about mangling their names manually.

[bjorn3]

#[no_mangle] is for when your functions are not private.

[RReverser]

@bjorn3 Right, the situation is a bit different from this issue / my original comment it refers to, but the point is still the same in that #[no_mangle] shouldn't affect privacy but only name generation.

[bjorn3]

Mangling is the way to achieve privacy. While ELF and other object formats technically have different visibility modes, those are less useful for Rust than for C. Rust splits up a single crate into multiple codegen units, so any function used in a different codegen unit from the one defining it has to have an external linkage, making it available to all object files used in the linker invocation, thus your malloc would affect all crates not just your own. In addition there is the concept of visbility in ELF, which allows you to hide symbols from the linked library, but this is still not invisible to other crates. If you want to override the memory allocator for other crates, then you should use #[global_allocator]. If you only override malloc for Rust code, that will cause inconsistencies where an explicit malloc is used in C/Rust and then free on the other end of ffi.

[RReverser]

which allows you to hide symbols from the linked library, but this is still not invisible to other crates

This is all I want (as well as what @hsivonen describes). I want to be able to expose functions to the C library that is statically linked in as part of build.rs, but I don't want them to end up being publicly visible out of my final artifact.

[bjorn3]

If you only override malloc for Rust code, that will cause inconsistencies where an explicit malloc is used in C/Rust and then free on the other end of ffi.

This problem extends to overriding it for both Rust code and statically linked C code without also overriding it for dynamically linked C code like libc.

[luser]

To speak to what we did in C/C++ for Firefox, we used the -fvisibility=hidden compiler flag to default all symbols to hidden visibility, and then explicitly annotated public symbols with __attribute__ ((visibility ("default"))). We also implemented a system to generate wrappers for system headers whose declarations do not contain visibility annotations. The wrapper headers boil down to basically:

#pragma GCC visibility push(hidden)
#include_next <real_header>
#pragma GCC visibility pop

To @hsivonen's point—the Rust code in Firefox needs to provide a C API to the C/C++ code for FFI purposes, and #[no_mangle] is a part of that. Separating the concerns of name mangling and symbol visibility sounds useful so that in situations like this where the Rust compiler is only producing an intermediate static library that gets linked into the final binary, developers can fine-tune the behavior they need. Additionally, providing solutions that allow overriding behavior is useful because otherwise it's one more thing you need to audit in your dependency graph, and crate authors can't all be expected to understand the nuance here.

[bjorn3]

I think this is basically what #[rustc_std_internal_symbol] does.