`fn_cast!` macro

[Darksonn]

Since Rust 1.76 we document that it's valid to transmute function pointers from one signature to another as long as their signatures are ABI-compatible. However, we have since learned that these rules may be too broad and allow some transmutes that it is undesirable to permit. Specifically, transmutes that change the pointee type or constness of a pointer argument are considered ABI-compatible, but they are rejected by the CFI sanitizer as incompatible. See rust-lang/unsafe-code-guidelines#489 for additional details and #128728 for a concrete issue.

This issue tracks a proposed solution to the above: Introduce a new macro called fn_cast! that allows you to change the signature of a function pointer. Under most circumstances, this is equivalent to simply transmuting the function pointer, but in some cases it will generate a new "trampoline" function that transmutes all arguments and calls the original function. This allows you to perform such function casts safely without paying the cost of a trampoline when it's not needed.

The argument to fn_cast!() must be an expression that evaluates to a function item or a non-capturing closure. This ensures that the compiler knows which function is being called at monomorphization time.

As a sketch, you can implement a simple version of the macro like this:

macro_rules! fn_cast {
    ($f:expr) => {
        #[cfg(not(any(sanitize = "cfi", sanitize = "kcfi")))]
        {
            // we need $f coerced to a function pointer
            core::mem::transmute::<fn(_) -> _, _>($f)
        }
        
        #[cfg(any(sanitize = "cfi", sanitize = "kcfi"))]
        {
            |arg| {
                let arg = core::mem::transmute(arg);
                let ret = $f(arg);
                core::mem::transmute(ret)
            }
        }
    };
}

This implementation should get the point across, but it is incomplete for a few reasons:

• It assumes that the function takes one argument, but a real fn_cast! should be improved to work with functions of any arity.
• With CFI, it always generates a trampoline using a closure. However, if this was a compiler built-in, then it could modify the list of signatures allowed by the target function so that CFI does not reject the call. The trampoline would only be needed if the function is in a different compilation unit.
• With KCFI, we can't add signatures to the target function, but we still don't always need a trampoline. For example, changing fn(&T) to fn(*const T) is allowed because &T and *const T is treated the same by KCFI. The compiler could detect such cases and emit a transmute instead of a trampoline.

By adding this macro, it becomes feasible to make the following breaking change to the spec:

When you make a function call, then the caller and callee must agree on what the function signature is exactly. Otherwise:

• If the signatures are ABI-compatible, then it is EB (errornours behavior). That is, similiarly to integer overflow, sanitizers such as cfi, kcfi, or miri could trigger an error when it happens. But otherwise the call is allowed through by transmuting each argument.
• Otherwise, it is UB (undefined behavior).

Here, the change is that ABI-compatible calls are considered EB. However, even without the spec change the macro is useful because it would allow for a more efficient implementation of #139632 than what is possible today.

This proposal was originally made as a comment. I'm filing a new issue because T-lang requested that I do so during the RfL meeting 2025-05-07.

[RalfJung]

When you make a function call, then the caller and callee must agree on what the function signature is exactly.

So in such a world, the docs for the macro would say that this generates a new function? Because otherwise it seems like this list here has to account for the macro as well.

The macro needs to be unsafe of course, since function arguments are still being transmuted. We could have the macro ensure that the signatures are ABI-compatible -- but this can only be fully checked during monomorphization.

[Darksonn]

Well, yes it semantically creates a new function even if it has the same address. How exactly we word that is up to debate. I guess we might not want provenance for function pointers (?), so if fn_cast! returns a fn pointer with the same address, then we probably have to say that this function is valid to call with those two signatures.

[RalfJung]

I guess we might not want provenance for function pointers (?)

I mean, we could.^^ But yeah it's probably better if we avoid using provenance wherever possible.