[WIP] der: remove lifetime from BitStringRef [BREAKING]

[tarcieri]

Following the pattern of #1921 and what #1998 did for OctetStringRef, this removes the lifetime from the struct, instead changing BitStringRef to a proper reference type to be used as &BitStringRef, which can implement the Borrow and ToOwned patterns needed to work with Cow.

To make this work, this uses a hack described in #2298 where we abuse a fat pointer, namely a ZST slice [UnsafeCell<()>], to carry the pointer to the *const u8 buffer that backs the BitStringRef, and carries the bit length along as the length of the ZST slice.

To get the original byte length back we can div_ceil(8) and then reconstruct the original [u8].

[tarcieri]

This one, it seems, fails under stacked borrows:

https://github.com/RustCrypto/formats/actions/runs/24847701760/job/72739437052#step:5:78

error: Undefined Behavior: trying to retag from <177556> for SharedReadOnly permission at alloc64685[0x0], but that tag does not exist in the borrow stack for this location
  --> der/src/asn1/bit_string2.rs:88:13
   |
88 |             slice::from_raw_parts(self.inner.as_ptr().cast(), byte_length(self.inner.len()))
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this error occurs as part of retag at alloc64685[0x0..0x3]
   |
   = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
   = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <177556> would have been created here, but this is a zero-size retag ([0x0..0x0]) so the tag in question does not exist anywhere

@RalfJung I don't suppose you know a way to pull off this particular hack that would actually be compatible with stacked borrows? (trying to make a bitstring-like reference type that can be used with Borrow and ToOwned patterns so it works with Cow) Apparently stacked borrows doesn't like me using a ZST slice like [UnsafeCell<()>] so I can store the bit length in a fat pointer.

[tarcieri]

This issue and comment seem relevant: rust-lang/unsafe-code-guidelines#256 (comment)

[RalfJung]

I assume the thread you started on Zulip is the same question? Thanks for moving it there, that means other people can help answer when I am swamped (which is my default state these days ;).

[tarcieri]

This fails under stacked borrows, but passes with 72211d8 which enables -Zmiri-tree-borrows.

Given this is for a data type used to represent untrusted data that lives at the network edge, I'm disinclined to merge something with soundness that appears somewhat unclear:

• Stacked Borrows: raw pointer usable only for T too strict? rust-lang/unsafe-code-guidelines#134
• Storing an object as &Header, but reading the data past the end of the header rust-lang/unsafe-code-guidelines#256

It seems stacked borrows and tree borrows disagree about how to handle this scenario and it's unclear to me which one is "right": is stacked borrows being too strict, or is this an edge case in tree borrows that isn't being handled correctly?

That's further compounded by it being a breaking change.

So, I'm going to go ahead and close this, but perhaps we can reconsider something like this in the next breaking release of der when perhaps there will be some progress on soundness in the intervening time.

[RalfJung]

It seems stacked borrows and tree borrows disagree about how to handle this scenario and it's unclear to me which one is "right": is stacked borrows being too strict, or is this an edge case in tree borrows that isn't being handled correctly?

This is a grey area of the language where no decision has been made yet. If you want to write code that you can be sure will work with all future versions of Rust, then you can't do tricks like this. That's why Stacked Borrows rejects the code.

If you want to write code that should work with today's Rust, but that is not covered by Rust's stability guarantees, then you can do tricks like this. That's why Tree Borrows accepts the code. But if you do then be aware that while the code should not be miscompiled by today's rustc, it might well be miscompiled in the future when we narrow down the spec. There's a reason Tree Borrows is not the default yet.

We will likely want to allow the &Header pattern in some cases. If you want to help make progress on this, I think a proposal which documents that repr(C) structs and arrays/slices do not have subobject provenance would make a lot of sense.

[tarcieri]

I think grey areas are something we probably want to avoid. Even if it so happens to work today I can imagine we would still have to deal with people asking us why it fails under Miri.

Hopefully by the time we're ready to ship the next breaking release of this crate in a year or two there will have been some developments in Rust the language like proper custom DSTs or even just some decision being made about how this is specified so we know there won't be future miscompilation.

[RalfJung]

Yeah, hopefully by then someone will step up to drive such developments forwards.