Restructure ptr_metadata to minimal support

[CAD97]

Proposal

Problem statement

feature(ptr_metadata) is in feature limbo. Being able to split pointers into raw pointer and metadata unlocks a lot of cool abilities in library code, but as currently architected the metadata APIs expose more than is really comfortable to stabilize in the short term. Notably, Pointee is a very special lang-item, being automatically implemented for every type. Plus, it's unfortunately tied in identity to both "DynSized", extern type, and custom pointee metadata.

This API offers a way to expose a minimal feature(min_ptr_metadata) subset of feature(ptr_metadata) that unblocks the simple use cases of separating address from metadata, while remaining forward-compatible with any¹ design for exposing more knobs later on.

In short: this splits feature(ptr_metadata) into granular subfeatures such that min_ptr_metadata might be stabilizable this year.

Motivation, use-cases

My main motivating use case is custom reference-like types.

Consider something like a generational arena. When using an arena, instead of using something like rc::Weak<Object>, you use Handle<Object>, where Handle is some index; perhaps { ix: u32, salt: u32 }. This works well when your arena stores a single sized type; you Arena::<Object>::resolve(&self, Handle<Object>) -> &'_ Object and all is good.

This is simple enough to extend to storing arbitrary sized objects in the arena as well, so long as you use some kind of allocation strategy which can handle the mixed-layout allocations rather than a simple array storage. This also means that your index changes from an element index to a byte index, and you need some way to ensure that a handle is only used for the arena that it came from.

But what about unsized types? For example, in an actor-based game engine (as opposed to an ECS-based solution), you might have Handle<PlayerPawn> and Handle<EnemyPawn>, and want to be able to store a handle to either one as Handle<dyn Pawn>. Without feature(ptr_metadata), such a handle is forced to store some ptr::NonNull<dyn Pawn>, where what actually wants to be stored is { ix: u32, salt: u32, meta: ptr::Metadata<dyn Pawn> }.

Being able to separate pointer metadata from the actual pointer is a space-saving mechanism for such designs in cases where the storage is known to be pinned, as resolving a handle could be implemented to validate the index, salt, address, etc. match and then dereference the stored pointer with its metadata. When the storage is not pinned, however, allowing dyn-erased handles is just not possible, as there is no (stable) way (yet) to graft one pointer's metadata onto another's.

Solution sketches

Due to the specifics of this proposal, the proposed API is a slight evolution of the existing API, and heavily split into multiple features. For continuity, all of these smaller features should be considered implied by the ptr_metadata feature.

// mod core::ptr

#[unstable(feature = "ptr_metadata_trait")]
pub trait Pointee {
    type Metadata: MetadataOf<Self>;    
}

// compiler-generated
impl<T: Sized> Pointee for T {
    type Metadata = ();
}

// compiler-generated
impl<T: Sized> Pointee for [T] {
    type Metadata = usize;
}

// compiler-generated
impl<trait Trait> Pointee for dyn Trait {
    type Metadata = DynMetadata<dyn Trait>;
}

#[unstable(feature = "thin_ptr_metadata")]
pub trait Thin = Pointee<Metadata = ()>;

#[unstable(feature = "min_ptr_metadata")]
struct Metadata<T: ?Sized> {
    /*priv*/ raw: <T as Pointee>::Metadata,
}

impl<T: ?Sized>
Copy, Clone, Send, Sync, Debug, Ord, PartialOrd, Eq, PartialEq, Hash, Unpin
for Metadata<T>;
// but not StructuralEq!

#[unstable(feature = "ptr_metadata_access")]
impl Metadata<T: ?Sized> {
    pub const fn into_raw(self) -> <T as Pointee>::Metadata;
    pub const fn from_raw(<T as Pointee>::Metadata) -> Self;
}

#[unstable(feature = "min_ptr_metadata")]
pub const fn metadata<T: ?Sized>(*const T) -> Metadata<T>;

#[unstable(feature = "from_ptr_metadata")]
pub const fn from_raw_parts<T: ?Sized>(*const (), Metadata<T>) -> *const T;
#[unstable(feature = "from_ptr_metadata")]
pub const fn from_raw_parts_mut<T: ?Sized>(*mut (), Metadata<T>) -> *mut T;

impl<T: ?Sized> NonNull<T> {
    #[unstable(feature = "min_ptr_metadata")]
    pub const fn metadata(self) -> Metadata<T>;
    #[unstable(feature = "into_ptr_metadata")]
    pub const fn to_raw_parts(self) -> (NonNull<()>, Metadata<T>);
    #[unstable(feature = "from_ptr_metadata")]
    pub const fn from_raw_parts(NonNull<()>, Metadata<T>) -> Self;
    #[unstable(feature = "with_ptr_metadata")]
    pub const fn with_metadata<U: ?Sized>(self, Metadata<U>) -> NonNull<U>;
    #[unstable(feature = "with_ptr_metadata")] // currently feature(set_ptr_value)
    pub const fn with_metadata_of<U: ?Sized>(self, *const U) -> NonNull<U>;
}

impl<T: ?Sized> *const T {
    #[unstable(feature = "min_ptr_metadata")]
    pub const fn metadata(self) -> Metadata<T>;
    #[unstable(feature = "into_ptr_metadata")]
    pub const fn to_raw_parts(self) -> (*const (), Metadata<T>);
    #[unstable(feature = "from_ptr_metadata")]
    pub const fn from_raw_parts(*const (), Metadata<T>) -> Self;
    #[unstable(feature = "with_ptr_metadata")]
    pub const fn with_metadata<U: ?Sized>(self, Metadata<U>) -> *const U;
    #[unstable(feature = "with_ptr_metadata")] // currently feature(set_ptr_value)
    pub const fn with_metadata_of<U: ?Sized>(self, *const U) -> *const U;
}

impl<T: ?Sized> *mut T {
    #[unstable(feature = "min_ptr_metadata")]
    pub const fn metadata(self) -> Metadata<T>;
    #[unstable(feature = "into_ptr_metadata")]
    pub const fn to_raw_parts(self) -> (*mut (), Metadata<T>);
    #[unstable(feature = "from_ptr_metadata")]
    pub const fn from_raw_parts(*mut (), Metadata<T>) -> Self;
    #[unstable(feature = "with_ptr_metadata")]
    pub const fn with_metadata<U: ?Sized>(self, Metadata<U>) -> *mut U;
    #[unstable(feature = "with_ptr_metadata")] // currently feature(set_ptr_value)
    pub const fn with_metadata_of<U: ?Sized>(self, *const U) -> *mut U;
}

#[unstable(/* indeterminate */)]
unsafe trait MetadataOf<T: ?Sized>: Send + Sync + Copy + Debug + Ord + Hash + Unpin {
    // indeterminate
}

unsafe impl<T: Sized> MetadataOf<T> for () {}
unsafe impl<T: Sized> MetadataOf<[T]> for usize {}

#[unstable(feature = "dyn_ptr_metadata")]
pub struct DynMetadata<T: ?Sized> {
    /*priv*/ raw: WellFormed<__CompilerGeneratedVtable<T>>,
}

impl<T>
Copy, Clone, Send, Sync, Debug, Ord, PartialOrd, Eq, PartialEq, Hash, Unpin
for DynMetadata<T>;
// but not StructuralEq!

// compiler-generated, if statically restricting to just dyn Trait objects
unsafe impl<trait Trait> MetadataOf<dyn Trait> for DynMetadata<T> {}

#[unstable(feature = "ptr_metadata_layout")]
impl<T: ?Sized> DynMetadata<T> {
    pub fn size(self) -> usize;
    pub fn align(self) -> usize;
    pub fn layout(self) -> core::alloc::Layout;
}

The key change is that rather than expose concrete metadata types, we only expose a single uniform opaque ptr::Metadata<T> type. This can have additional other benefit -- one key one I'm interested in is that ptr::Metadata<T> can CoerceUnsized, allowing pointer-like types wrapping ptr::Metadata to get unsizing coercions as if they were holding an actual pointer.

The other key removal of surface area of ptr_metadata in min_ptr_metadata is the moving of from_ptr_metadata into a separate feature. This means that the choice of using *mut () or *mut u8 for the untyped pointer is deferred to a separate feature. The alternative is with_metadata[_of].

Other notes on API design choices

• It is valid to (as) .cast() between pointers which have the same <T as Pointee>::Metadata type.
  • This, along with the existing (*const T, usize) -> *const [T] functions, is why <[T] as Pointee>::Metadata is usize rather than a SliceMetadata<T> type.
• core::ptr::WellFormed is a raw pointer that is guaranteed non-null and aligned, per Commit to safety rules for dyn trait upcasting rust#101336
• The metadata size function probably wants to wait for the public exposing of a ValidSize type which is bound to 0..=isize::MAX.
• The metadata align function might want to wait for the public exposing of a ValidAlign type which is restricted to powers of two.
• The metadata size function is fallible because it's possible to safely construct *const [T] describing too-large objects.
• The metadata align function is infallible because I don't see any utility in dynamic alignment requirements, but perhaps it should be fallible for uniformity?
• The potential ability to add Pointee as a default bound and make extern type be !Pointee I find intriguing. Doing so might be able to unblock further experimentation with extern type, as the current direction seems to be forbidding the use of extern type in generics which do not opt-in to supporting extern types, if opting in is allowed at all. In such a case, perhaps the trait really should evolve to be DynSized.

---

Links and related work

• Tracking Issue for pointer metadata APIs rust#81513
• Implement RFC 2580: Pointer metadata & VTable rust#81172
• RFC: Pointer metadata & VTable rfcs#2580
• Implement pointee metadata unsizing via a TypedMetadata<T> container rust#97052

What happens now?

This issue is part of the libs-api team API change proposal process. Once this issue is filed the libs-api team will review open proposals in its weekly meeting. You should receive feedback within a week or two.

Footnotes

1. Asserted without proof. ↩

[thomcc]

The original RFC was dual t-lang and t-libs-api, so this feels like it might need lang decision too... Although perhaps it's enough to have that on some FCP later on.

[CAD97]

The intent of the ACP as-written is to introduce no new language semantics, and be implementable solely in terms of today's feature(ptr_metadata) APIs/features. In that light, I think this could be considered just t-libs-api, but yes, it makes sense to include t-lang at some point.

As far as I can tell, the point in the API as proposed here that would require T-lang sign-off is providing layout from the fully generic ptr::Metadata<T>. Previously, T-lang did commit to [size|align}_of_val_raw working for once-valid-but-dropped pointers.

I'm striking the generic parts of my feature("ptr_metadata_layout") for now because of this, and because this caused me to recall that we (of course) want to support thin pointers to C++-style vtable-prefixed pointees, which means getting layout information from just the metadata is not possible in general.

[WaffleLapkin]

The other key removal of surface area of ptr_metadata in min_ptr_metadata is the moving of from_ptr_metadata into a separate feature. This means that the choice of using *mut () or *mut u8 for the untyped pointer is deferred to a separate feature. The alternative is with_metadata[_of].

I want to note that while this may be a nice distinction from the stabilization pov, since it's easier to reason about smaller APIs, we still need from_ptr_metadata to make almost any use of min_ptr_metadata. Since ptr::Metadata<T> is opaque (at least in this proposal; at least by-default/at the start), to actually use it, you need to make a pointer/reference, so you need from_ptr_metadata.