resumable uploads for ActivityPub

[reiver]

ActivityPub-based applications often allow users to upload audio, images, videos, and other file types, though most currently rely on non-ActivityPub APIs for this functionality.

These files can be large — for example, a video may be several gigabytes in size. Uploads of this scale are prone to interruption before completion. When that happens, it is important to resume the upload from the point of failure rather than restarting from the beginning.

An ActivityPub extension for media uploads exists:

https://www.w3.org/wiki/SocialCG/ActivityPub/MediaUpload

However, neither ActivityPub nor this extension currently defines a protocol for resumable uploads.

Because resumable uploads are already a practical requirement in existing ActivityPub applications, the ActivityPub protocol should support them.

---

[1] [2] [3]

[mariusor]

Not entirely related to the point @reiver is trying to make, so apologies for piggybacking on this ticket.

If changes are still up for consideration, I'd like to propose a slight simplification to the MediaUpload extension as it's presented in the above SocialCG link.

Instead of pushing both binary content and the "shell" ActivityPub Create activity in which to slot the uploaded media, the server could handle strictly the media upload, and return an incomplete ActivityPub Object (a "shell" as it was called) containing the properties that have been inferred from the uploaded media:

• Type: Audio, Video, Image, etc.
• MediaType: audio/..., video/..., image/..., etc.
• URL: containing the direct link to the uploaded media.
• AttributedTo: inferred from the authorization the upload was effected under, if it refers to a valid user
• etc

Then the client the user used for the upload can append the additional elements the user specifies (Summary, Name, ID, etc) and use a Create activity to actually create the ActivityPub object.

Additionally this allows for servers that operate fancier upload mechanisms (eg, creating different web-optimized versions for the data, or, as @reiver asked, a server that can support partial uploads) but still represent them as valid ActivityPub objects.

To me, this is a better API as it separates the concerns between the actual media upload and the ActivityPub processing, and in my case, the reason why I like this architecture better is that it allows for the mediaUpload to be a separate micro-service, independent of the ActivityPub server.

[evanp]

@mariusor I made a new ticket for your idea. In the future, please don't piggy-back on other issues; it helps us track the work if you concentrate on a single issue.

[evanp]

I think there's a new process with this draft RFC:

https://www.ietf.org/archive/id/draft-ietf-httpbis-resumable-upload-01.html

We can't refer to a draft RFC from ActivityPub until it's complete. However, this might be something we could track. Or we could just define the same process in the Media Upload mechanism.

It feels like we don't yet have any media upload, so I'd kind of like to get the basics done for the next version, and then optimizations like resumable uploads for future versions? But that might be a luxury of time we don't have.

[TallTed]

We can't refer to a draft RFC from ActivityPub until it's complete

In general, W3C specifications can refer to external documents that are at the same or the next-previous level of maturity.

We can certainly refer from a W3C Editor's Draft to a Draft RFC. I believe we could refer from a W3C Candidate Recommendation (CR) to a Draft RFC, but I'm not certain about RFC maturity levels.