The state of file uploads and possible solutions

[wout]

Currently there isn't a canonical way to do file uploads in Lucky. Stephen did a good video on LuckyCasts, and the Handling Files section of the guides describes more or less the same approach. But those are simply storing the file id (e.g. name/path) on a record and don't provide all the features that Shrine has to offer, like metadata extraction etc.

Unfortunately Shrine.cr seems to be abandoned. Håkan Nylén opened a PR more than three years ago that never got merged. Two weeks ago I opened a PR with a styling updates and a direct ping to Igor asking what his plans are, but it's been nothing but crickets so far.

I see two routes we can consider for a comprehensive file upload solution in Lucky/Avram.

Option 1: Lucky-native uploads

Basically this would mean building an implementation tailored to Lucky, just like Marten and Rails approach it. That would mean taking out the best parts of Shrine.cr and building something lightweight with just enough features.
The good: ligthweight, Lucky-style uploads that are easy to maintain
The bad: some users may still want specific features that we may not choose to implement (yet)

Option 2: Build on top of Shrine.cr

This approach would mean creating a fork of Shrine.cr, completing the work on Shrine::Attachment and cleaning up the codebase (it's full of .not_nil! calls for example). Then it probably also needs some work to make the Shrine::Attachment subclasses play nice with Avram.
The good: Shrine.cr already offers a lot of features, and the work benefits the crystal community as a whole
The bad: It's a fork, which means making the community aware, and also more maintenance work (for me)

---

I'm down for either route, or an alternative third option, but I would like your input first because maybe I'm overlooking something. Do you have any suggestions? Perhaps a wishlist?

In any case, this is something that I'll need to tackle rather sooner than later because uploads are a fundamental part of the project I'm working on right now.

[jwoertink]

I do like option 1. I wonder if this is something we'd need to consider when it comes to integration with a client-side library like https://uppy.io/ ? I'm not really great with the whole file upload stuff, so I'm not sure what's more common here... But I do think having something a bit more native in Lucky would be better in the long run which avoids the case of a 3rd party lib becoming stale and us not having access to fix it.

[wout]

I also lean towards option 1, less legacy code to deal with and easier to future-proof.

I've never used Uppy in a project but I don't think anything special is needed. Uppy just sends the file to an action, and then you'd have to take it from there by getting the file(s) into an operation for processing.

The idea is to make the implementation Shrine-compatible in terms of storage. So a single database column per uploader, called [NAME]_data (for example image_data) which is a JSON containing the file id (path) and any additional metadata. That way, users can migrate from and to Shrine without having to change the data.

Shrine works with plugins for metadata extraction (dimensions, mime type, etc), but I'd just build those in as features that can be enabled if necessary, via an annotation for example.

In terms of storage, we'd have local file storage, S3/MinIO/Hetzner, and memory for testing. And of course also support for cached and final storage to avoid re-uploading when validations fail on other fields.

It's a lot but I've worked enough with file uploads and Shrine in Ruby to get this done. With some of that delicious Lucky sauce of course. 😄 I'll work on this as I build my app so that I can test out different approaches and iron out the kinks.

[wout]

I've been working on the first sketch of the API and this is the idea.

Configuration

Avram.configure do |settings|
  settings.upload.cache = Avram::Upload::FileSystem.new("uploads", prefix: "cache")
  settings.upload.store = Avram::Upload::S3.new(bucket: "bucket_name", prefix: "prefix")
end

Migration
Initially I wanted to do it like Shrine (image_data) but without the _data suffix it actually works out better.

create table_for(Profile) do
  # ...
  add image : JSON::Any
end

Model
Nothing special, just as you'd expect it to be in a Lucky app.

class Profile < BaseModel
  table do
    column image : ImageAttachment
  end
end

Attachment
Here is where all the heavy lifting happens.

struct ImageAttachment < Avram::Attachment 
  # using the default Avram::FileExtractor
  extract :mime_type
  # or using another built-in extractor
  extract :mime_type, Avram::ContentTypeExtractor
  
  # using the default Avram::IdentifyExtractor depending on imagemagick
  extract :dimensions
  # or with a custom extractor 
  extract :dimensions, MyDimensionsExtractor

  # add arbitary metadata
  metadata signature : String, -> { Digest::SHA256.hexdigest(io.gets_to_end) }

  # create variants (could be image sizes, types, video poster, pdf poster, etc..)
  variant :thumb, { width: 300, height: 200, method: :crop, gravity: {50,50}, type: :webp }
  variant :large, { width: 1280, height: 720, method: :fill }
end

# accessing metadata
profile.image.mime_type # => "image/png"
profile.image.width # => 1280
profile.image.metadata.signature # => "e3b0c44298..."

# accessing the image url
profile.image.url
profile.image.url(:thumb)

There will probably also be a task to generate attachments. Something like lucky gen.attachment Image for example.

The whole variants story will have to be a separate project because it would (ideally) require backgrounding. That's something I need to test out in practice.

What do you think?

[jwoertink]

I like the API. My only (very slight) concern / question might be, does it make sense to tightly couple file uploads with Avram? Like, if someone wanted to use a different ORM with Lucky, would they also have to use some other uploading mechanism? Or what if someone wanted uploads without an ORM?

Right now we have https://github.com/luckyframework/lucky/blob/main/src/lucky/uploaded_file.cr for managing your uploaded file, and then https://github.com/luckyframework/avram/blob/main/src/avram/uploadable.cr for creating the interface to the ORM.

[wout]

Very good point! I imagined it to be tightly integrated with Lucky/Avram so I didn't really consider different ORMs. But I can see the benefit of a clear boundary because it’s a question that will come for sure. I'm going to try out a few things and see where I get with a clearer separation.

So, do you see this more as a part of Lucky then? Or as a separate shard altogether?

[jwoertink]

If we can make it as a part of Lucky directly, then we can add some extensions in to Avram like we have in here https://github.com/luckyframework/avram/tree/main/src/lucky then that way if someone wanted to use it with say Granite, then they'd just have to make their shim to bridge what's missing.

[wout]

I'm going to continue documenting the intended approach here. This is something I'll implement while building file uploads in Fluck, so it will take a few weeks before I have a concrete proposal. Here's where I'm at.

Model

Declaring a file attachment column is done via a new macro:

class User < BaseModel
  table do
    column email : String
    column encrypted_password : String

    attach profile : ImageAttachment
  end
end

Note

I chose attach here because it has the same length as column, but perhaps a noun is more fitting (e.g. attachment)?

In the background this will declare a profile_data column with type JSON::Serializable. It will also generate convenience methods on the model to access the image URL, optionally with any possible variant defined in the ImageAttachment. So you'd be able to:

user.profile.url
# => "https://..."
user.profile.url(:small)
# => "https://..."

This will work similarly to how it's done in AvramEncrypted::Model.

Attachment

Attachments declare functionality of the uploaded file and they live under src/attachments. One Lucky::Attachment can be used for multiple models. They define variants, metadata extraction such as size, mime type, etc., and upload destinations. Here's what they could look like:

struct ImageAttachment < Lucky::Attachment
  # shorthand
  storage :main
  # is the same as
  storage cache: :main, store: :main
  
  # using the default Lucky::FileExtractor
  extract :mime_type
  # or using another built-in extractor
  extract :mime_type, Lucky::ContentTypeExtractor
  
  # using the default Lucky::IdentifyExtractor depending on imagemagick
  extract :dimensions
  # or with a custom extractor 
  extract :dimensions, MyDimensionsExtractor

  # add arbitary metadata
  metadata signature : String, -> { Digest::SHA256.hexdigest(io.gets_to_end) }

  # create variants (could be image sizes, types, video poster, pdf poster, etc..)
  variant :thumb, { width: 300, height: 200, method: :crop, gravity: {50,50}, type: :webp }
  variant :large, { width: 1280, height: 720, method: :fill }
end

Configuration

Lucky::Attachment.configure do |settings|
  settings.stores = {
   :main => {
      cache: Lucky::Attachment::FileSystem.new("uploads", prefix: "cache"),
      store: Lucky::Attachment::S3.new(bucket: "bucket_name", prefix: "prefix")
    }
  }
end

[wout]

I'm working on this here now and there's a solid base already: main...wout:lucky:add-file-uploads-with-configurable-storage

At the end of this weekend I'll be able to create a first draft PR. There will also be some additions to Avram, but the majority of the code lives in the Lucky repo.

There's still quite some work to be done though. Like the S3 (and compatible) storage, variants, different mime type extraction strategies, and dimensions extraction strategies. Those are the ones I still want to add to at least create feature parity with the Shrine.cr shard. I've tried to stay as close to the Shrine data structure and way of working as possible, so users converting over their Lucky apps won't have too much trouble (and because I like Shrine.rb a lot).

I also would like to add the option to promote attachments and process metadata async in a background job, but that would also require some kind of adapter structure for background jobs in Lucky, maybe a bit like the Carbon::DeliverLaterStrategy. But that's for later.