fs_attachment: migrating a large number of attachments to S3

[Lionel-cl]

Hello,

This is not yet a [BUG] nor an [IMP]...

Before proposing a possible improvement to fs_attachment, I would like to better understand the recommended process for migrating an Odoo database (in my case: Odoo Online) to an on-premise installation using S3 storage.

The database dump itself is not the issue, but rather the migration of attachments to S3.

When using _force_storage_to_object_storage(), I encounter two problems:

1. the method is quite slow and becomes blocking when there are many attachments.
   In my case (~17,000 attachments), migrating 1,000 attachments takes about 10 minutes;

2. the creation of orphan attachments:
   when several attachments reference the same file in the filestore, migrating the first attachment may delete the source file at the end of the process.
   In a batch-processing scenario, this makes it impossible to migrate the other attachments (processed in another batch), even though the file is already present on S3.

How do you usually handle this kind of migration?

It seems to me that another process should probably be involved before or during the migration, but I would first like to understand the existing recommendations/best practices before proposing any change to the module.

thanks

[Lionel-cl]

What do you think about proceeding with:

• bulk transfer of the filestore using rclone (or equivalent)
• an option in fs_attachment to preserve the Odoo naming convention (based on the hash)
• a method to modify the store_fname in the database without moving the attachment, and to add fs_storage_code