It's hard for bots to handle pre-messages

[Hocuri]

With pre-messages, it's hard for bots to wait for the message to be fully downloaded and then process it.

Up until now, bots used get_next_msgs() to query the unprocessed messages, then set last_msg_id after processing a message to that they won't process it again.

But this will now also return messages that were not fully downloaded.

Some solutions discussed for this are:

• The bot just waits up to ~30s until the message is downloaded, by listening to MsgsChanged events until the download state is done, and doesn't process any messages until then. The MsgsChanged event will contain the correct msg_id. We could easily add a new event for this in core, if that helps.
  • This approach doesn't require core changes, but is hacky and can block the whole bot for up to 30s.
  • If the sender side always sent the post-message first and the pre-message later (even though then the namings would be very weird), then this would lead to a lot less blocking on the bot side (only if the large message gets lost or gets reordered on the server due to its size)
• We add a new kind of message id that only gets assigned once the message is fully downloaded. We then use this in a new API similar to get_next_message_ids.
  • This might lead to message reordering, because core will first fetch all pre-messages and then all post-messages
• We add a table that contains the message ids of all messages that have been downloaded, but not processed yet. We then add an API to query this table, and an API to remove a message id from this table.
  • Something like this could as well be implemented outside of core
  • This might lead to message reordering, because core will first fetch all pre-messages and then all post-messages
• In core, if bot is true AND auto-download is "all", then we don't add a row into the msgs table until the post-message is received.
  • This means that we need to reason about multiple download logics in core, but it might be the nicest option from the bot perspective.

[missytake]

The whole concept of pre- and post-messages should not be exposed to bot developers in my opinion. The state of library documentation is bad enough as it is. I think if we want people who are not part of the core team to develop bots, anything but option 4 will be insufficient.

1. e.g. in an event hook collected by https://github.com/chatmail/core/blob/main/deltachat-rpc-client/src/deltachat_rpc_client/events.py#L269, how can I even listen for new events? Will the events which don't have the expected msg_id be handled by other hooks, or simply thrown away? Where can I store the associated message state until the message is fully downloaded?
   I guess this would be kinda okay if the fully downloaded message would trigger an EventType.INCOMING_MSG as well. Then I can ignore INCOMING_MSG events which are not fully downloaded yet, and just trigger my function on the INCOMING_MSG which completed the download. But it is still an unergonomic way that would need to be documented, and lots of things need to be documented already.

2+3. I can't judge this, but I assume those are slightly different tradeoffs in complexity in core vs. client-side handling?

I'm slightly annoyed that bot development gets forgotten when introducing the recent big core changes. I just spent weeks porting all of my bots to 2.43.0, just to realize now that I should have stayed on an earlier versions (and now I can't downgrade). But I know that the current changes are already overwhelmingly big without having to think about bot development as well...

[r10s]

yip, ideally the detailed concept of pre/post messages should not be exposed to UI or bots. that way we can do changes to the internals without worrying about ppl using core.

what about the following if is_bot is set:

• as bots usually do not have traffic issues as mobiles, let's ignore the whole configurable auto_download setting for bots and assume auto-download = all for bots- this is even the default for UI, so that should be fine. sure, one can think of usecases where that may be helpful, but they are not supported then. things are complex enough

• then, ignore all pre messages if is_bot is set, before they would be added to the database. pre-messages need to be downloaded before they can be ignored, but that's fine - they're small and, again, usually also UI downloads both ¹. also, core anyways need to deal with the case a post-message arrives without a pre-message.

these should be very simple changes. it is crucial for future core supporting bots that there is as few extra code or api as possible. we have the same in other rarely used code paths - doing too special things will just be forgotten to maintain and will stop working. it is important to support these usecases, and to do what is needed, but we should avoid the incentive to do too special things now

Footnotes

1. there are ideas around to send post-message before pre-message, but that is an implementation detail ↩