fix: do not sort received messages below the last seen one

[link2xt]

Fixes #8030

• test: remove test_old_message_5: see commit message, the test seems to check for the incorrect behavior

• test: do not rely on loading newest chat in load_imf_email(): change helper function to avoid loading the wrong message if there is already a message with a newer timestamp

• test: use load_imf_email() more: some test refactoring, since we have load_imf_email, can as well use it instead of copypasting

• fix: always sort "Messages are end-to-end encrypted" notice to the beginning: see commit message, this ensures nothing can be added above this message for new chats (existing chats have non-zero timestamp so you can technically backdate something before "Messages are end-to-end encrypted", but this is not really a problem)

• fix: never sort the message before chat joining timestamp: avoid adding "Member added" message before securejoin system messages

• fix: do not sort received messages below the last seen one: the last change, see the code and comments in the code. I have reviewed "golden tests" breakage, this is where commits above come from, the rest of the breakage looks expected. I also left one comment myself on one of the tests.

• There is a problem with "Messages are end-to-end encrypted." message. It has current timestamp, but incoming message from contact requests is from the past, so contact request chats start with the incoming message followed by "Messages are end-to-end encrypted.". Need to set the timestamp for the info message to the timestamp of incoming message or fix it the other way. (tests::verified_chats::test_outgoing_encrypted_msg)

• Another problem is with "Member added" message in sync::tests::test_unpromoted_group_qr_sync, it did not end up at the end:

    ========== Chats of fiona: ==========
    Group#Chat#6002: the chat [2 member(s)]
    --------------------------------------------------------------------------------
    Msg#6003: info (Contact#Contact#Info): Messages are end-to-end encrypted. [NOTICED][INFO]
    Msg#6004: info (Contact#Contact#Info): alice@example.org invited you to join this group.

    Waiting for the device of alice@example.org to reply… [NOTICED][INFO]
    Msg#6008🔒:  (Contact#Contact#6001): Member Me added by alice@example.org. [FRESH][INFO]
    Msg#6006: info (Contact#Contact#Info): alice@example.org replied, waiting for being added to the group… [NOTICED][INFO]
    --------------------------------------------------------------------------------

[link2xt]

There is some reordering of Alice's own messages in chat::chat_tests::test_sync_broadcast_and_send_message which results in one device showing as if Alice has sent a message to own channel before adding Bob while in fact it was after Alice added Bob. This seems to work as expected, if Alice backdates some message in the channel then it is not possible for the second device to know that it was not actually sent before Bob was added.

Essentially the problem boils down to "smeared timestamps" that cause these problems to golden tests that are not real problems. In practice devices may simply have slightly desynchronized clocks. Maybe we should not add too many golden tests as they are much easier to write than tell whether the breakage is important or not when they break.

[link2xt]

Two things still needed:

• Adding a test for the issue Message reordering with multi-relay #8030, something demonstrating the problem.
• If we think that backdating the message too far in the past is bad, can add logic to sort the message to the end if it appears to be from more than a week ago. But it will again break some golden tests in places where dates are hardcoded in test messages, they are definitely more than a week old. I am not sure it is an issue if we mark the message as unread and scroll to it (desktop already does this). Someone maliciously backdating their message is not something to protect against IMO, there are worse things to happen if you are chatting with adversaries.

[iequidoo]

Do other messengers, when a chat is open, insert new delayed messages not at the bottom, but somewhere, maybe even outside the screen?

Have you considered an approach that changes the current behavior less (particularly, for a chat being open): when an MDN for an InFresh message is received, recalculate its sort timestamp as if received was false?

[iequidoo]

So messages may still be ordered differently on devices if some chat members have clocks in the future and devices e.g. come online at different time.

[link2xt]

Yes, this prevents the message from someone with such clock set into the future from sticking at the end of the chat, even after outgoing messages.

[iequidoo]

Yes, this ceiling is good, i just want to say that receiver-side reordering (and different order on user's devices) still may happen if sender's clock is in the future.

Also, sender- or server-side reordering could happen even before multi-relay, e.g. for messages sent while being offline or using a bad network, but this has never been considered as a problem, instead, messages inserted at the bottom always was a "feature".

We could guarantee the same message order on devices, but then we need to know the minimum "received" smeared time on devices...

[Hocuri]

Thanks for fixing! Let's hope that this improves things 🤞