Gmail: self-sent mail is silently removed from INBOX — relocate collapses the INBOX/Sent label pair because All Mail is skipped

[CatGoesMeow]

Summary

On Gmail accounts, any message carrying both the INBOX and Sent labels (i.e. mail you send to yourself, or a reply you send in a thread you also received) disappears from the MailFlow inbox after [Gmail]/Sent Mail syncs. The row is not lost — it is rewritten in place so folder = '[Gmail]/Sent Mail' — but the inbox view loses it permanently.

The effect is progressive: the inbox count drifts further below the server's on every sync cycle. On my account INBOX went from 114 → 102 rows over ~24h while the server reported 154.

• Version: 3.3.0 (ghcr.io/maathimself/mailflow-backend:3.3.0, ghcr.io/maathimself/mailflow-frontend:3.3.0), upstream images, unmodified
• Provider: Gmail over IMAP (app password)

Root cause — two Gmail-specific behaviors interacting

Gmail is a label store, so the same Message-ID legitimately exists in INBOX and [Gmail]/Sent Mail at once. RELOCATE_MESSAGE_SQL (backend/src/services/imapManager.js:879) treats "same Message-ID, different folder" as a move and rewrites the existing row:

UPDATE messages SET folder = $1::text, uid = $2::bigint, …
 WHERE account_id = $3::uuid
   AND message_id = $4::text
   AND (folder != $1::text OR uid != $2::bigint)
   AND 1 = (SELECT COUNT(*) FROM messages WHERE account_id = $3::uuid AND message_id = $4::text)
   AND COALESCE((SELECT special_use FROM folders WHERE account_id = $3::uuid AND path = $1::text), '')
       NOT IN ('\All', '\Important')

Two guards are meant to prevent exactly this, and neither covers the INBOX/Sent pair:

1. The NOT IN ('\All','\Important') special-use exclusion. [Gmail]/Sent Mail is \Sent — not excluded. So a sync of Sent Mail is allowed to relocate an INBOX row onto itself.

2. The COUNT = 1 guard, whose comment at imapManager.js:2696 reads:

   The COUNT=1 guard prevents incorrectly merging Gmail's virtual-folder copies (same message_id in INBOX and [Gmail]/All Mail simultaneously).

   That reasoning holds only if the sibling row actually exists — but PROVIDERS.google.skipFolderPatterns includes 'all mail', so All Mail is never backfilled and the sibling is never created. COUNT stays 1, and the relocate fires.

So the guard that was supposed to catch Gmail's multi-label case is disarmed by the other Gmail-specific optimization. Backfill order (backfillAllFolders runs INBOX first, then the remaining folders) decides the winner: Sent Mail syncs after INBOX, so Sent wins and the inbox copy is destroyed.

Both call sites are affected — the sync path (imapManager.js:2703) and the backfill path (imapManager.js:3372).

The plugin relocateExemptFolders collect-hook does protect label folders correctly (my GTD state folders keep their sibling rows — 21 Message-IDs survive in >1 folder, all of them GTD's), which confirms the sibling-row model is the right one; \Sent just never reaches it.

Reproduction

1. Add a Gmail account over IMAP.
2. From that account, send a mail to itself. In Gmail it now shows in both Inbox and Sent.
3. Let the initial backfill run to completion (Backfill complete for …/INBOX, then …/[Gmail]/Sent Mail).
4. Open the MailFlow inbox — the message is absent. It is present under Sent Mail.

-- one row, filed under Sent, none in INBOX
SELECT folder, uid, date, from_email, subject
  FROM messages
 WHERE account_id = '<gmail account>' AND message_id = '<the self-sent message id>';

Observed on my install

• 18 messages from Dec 2024, all self-sent, all present in the DB — every one filed under [Gmail]/Sent Mail, none in INBOX. Oldest INBOX row is 2025-06-04, so the whole 2024 stretch of self-sent mail is invisible in the inbox.
• folders.total_count for INBOX = 154 (server), messages rows with folder='INBOX' = 102.
• [Gmail]/Sent Mail: 4348 rows = exactly the server total — the deficit moved there.
• 225 Sent-Mail rows are addressed to the account's own address, i.e. the pool this keeps drawing from.
• The INBOX backfill itself is healthy and reports success — Backfill w***@gmail.com: 56 missing of 153 (98 already in DB) → Backfill complete. The rows are inserted correctly and then relocated away afterwards, so nothing in the logs indicates a problem.

Impact

• Silent data-visibility loss on the primary view, on the most common provider.
• Self-conversations and sent replies in received threads vanish from the inbox.
• Self-correcting only by accident; /reindex re-inserts the INBOX rows and the next Sent Mail sync removes them again.
• No server-side damage — the relocate is a DB UPDATE and issues no IMAP command, so the Gmail labels are intact.

Suggested fixes

• Minimal: add '\Sent' to the special-use exclusion at imapManager.js:901. A message in Sent legitimately co-exists with an INBOX copy on every label-based provider; it is never the destination of a real move that should erase the source.
• More correct: make the relocate skip any provider whose profile declares label semantics (a labelStore: true flag on PROVIDERS.google) and insert a sibling row instead — the same model the relocateExemptFolders hook already implements for plugin label folders.
• Also worth fixing: the COUNT = 1 guard's comment documents an invariant (INBOX and All Mail rows coexist) that skipFolderPatterns: ['all mail'] makes false for Gmail. Either the comment or the guard should be brought back in line, otherwise the next folder pair to hit this will be just as invisible.

Happy to test a patch against a live 30k-message Gmail account.

[maathimself]

Fixed on main in 50d3cb7 (v3.4.0).

Relocate-by-Message-ID has been removed entirely. Two live UIDs may legitimately share a Message-ID, across folders or within one, so it was never safe evidence of a move — it also caused reading one message to mark a second as read. Folder membership is now identified by account + folder + UID on every provider, and explicit moves use confirmed source/destination UID mappings.

Regression coverage was added for Gmail label pairs (INBOX + [Gmail]/Sent Mail), cross-folder shared Message-IDs on generic IMAP, PurelyMail and iCloud, and two live UIDs sharing a Message-ID inside a single folder.