Signals: no supported way to keep shared state consistent with a database

[totally-not-ai]

Part of #138, and following #138 (comment).

When the parity gaps were filed, CollaborationMessagePersister was left out on the grounds that a plain write-through replaces it and the remaining edge cases are narrow enough to leave to applications. Leif's response is the reason to file it anyway:

It might be reasonable to put the responsibility on the application for now. But eventual clustering support will lead to a situation with many more subtle edge cases to check for so that it will no longer be reasonable to expect an application to get it right.

That reframes this. The question is not whether an application can paper over today's gaps — it can — but whether the same approach survives clustering. It does not, and the cost of finding that out later is a design that has to change rather than an implementation that has to be fixed.

Real use case

A comment thread on a document, persisted to a database so it survives a restart. Under Collaboration Kit this is a MessageManager with a CollaborationMessagePersister. Migrated to signals it becomes a SharedListSignal<Comment>, seeded from the repository when the shared state is created, and a submit handler that saves and then inserts.

Three things that protocol was doing become the application's problem.

1. The save and the insert are two operations with nothing joining them.

Comment saved = commentService.save(documentId, text);   // succeeds
messages.insertLast(saved);                              // never runs

If anything fails in between, the comment is in the database and on nobody's screen, and stays invisible until the shared state is discarded and reloaded. A shared signal cannot participate in a database transaction, so there is no way for the application to make this atomic. Collaboration Kit avoided the window by treating the database as the source and re-reading after every write.

2. Writes from outside the UI never arrive. A mail-to-comment gateway, a batch import, an admin tool: anything that writes a comment directly to the database is invisible to every user with the document open. Collaboration Kit's re-fetch after each submit picked those up as a side effect. With a write-through, the in-memory copy and the database diverge silently and stay diverged.

3. With clustering, initialization itself races. This is Leif's point in #145: node A resolves the document and seeds a shared signal from the database at the same moment node B commits a change to that document. Depending on ordering, the shared copy is stale from birth, and every later reader trusts it. No amount of care in the application fixes that, because the application cannot see the other node's transaction boundary.

What Collaboration Kit does

CollaborationMessagePersister is a protocol, not a save hook. MessageManager drives it:

• On first connection to a topic, fetch the history with a FetchQuery carrying a since timestamp.
• Cache the result in the topic, so later managers on the same topic do not re-query.
• On submit, write to the backend first, then re-fetch everything at or after the last known timestamp.
• De-duplicate the overlap, since the boundary message comes back again.
• Re-enter catch-up mode when a connection deactivates and reactivates.

It also validates the implementation rather than trusting it: FetchQuery throws if getSince() or getTopicId() was not called, and MessageManager fails with an explicit message if a re-fetch does not include the previous boundary message.

The protocol exists because getting this right is fiddly, and the validation exists because implementations got it wrong.

What signals have

Nothing. A shared signal is an in-memory value with no notion of a backing store, no initialization hook, and no way to take part in a database transaction. The guide's advice — seed on creation, write through on change — is the best available today and covers the common case.

Suggested direction

Not a port of CollaborationMessagePersister; its message-and-timestamp shape is specific to chat. What generalizes is the contract:

• Initialization — resolve shared state for an entity by loading from a store, once, with a defined ordering against concurrent writes to that store. This is the same operation Signals: no way to resolve a shared signal by topic id, and no expiration #145 needs for resolve-by-identifier and SharedListSignal: no emptiness condition for transactions (ifEmpty / ifNotEmpty parity) #149 calls exactly-once initialization when an entity becomes active. Whatever is built there should carry this.
• Write-through with a defined failure mode — a supported way to say "this change goes to the store first, and the shared state reflects what the store accepted". The application should not have to decide what happens when half of that succeeds.
• Reconciliation — a way to re-read the store and converge, for writes that did not come through the UI.

Worth deciding before clustering rather than after: whichever shape this takes, the single-node implementation has to be the degenerate case of the clustered one, otherwise applications written against the single-node behaviour break when clustering arrives. Related: vaadin/flow#23413.

[Legioth]

There's one more essential thing that becomes the application's problem if done manually: ordering between concurrently submitted changes.

If thread 1 inserts message A into the database and then adds the same message to a signal list, and thread 2 concurrently does the same with message B, then there's a possibility that the ordering in the database is AB whereas the ordering in the signal list is BA, or vice versa.

The solution is the same though: write to the database first and then read back the state from the database and use that data to populate the signal. In case of a concurrent update, one thread might then end up applying both changes to the signal whereas the other thread finds out that the change it committed was already applied to the signal so that it doesn't need to make any signal updates.