Skip to content

MessageData is lossy for multimodal UserMessage content #12

Description

@jasperblues

MessageData (embabel-chat-store/src/main/kotlin/com/embabel/chat/store/model/StoredMessage.kt) stores message content as a single content: String. Round-tripping a multimodal UserMessage through the store silently discards every non-text part.

The loss happens at both ends:

  • MessageData.from(message) captures only message.content. For a BaseMessage that is textContent — the concatenation of TextParts only, with ImagePart, DocumentPart and MediaPart dropped.
  • MessageData.toMessage() rebuilds UserMessage(content = content, ...), whose convenience constructor wraps the string into listOf(TextPart(content)).

So a rehydrated conversation is text-only even when the original message had attachments. In embabel-agent, BaseMessage explicitly models this content (parts: List<ContentPart>, with imageParts, documentParts, mediaParts, and isMultimodal accessors), so the capability exists upstream and is lost purely at the persistence boundary.

Impact: an application that reloads a conversation and re-sends it cannot rely on the store for attachments. The workaround is to keep the original in-memory message alongside the persisted one and substitute it back before invoking the LLM — which defeats the point of having a conversation store, and only works while the original is still in memory.

Question: is multimodal persistence intended to be in scope for the chat store? If so, could MessageData preserve the parts — either inline or as durable references (URI + mime type + size) with the bytes in a separate store?

Note the same class of gap exists on the assistant side: AssistantMessage carries assets: List<Asset> and implements AssetView, but MessageData has no assets field either. Related: embabel/embabel-agent#1821.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions