Skip to content

feat: add Kafka key serialization/deserialization behaviors#256

Draft
danmalcolm wants to merge 1 commit intoBEagle1984:masterfrom
danmalcolm:feature/kafka-key-behaviors
Draft

feat: add Kafka key serialization/deserialization behaviors#256
danmalcolm wants to merge 1 commit intoBEagle1984:masterfrom
danmalcolm:feature/kafka-key-behaviors

Conversation

@danmalcolm
Copy link
Copy Markdown

@danmalcolm danmalcolm commented Mar 5, 2026

Work in progress on solution to issue #249.

A few choices / observations made along the way:

  • Key serialization/deserialization (serde) using behaviors is an opt-in thing for Kafka endpoints. If no behaviors are specified, then the existing built-in UTF8 string keys will be used
  • Common (not Kafka-specific) key serde logic (configuration, adding raw key to envelope data structures) added at common Silverback.Integration level. Any broker technologies that use it (just Kafka at present) need to extend their own endpoint configuration builders.
  • Silverback.Integration.Kafka extended to support key serde configuration and execution
  • Schema Registry key serdes and configuration / builders added
  • Silverback currently works only with Kafka message keys of type string. Changing Silverback to support other Kafka key types would be a breaking change as mentioned in the issue discussion. However, it seems reasonable to extend serialization/deserialization for string keys in v5.
  • I tried to follow existing patterns for tests - most of the Schema Registry key serde logic is covered by E2E tests, but this seems in line with main message serialization tests. Happy to extend unit test coverage in any areas needed.
  • I've tried to get build to pass, after fixing some code style issues, there seem to be some intermittent failing tests in areas I haven't touched
  • The new v5 codebase is very nice to work with - consistent and easy to follow

A few things left to tidy up - looking for some general early feedback at this stage.

Couple more questions:

  • Is any local setup needed to run the tests that use Ductus-managed containers.
  • I can also help with documentation changes for the new builder API methods

@danmalcolm danmalcolm force-pushed the feature/kafka-key-behaviors branch 3 times, most recently from eb7527a to b948c31 Compare March 5, 2026 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant