-
Notifications
You must be signed in to change notification settings - Fork 487
Update Kafka Progress Topic config to avoid long startups #34469
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Kafka Progress Topic config to avoid long startups #34469
Conversation
dcb62be to
bd21cf6
Compare
bd21cf6 to
b1b4d77
Compare
7fdc26a to
36f6e02
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, thanks @patrickwwbutler !
There's also a section for the progress topic in the docs that could use a mention of the segment size: https://materialize.com/docs/serve-results/sink/kafka/#automatic-topic-creation
DAlperin
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thank you!
martykulma
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🐟
Updates the segment.bytes config value to 128MiB to avoid having to process large amounts of progress data when creating new sinks.
Motivation
https://github.com/MaterializeInc/database-issues/issues/9894
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.