Merge pull request #34469 from patrickwwbutler/patrick/kafka_progress

Update Kafka Progress Topic config to avoid long startups

Author: patrickwwbutler

doc/user/content/serve-results/sink/kafka.md

@@ -70,6 +70,8 @@ running `CREATE SINK`, observe the following guidance:
 | Progress topic | Compaction          | We recommend enabling compaction to avoid accumulating unbounded state. Disabling compaction may cause performance issues, but will not cause correctness issues.
 | Progress topic | Retention           | **Must be disabled.** Enabling retention can cause Materialize to violate its [exactly-once guarantees](/sql/create-sink/kafka#exactly-once-processing).
 | Progress topic | Tiered storage      | We recommend disabling tiered storage to allow for more aggressive data compaction. Fully compacted data requires minimal storage, typically only tens of bytes per sink, making it cost-effective to maintain directly on local disk.
+| Progress topic | Segment bytes.      | Defaults to 128 MiB. We recommend going no higher than 256 MiB to avoid
+slow startups when creating new sinks, as they must process the entire progress topic on startup.
 {{< warning >}}
 {{% kafka-sink-drop %}}
 {{</ warning >}}

src/sql/src/plan/statement/ddl/connection.rs

@@ -334,6 +334,7 @@ impl ConnectionOptionExtracted {
                         }).transpose()?,
                         topic_config: btreemap! {
                             "cleanup.policy".to_string() => "compact".to_string(),
+                            "segment.bytes".to_string() => "134217728".to_string(), // 128 MB
                         },
                     },
                     options: BTreeMap::new(),

test/testdrive/kafka-sink-topic-config.td

@@ -105,4 +105,4 @@ $ kafka-verify-topic topic=testdrive-kafka-progress-${testdrive.seed} topic-conf
   FORMAT JSON
   ENVELOPE UPSERT
 
-$ kafka-verify-topic topic=testdrive-kafka-progress-${testdrive.seed} topic-config={"cleanup.policy": "compact"} replication-factor=1 partition-count=1
+$ kafka-verify-topic topic=testdrive-kafka-progress-${testdrive.seed} topic-config={"cleanup.policy": "compact", "segment.bytes": "134217728"} replication-factor=1 partition-count=1