PyConnectSources should be able to share a kafka topic so we do not have to create a new topic for each source. Sources will have to get an ID similar to the consumer group id which will be used to key their offset within the topic.
The ID can also be used as consumer group id by the consumer that reads from that offset topic so it won't have to go through the whole topic history.
In order to further cut down seek time, we can also initialize the topic with a rather high amount of partitions (say 50) and enable compaction on it.
PyConnectSources should be able to share a kafka topic so we do not have to create a new topic for each source. Sources will have to get an ID similar to the consumer group id which will be used to key their offset within the topic.
The ID can also be used as consumer group id by the consumer that reads from that offset topic so it won't have to go through the whole topic history.
In order to further cut down seek time, we can also initialize the topic with a rather high amount of partitions (say 50) and enable compaction on it.