What are the load balancing algorithms supported for the stateless brokers?

[shaeqahmed]

Does Tansu support load balancing based on consistent hashing or balancing the incoming connections? What are the options supported and does system do anything to detect and balance hot partitions or to evenly spread the load of overpartitioned topics / large number of topics with a highly skewed load?

[shortishly]

None at present. However, there is work in progress on authentication (#47) that could enable throttling of clients, and routing of requests depending on a QOS (with accompanying ACL/config). There is also a tansu proxy that could be used to route/layer requests:

tansu/tansu-proxy/src/lib.rs

Lines 174 to 200 in ad005c6

	let meta = HijackLayer::new(
	FrameApiKeyMatcher(MetadataRequest::KEY),
	(
	FrameRequestLayer::<MetadataRequest>::new(),
	MapRequestLayer::new(move |request: MetadataRequest| {
	MetadataRequest::default()
	.topics(request.topics.map(|topics| {
	topics
	.into_iter()
	.map(|topic| {
	MetadataRequestTopic::default()
	.name(topic.name)
	.topic_id(topic.topic_id.or(Some(NULL_TOPIC_ID)))
	})
	.collect()
	}))
	.allow_auto_topic_creation(
	request.allow_auto_topic_creation.or(Some(false)),
	)
	.include_cluster_authorized_operations(
	request.include_cluster_authorized_operations,
	)
	.include_topic_authorized_operations(
	request.include_topic_authorized_operations.or(Some(false)),
	)
	}),
	MapResponseLayer::new(move |response: MetadataResponse| {

it currently adapts metadata/find_coordinator responses, but could provide routing instead (requests for topic A all go to broker B).

As all storage/metadata is delegated to PostgreSQL/S3/SQLite/..., it could be used to route client A to SQLite, B to PostgreSQL/etc.

One of the original concepts, was having lots of "small" single broker ephemeral clusters rather than having a single "big" cluster with all the issues of noisy neighbours/etc.

I am also looking at slatedb (#500) for read heavy topics on S3 that might benefit from a layer of caching.

Happy to discuss in more detail if you have particular questions.