Feature proposal: built-in multi-host balancer and reconnection with backoff

[0xCA]

Is your feature request related to a problem?

I plan to implement and MR further enhancements upon #110

Describe the solution

Goals:

• Provide a native support for multi-node deployments. Essentially, there are going to be multiple hosts with the same credentials, and a connection to each host.
• Single-node will just provide one host in the New(hosts []string, ..)
• Exponential backoff for re-connections. This is mainly needed for the case where clients are connecting to the db directly (each authenticated with highly limited permissions), instead of going through a backend, to avoid db (cluster) overloads after temporary outages.
• Possibly keep a way for complete external implementations of the same, while disabling internal one

Requirements:

• Stable ws implementation
• Stable wsrc implementation

Alternative methods

Only expose necessary stuff for everyone to implement the thing on their own?

Reasoning:

Convenience for everyone, and a much nicer balancer implementation, with a call to it hidden within a send method

[mumoshu]

Hey @0xCA! Thank you very much for your continued effort on this!

So to be clear, am I correct that this is going to be a client-side load balancer that is implemented as a Go library used directly by apps, rather than a proxy server that sits between the app and SurrealDB?

To be even clearer, I don't think this could be used to build a proxy server, because to do so, you need to transparently restore sessions without the client knowing when reconnection occurs. This could be useful only for use as a client-side load balancer.

I think it would be a good fit for a new contrib package within this project (part of #229) if it's the former.

[0xCA]

Hey @mumoshu! It is of course client-side, but it's kinda neither of what you suggest.
I want it to sit deeply in the SDK itself. So that all the end user need to do is to provide multiple hosts when they are instantiating a new DB. The rest is fully transparent and automatic.
The change is kinda simple and basically is an expansion of rews. I don't see it as an add-on tool that is used from a separate package by the SDK consumer.

[nickchomey]

I also think that this would be better as part of #229

[0xCA]

@nickchomey how do you think I should integrate it then?

[nickchomey]

No idea. But it seems beyond the scope of the core sdk.

[mumoshu]

The change is kinda simple and basically is an expansion of rews. I don't see it as an add-on tool that is used from a separate package by the SDK consumer.

@0xCA Thank you for your feedback!

To be clear, I would consider extracting it out of contrib only after thorough testing and feedback from active contributors and testers. The rationale is that I believe none of us has enough confidence in how it should be built and how it should work. I had already realized that I was missing the fact that we need to transparently restore sessions, and finding a good abstraction and implementation to do so was quite a stretch for me.

Not saying I won't integrate it deeply into the SDK forever, but I don't think rushing to do so is good.

Let's build it, and then decide whether to integrate it deeply into the SDK. It's a matter of enabling rews in surrealdb.FromEndpointURLString (former surrealdb.New) when we decided to make the SDK transparently use rews. Development of packages like rews can be done hassle-free even though it is in contrib.

[0xCA]

@mumoshu makes sense! One of the reasons why I wanted to implement it now is that the SDK seem unfinished to me, i.e. not production ready. (and it was kinda abandoned just a month ago) So I thought this could be tested along with all the connectivity changes, as it's basically an another one, mixing in multi-node to a reconnection.

Let's at least agree on the integration part for now. Should I implement this as an alternative connection.WebSocketConnection? So that it is injected in DB via FromConnection?

[mumoshu]

@0xCA Thanks for the feedback, as always!

Honestly speaking, I have mixed feelings.

This can be seen as "finished" in the sense that I fixed all the critical issues that had been hanging around for a year or two, making the SDK now usable as a simple wrapper around the SurrealDB RPC v1.

If I deploy apps built using this SDK today, I would deploy it into e.g. K8s and set the liveness check to check the connection, so that the app automatically restarts once the connection is lost, giving you the same benefit as rews without rews. No need for complex session restoration on the SDK side, making debugging things when something goes wrong a breeze.

It would also work in a multi-node scenario; you randomly choose one of the known hosts on app startup, and everything else remains the same. The app restarts once the connection is lost, resulting in the restarted app randomly choosing one of the hosts to connect again.

That being said, I agree we should test this multi-host client LB thing end-to-end in this repository. Let's do it!

Usually, anything outside of the standard library will have its repository.
At this stage, though, I believe consolidating community efforts into this repository for better and thorough integration testing when we make aggressive changes towards v1 would be beneficial to all of us.

Regarding the integration part, you should implement an alternative to the connection.WebSocketConnection interface. So, it should reside within contrib/multihost or something like that, which will have a New function to return the connection object that can be passed to surrealdb.FromConnection(conn) for use.

One potential implementation I had in mind is to tweak rews to randomly choose a host to connect to on every reconnect attempt.
I don't think load balancing each RPC over connections across multiple hosts is a good idea. You should load-balance on connect.
Otherwise, you need to "partition" live queries and their notifications; otherwise, you end up receiving duplicate lq notifications from multiple hosts, which is a waste of compute resources.

[0xCA]

Hey @mumoshu!

If I deploy apps built using this SDK today, I would deploy it into e.g. K8s and set the liveness check to check the connection, so that the app automatically restarts once the connection is lost, giving you the same benefit as rews without rews. No need for complex session restoration on the SDK side, making debugging things when something goes wrong a breeze.

This only works for simpler/highly fragmented services. In many cases it's not acceptable to kill the service if it lost the db connection.

It would also work in a multi-node scenario; you randomly choose one of the known hosts on app startup, and everything else remains the same. The app restarts once the connection is lost, resulting in the restarted app randomly choosing one of the hosts to connect again.

Again, only works for small and highly horizontal services. Otherwise, one such service can overload an sdb node, by only using one all the time.

I don't think load balancing each RPC over connections across multiple hosts is a good idea.

True, can be bad for cache locality. Maybe it should switch host after some threshold amount and frequency of requests hit one node? Kinda "dynamic real-time batching".

Otherwise, you need to "partition" live queries and their notifications; otherwise, you end up receiving duplicate lq notifications from multiple hosts, which is a waste of compute resources.

Oh! This part I completely missed. I don't use LQ yet, and don't know how it works, only how to use it.
Is LQ fully restored after a ws reconnect? Does it start from where it left off on disconnect?

Basically, I need to know how LQ behaves in these situations:

• Temporary network issue between client and sdb node
• One sdb node going down, and:
  Client connects to a different node (I guess, LQ is never restored, as the only thing sdb nodes have in common is the TiKV storage?)
  Client connects to a different node, then the original node goes up

As for partitioning, I would certainly tie each LQ to a particular node, and break it on node failure, after a set timeout, that is chosen by SDK consumer for each LQ.

[mumoshu]

Hi @0xCA! Thank you very much for your continued efforts and the insights.

Here's a lot of two cents from me;

This only works for simpler/highly fragmented services. In many cases, it's not acceptable to kill the service if it loses the db connection.

Yes! I'd rather say "it works when you can accept the tradeoff." That's a typical architecture used in services built over stateful connection-oriented RPC, like networked games, as far as I understand.

And to be clear, if you prefer WebSocket over HTTP for the SurrealDB RPC protocol, did you want a stateful session, LET, UNSET, RESET, and LQ in the first place? You need to accept the fundamental tradeoff there. You cannot make it look like load-balanced per request with a client-side load balancer for stateful connection-oriented RPC. I believe that's not what you are trying to do, but I wanted to be clear on that!

If I need per-request load balancing, I use RPC over HTTP first, and use WebSocket only when necessary. I think you can even go further by transparently switching between HTTP and WebSocket at the load balancing library layer, depending on the RPCs used.

Again, only works for small and highly horizontal services. Otherwise, one such service can overload an sdb node, by only using one all the time.

It depends on your requirements, but if I need to prevent noisy neighbors, I would implement a "quota" layer (at the app layer, not the sdb proxy layer) to force disconnect bad actors, so load-balancing-at-connect-time is enough.

All in all, it's apples vs oranges. Even a client-side loadbalancer library for stateful protocol "only works" when you can implement transparent session restoration, which is not trivial (to me).

True, can be bad for cache locality. Maybe it should switch host after some threshold amount and frequency of requests hit one node? Kinda "dynamic real-time batching".

Yeah, that would work! Good idea.

Is LQ fully restored after a ws reconnect? Does it start from where it left off on disconnect?

As far as I know, no! LQ "session" is tied to the connection emitted LIVE SELECT. You get the UUID of the live notification "session" on LIVE SELECT, but there's no way to "continue" reading that session on another WebSocket connection, as far as I know.

To be clear again, you use LQ to build an auto-refreshing, eventually consistent view of data. It isn't a change log reader, so an ability to continue previous sessions across connections isn't a must-have; and that's why it's not there—just my two cents.

I might not be the best person to talk about LQ, but from what I know;

Temporary network issue between client and sdb node

The session is kept as long as the websocket connection is seen as healthy from both the client and the server perspective.

One sdb node going down, and:
Client connects to a different node (I guess, LQ is never restored, as the only thing sdb nodes have in common is the TiKV storage?)

The session is lost- you need to resend the original LIVE SELECT query from the new ws connection. The new LIEVE SELECT starts from there. You cannot "continue" from where you left off.

Client connects to a different node, then the original node goes up

If you retained the ws connection to the original node and both the client and the server thinks the websocket connection is still healthy, you will start receiving live notifications from the original node too.

This is more about how you handle connections in that case! If your load-balancing library disconnects from the original node when connecting to a different one, you won't receive duplicate live notifications from both nodes.

Hoping this helps with your excellent work!

[0xCA]

Hi again, @mumoshu!

did you want a stateful session

Actually, no. I try to account for all the possible use cases here, so that SDK is useable for everyone, but this is clearly a case that I missed.

You cannot make it look like load-balanced per request with a client-side load balancer for stateful connection-oriented RPC

That's true, unfortunately. I never thought of statefulness beyond auth, as I don't use stateful features.

I'm not sure how to solve this. For LQ it's easy. It's obvious when it's dead. But for LET.. Store all of them in memory, so that an SDK consumer could check if they are still there? Or even automatically re-let them? Might be a decent idea.

Also, there should be an option to create an explicitly stateless balancer. Maybe with the exception of LQ.

I use RPC over HTTP first

I prefer WS for efficiency, not for stateful RPCs. And I see no issue in having multiple WS connections to sdb nodes from a client.
For example, I have a real-word situation where one client (server, not end user) can physically saturate one node's (external) network, so I need my requests to be balanced (also for instantaneous HA).

I would implement a "quota" layer (at the app layer, not the sdb proxy layer) to force disconnect bad actors, so load-balancing-at-connect-time is enough.

No, not an issue I'm trying to solve. See the setup and reasoning above.
For end-user client, connection to one node seem sufficient indeed, and some protections are necessary, of course. (also, is there a per-user rate limit in sdb itself?..)

---

Thanks for LQ insights!
If it's tied to a WS connection, then the handling is pretty obvious. Transparently choose which node to open LQ at, and simply do nothing (after closing the channel) about it after a reconnect or a permanent node loss. The responsibility of opening a new LQ is on a SDK consumer in this case.
The alternative would be the same as with LET, to transparently restore it on a different node, or the old one after a reconnect.

Either way, I'll start with a stateless implementation first.

[mumoshu]

Hi @0xCA! I really enjoy discussing about this complex topic with you!

I'm not sure how to solve this. For LQ it's easy. It's obvious when it's dead. But for LET.. Store all of them in memory, so that an SDK consumer could check if they are still there? Or even automatically re-let them? Might be a decent idea.

Yeah absolutely... I think there's no single answer that serves everyone, so there could be options to auto-re-let or not, auto-restart LQ on reconnect, whether to have two or more "replica" sessions for max availability in exchange to resource waste, and so on...

FYI, I've built rews to do this "auto-re-let" thing on reconnect:

surrealdb.go/contrib/rews/ws_reconnect.go

Lines 245 to 288 in e826560

	// reconnect attempts to reconnect the WebSocket connection.
	//
	// This enhances the Connect method by re-authenticating with the token in the following cases:
	// - SignUp succeeded before the reconnection (SignUp returns a token and it can be used for re-authentication).
	// - SignIn succeeded before the reconnection (SignIn returns a token and it can be used for re-authentication).
	// - Authenticate succeeded before the reconnection (Authenticate accepts the token and it can be used for re-authentication)
	//
	// The token might be expired or invalid which could result in an re-authentication failure.
	// But rews does not handle the re-authentication failure.
	// It is the caller's responsibility to handle the re-authentication failure.
	func (arws *Connection[C]) reconnect(ctx context.Context) error {
	if err := arws.Connect(ctx); err != nil {
	arws.logger.Error("rews.Connection failed to reconnect", "error", err)
	return fmt.Errorf("rews.Connection failed to reconnect: %w", err)
	}
	if arws.sessionNS != "" && arws.sessionDB != "" {
	arws.logger.Debug("rews.Connection is restoring namespace and database", "namespace", arws.sessionNS, "database", arws.sessionDB)
	if err := arws.Use(ctx, arws.sessionNS, arws.sessionDB); err != nil {
	arws.logger.Error("rews.Connection failed to restore namespace and database", "error", err)
	return fmt.Errorf("rews.Connection failed to restore namespace and database: %w", err)
	}
	arws.logger.Debug("rews.Connection restored namespace and database successfully")
	}
	if arws.sessionToken != "" {
	arws.logger.Debug("rews.Connection is re-authenticating with the session token")
	if err := arws.Authenticate(ctx, arws.sessionToken); err != nil {
	arws.logger.Error("rews.Connection failed to re-authenticate with the session token", "error", err)
	return fmt.Errorf("rews.Connection failed to re-authenticate with the session token: %w", err)
	}
	arws.logger.Debug("rews.Connection re-authenticated successfully with the session token")
	}
	for key, value := range arws.sessionVars {
	arws.logger.Debug("rews.Connection is restoring session variable", "key", key, "value", value)
	if err := arws.Let(ctx, key, value); err != nil {
	arws.logger.Error("rews.Connection failed to restore session variable", "key", key, "error", err)
	return fmt.Errorf("rews.Connection failed to restore session variable %s: %w", key, err)
	}
	}
	return nil
	}

But to be clear, I think this is for 80% of people. 20% of people would prefer no auto-re-let.

Also, there should be an option to create an explicitly stateless balancer. Maybe with the exception of LQ.

Sounds great. I'd see use-cases where an explicit stateless balancer that forbids random "let", "unset", "reset", and live notifications, so that, except reauth, they are almost "stateless", simplifying the implementation of the balancer.

I prefer WS for efficiency, not for stateful RPCs. And I see no issue in having multiple WS connections to sdb nodes from a client.
For example, I have a real-word situation where one client (server, not end user) can physically saturate one node's (external) network, so I need my requests to be balanced (also for instantaneous HA).

Very interesting!

Am I correct that you cannot saturate the node if you use RPC over HTTP?
I was thinking that HTTP with minimal headers(where headers are used to make it look like stateful even with the protocol being stateless) is light-weight enough so that the difference between it and RPC over WebSocket is negligible.
They are both over TCP in the end.

also, is there a per-user rate limit in sdb itself?

As far as I know, no- and I think that's not all kinds of DBs (should) support.

Rate limiting doesn't need to be tied to various guarantees a database provides (isolation, durability, etc)- doing it within the DB can stress more important work that only the DB can do.
I believe rate limiting is better done in a "proxy" layer, including a client and server-side load balancer.

I thought a database like FoundationDB had its own rate limiting feature, but from the architecture standpoint it was done in a "proxy" layer of the database, not core transaction or storage layers of it, as far as I remember.

[0xCA]

Hi @mumoshu, I enjoy it as well! We can discuss other aspects of the SDK too, if you wish. (discord or elsewhere)

FYI, I've built rews to do this "auto-re-let" thing on reconnect:

Interesting! However, it doesn't cover the case of using LET in complex queries. I guess, this is impossible to hook, and these vars are inevitably lost.

But to be clear, I think this is for 80% of people. 20% of people would prefer no auto-re-let.

I would prefer no connection-scoped LET even. Only inside a transaction or just a multi-statement query.

Am I correct that you cannot saturate the node if you use RPC over HTTP?

No, that's not about RPC vs HTTP, it's about 10x bandwidth on the client server vs one node bandwidth. That's why I have to balance requests to multiple nodes at the same time in this case.

HTTP with minimal headers

Why sending auth on each request, if it can be implied from the connection? Sure, it's no big deal in low traffic scenarios, but I really like the efficiency of WS combined with a semi-stateless use of it.

I believe rate limiting is better done in a "proxy" layer, including a client and server-side load balancer.

Isn't the whole point of row auth in having no middleware services? So that clients could connect directly from a frontend, for example.
If I'm not missing anything, rate limiting should be possible to implement for row users using functions/events/perms/etc, however, it doesn't feel ideal.
And if the backend layer is inevitable, then there isn't much sense in row auth?