Repository navigation
feat: make D1 replicas opt-in as safer default and add support for Sessions API to mitigate stale data returned from replicas - #813
Conversation
b8f635e to
ff403b4
Compare
|
0f6dfa8 to
ca8896e
Compare
|
@yangshun thank you for the heads-up! |
552cda1 to
6bd4980
Compare
Brooooooklyn
left a comment
There was a problem hiding this comment.
Thanks for this, and for the careful write-up on replicas. The core runtime change is small and mostly right: outside a request, db still starts on primary for each access, auth keeps its own primary client, the new module adds no heavy imports, and the unit tests pass.
Requesting changes for:
| Level | Issue |
|---|---|
| medium | The docs middleware's setSession() throws when a loader that used db makes a server-side fetch from void/client (same request scope) |
| medium | The cookie and both-transports recipes turn off ISR and shared caching for every route |
| low | The generated .void/db.d.ts doesn't match the published void/db types (getBookmark generic, DbSessionStart) |
| low | Any property read or await on db locks setSession(); unclear error outside the request scope; constraint typos compile |
| low | defer() note, bookmark input note, test timeouts, placeholder |
Missing tests: the published module's out-of-scope path (each access uses first-primary) and the getBookmark(client) error for a client not backed by a session.
Also: the branch is behind main and needs the rebase requested earlier. The playground is a scope call for maintainers; trimming it and its browser suite would make it easier to maintain.
|
@Brooooooklyn thank you for the review. I'm looking into it. What do you think about making replicas opt-in? I can make the change disabling it in the same PR. |
|
Thanks! Yes, let's make replicas opt-in here. Please add a config option, have provisioning skip Both provisioning paths set it today:
The review comments above still apply, since the session helpers are still useful once replicas are turned on. |
a364521 to
ae21985
Compare
|
@Brooooooklyn I addressed the issues and reworked API a bit to avoid typos in Especially 675064c where I add the option and make it opt-in for new deployments. The GPT says even setting
I don't know how release workflow looks like and if we should (I think we have to) to communicate to existing users so they aware about the issue, upgrade and disable replicas. cc @cpojer |
… out of Void scope
675064c to
2f9ba18
Compare
database.readReplication config, add support for D1 Sessions API to mitigate stale data returned from replicas
database.readReplication config, add support for D1 Sessions API to mitigate stale data returned from replicas
This PR closes #783.
It adds two new primitives into D1
void/dbthat utilize D1 Session API:getBookmark- returns the current session bookmarksetSession- starts a new session from a previous bookmark.Using these allow to solve the D1 Replicas problem in the scenario
read OLD -> write NEW -> read OLDand ensure sequential consistency.It also makes replicas opt-in and adds a new config
database.readReplicationto enable/disable the feature:See guides for more information and usage examples:
TL;DR;
To utilize Sessions API the user has to create a middleware:
To make it work with client requests, instead of the cookie (or along with the cookie transport), the devs should use a header, e.g.,
X-DB-Bookmarkand make requests through a wrapper that sets the bookmark.Show the disabling replicas pitch
Why disabling replicas?
TL;DR; replicas are not safe and might result in support tickets, bugs and security issues when enable unknowingly.
Enabling them by default creates two problems not present in a default D1 setup and not common in other environments:
User might see a stale data right after they perform a write request.
User might see old data after seeing a new one.
An action triggering an account suspension might not propagate the account state and allow already suspended account to perform an action they should not be able. See Avoid stale reads with a primary-first session.
While the first two are annoyances and just a source of support requests, the last one is a security issue.
The fact that we enable replicas by default is hidden in the documentation and the user, even experienced with D1, could get caught by surprise and deploy unsafe code unknowingly.
Making it opt-in would allow devs that want/need replicas to understand cons and decide if they want it. Those who decide to opt-in will keep in mind these nuances and work around them. If they need the advantage that replicas provide, they will be happy to use said APIs regardless how awkward they are and how many docs they have to read.
For some apps it would be completely normal, but not for all.
Why not incorporate a bulletproof solution into Void?
The problem is that every query could produce fresh bookmark, even a read. So we have to track all request bookmarks and resolve the newest one.
The single cookie approach described in the guide breaks with multiple requests where a newer request could overwrite a fresh bookmark with an older one (i.e. dashboard).
So we'll have to track every query bookmark and store each in a separate cookie, so a middleware could resolve the freshest one. It might lead to hitting server/browser limits on the cookies number and break. Not to mention overhead of carrying these cookies with every request.
A built-in multi-cookie solution could work out and be a good default middleware for users that opt-in and ok with a possibility that the describe problems might appear.
But adding this overhead and still not giving 100% guarantee doesn't sound like a good default.
Reproduction
In cases where primary D1 is deployed to say
WEURbut the nearest edge isSINright after write, theSINreplica might not get the update fast enough and the user will see data before edits.Unless you're in Europe, where the primary D1 is deployed, you can see the problem in the playground (you might not have the access, ping me): https://voidzero-void-d1-bookmarks.skoss-individual-account.workers.dev/without-bookmark?experiment=immediate&trials=20&pressure=0.5&size=0.5&probes=20
To reproduce:
In "Read-only routing baseline" section click "Run probes" button. Repeat until you see the replica region closest to you (
replica / APAC / SINin my case:Then go a section below "Write/read experiment" and set:
Press "Run Trials" button. Unless you're unlucky or D1 got significant replication pressure and switched from replica to the primary D1, you'll see some of the requests getting stale data (stale data is only possible on a replica,
SINin my case):Depending on your region and current state of D1 instance you might not see stale data at all. If you see that read target is
primary / WEURwait for a 1-2 minutes and start from 1. You can also increase replication pressure a bit. A high pressure makes D1 switch to primary and only hit replicas occasionally. Most likely you will encounter a network error caused by D1 being overwhelmed.The working solution is on the "Manual bookmark" page where data is always fresh: https://voidzero-void-d1-bookmarks.skoss-individual-account.workers.dev/with-bookmark?experiment=immediate&trials=20&probes=20&pressure=0.5&size=0.5