Rework of bootstrap process

[mrForza]

Reviewers

• Main Reviewer: @Serpentian
• Second Reviewer: @Gerold103
• Team Lead: @sergepetrenko
• CTO: @sergos

Tickets

• Replica bootstrap / rebootstrap fails if role writes to sync space #11039

Changelog

• v1: Initial document.
• v2: Add a detailed explanation of the main solution (getting rid of final_join stage), add cons & pros and fix some minors in initial solution (Capturing vclock when limbo has no pending sync transactions).
• v3: Rework the 1st solution (changed bootstrap protocol), add a new solution (tracking stop_vclock right after sync _cluster write), change structure of document, add new reproducer of the bug.
• v4: Add a more detailed bug descriptions (master<->replica deadlock and master's inability for write load), split 1st solution into the two part for clarity, worked out the options of catch up mechanism, add "Using build_in_memory_checkpoint" solution which suggested Gerold103, get rid of a lot of code block in 1st solution for RFC's clarity.
• v5: Get rid of remaining implementation details, add 4th variant of catch up and describe all rejected alternatives, change "Restart between the checkpoint and register" section, make some minor style fixes.
• v6: Remove Harness raft 'learner' feature #6281 ticket from "Tickets", remove "master's inability for write load" bug from "Problem overview", reduce the main solution (leave only Part 1 which fixes master&replica deadlock), add more pros & cons for every solution, make some style fixex.

Problem overview

Master & Replica deadlock during bootstrap

Bug description:

• A master leads the synchronous limbo with quorum Q.
• A sync txn is prepared on the master; its data lands in WAL, but its CONFIRM is written a moment later (because some other replicas have not enough time to confirm it).
• A new replica starts bootstrapping right in that window.
• The master takes the snapshot, registers the replica, then captures stop_vclock while the CONFIRM is still unpersisted.
• During the FINAL_JOIN stage, the master sends the sync txn's data (≤ stop_vclock) but not its CONFIRM (> stop_vclock).
• The replica applies the sync txn → it sits in its limbo waiting for the CONFIRM.
• Bootstrap then runs the initial checkpoint, which waits for that sync txn to commit.
• The CONFIRM can only arrive via SUBSCRIBE, which starts only after the checkpoint - circular wait.
• The replica deadlocks until replication_synchro_timeout, then bootstrap fails.

The places in the code where the bug is reproduced:

Replica's side:

```
bootstrap_from_master
└── gc_checkpoint
    └── gc_do_checkpoint
        └── box_checkpoint_build_on_disk
            └── txn_checkpoint_build
                ├── txn_limbo_last_synchro_entry
                ├── txn_on_commit / txn_on_rollback
                ├── txn_journal_flush
                └── fiber_yield_deadline [LOOP]     ← wait until all sync txns in limbo will be confirmed or rollbacked
```

Master's side:

```
box_process_join
└── relay_final_join
    └── relay_final_join_f
        └── recover_remaining_wals
            └── recover_xlog
                └── if (signature >= stop_vclock->signature) ← filters out rows with signature ≥ stop_vclock, so CONFIRM is never sent
                    return;
```

Solutions

1. Changing the bootstrap protocol (getting rid of FINAL_JOIN stage)

In Tarantool there are 3 main ways how the bootstrap process can proceed:

1. Anonymous bootstrap

   
2. AnonToNamed bootstrap

   
3. Named bootstrap

   

The solution reorders the bootstrap so that registration (which increases the master's sync quorum) happens only after the replica is already an active, catching-up subscriber that can send ACKs. Concretely, a named replica no longer bootstraps via JOIN from a new master; instead it fetches the master's snapshot, builds its initial checkpoint, subscribes and catches up anonymously, and only then sends REGISTER.

1. Fix of Master & Replica deadlock during bootstrap Replica bootstrap / rebootstrap fails if role writes to sync space #11039

   We replace the JOIN-based bootstrap of a named replica with the sequence that anonymous replicas already use, plus a deferred registration: FETCH_SNAPSHOT → initial checkpoint → REGISTER → SUBSCRIBE. The key idea is a small reordering of bootstrap process — the replica's initial checkpoint is created before any WAL stream that can carry unconfirmed synchronous transactions:

   1. FETCH_SNAPSHOT instead of JOIN. The replica receives a consistent snapshot (with limbo/raft state in JOIN_META) via box_process_fetch_snapshot, which never sends unconfirmed rows. The replica recovers from it and creates its own initial checkpoint while its limbo is guaranteed clean, so nothing can be placed into the limbo during gc_checkpoint.

   2. REGISTER after the checkpoint. Once bootstrap has its checkpoint, the applier sends REGISTER and receives the "final-join" WAL stream via box_process_register. If this stream contains an unconfirmed sync transaction, it is now harmless: it just sits in the replica's limbo as a pending entry — the same supported steady state as any running replica receiving an in-flight sync transaction.

   3. SUBSCRIBE is unchanged from the old flow. The only difference is what it may find on arrival: if the REGISTER stage delivered synchronous transactions whose CONFIRM entries were not yet written on the master (they lay after stop_vclock), those transactions are now pending in the replica's limbo. The SUBSCRIBE resolves them: the replica ACKs the master through it, the master's pending sync transactions gather quorum and get confirmed, and the CONFIRM entries flow back to the replica through the relay, draining its limbo — just like for any running replica that received an in-flight sync transaction during normal replication.

   This part requires only replica's side changes, because our current master can serve any types of replication IPROTO requests in any way. On the replica's side we need to:

   1. Make applier_f always call applier_fetch_snapshot regardless of anon/named status of replica.
   2. Change the waiting of applier's statuses in box_bootstrap_from_master (from FINAL_JOIN/FETCHED_SNAPSHOT to APPLIER_FETCHED_SNAPSHOT).
   3. Remove applier_join entirely — named replicas no longer bootstrap via JOIN.
   4. Remove all unused applier's statuses — APPLIER_FINAL_JOIN and APPLIER_JOINED.

Pros & Cons:

• ❎ An architecturally more correct solution
• ❎ This solution needs only replica's side changes (no master)
• ❎ A lightweight solution (~50 lines of code)
• 📛 Many tests may stop working due to the deletion of the final_join
• 📛 All documentation notes should be updated due to deletion of final_join

Rejected alternatives

2. Using build_in_memory_checkpoint on the master side

The core idea of this solution: stop_vclock must not be a raw copy of instance_vclock, but a journal position V for which the master can prove that everything up to V is committed. That is exactly what box_checkpoint_build_in_memory produces: a pair of "journal vclock V" + "limbo state (confirmed_lsn, queue owner, term) captured at the exact moment the last pending sync txn of the build time got committed". The master uses the checkpoint's vclock as stop_vclock, streams (start_vclock, stop_vclock] as today, and then sends the checkpoint's limbo state as a "fake" PROMOTE meta row - the same mechanism send_join_meta already uses in the initial join. The replica applies it via txn_limbo_process and instantly commits all sync txns <= stop_vclock, so its initial checkpoint never waits for a CONFIRM that can only arrive via SUBSCRIBE.

The key insight (why this beats both the drain and the on-commit-trigger capture): we do not try to find a point in the xlog where every CONFIRM covering the data is also below it - Serpentian showed this is impossible without draining (T1 → T2 → CONFIRM(T1) → T3 → CONFIRM(T2) → … chases forever). Instead, we ship the limbo state next to the data, exactly like a snapshot does: snapshot recovery never replays CONFIRMs of the rows stored in it, it restores confirmed_lsn from the limbo state saved in the snapshot.

Pros & Cons:

• ❎ Preserves existing bootstrap protocol order.
• ❎ Zero limbo changes: no is_frozen, no drain, no new flags or states.
• 📛 Replica-side change + version/feature gate needed.
• 📛 Doesn't solve the underlying architectural issue.
• 📛 Testing matrix expands significantly (old/new master × old/new replica combinations).

3. Drain limbo during master freeze

The core idea of this solution: during bootstrap process the master should capture stop_vclock only when the limbo is "completed" (every sync txn with data in WAL also has its terminal record in WAL - CONFIRM or ROLLBACK).

High-level algorithm:

1. Block new sync-txn submissions until all sync entries are resolved.
2. Ensure that all existing entries in limbo are materialized in WAL (i.e. has a valid lsn).
3. Wait for the last sync txn to be resolved (COMMIT or ROLLBACK); loop until the limbo is empty of submitted sync txns.
4. If some sync txns can't be resolved - fail bootstrap process and start over.
5. Capture stop_vclock and then replay all final_join rows in the interval (start_vclock; stop_vclock] to newly added replica.

Implementation details:

1. Introduce ER_SYNC_LIMBO_DRAIN_TIMEOUT - an error which sends from master to replica if some pending sync txns don't have time to be resolved during bootstrap process in replicaset.

2. Introduce two new fields in struct limbo - bool is_frozen (when true, new sync-txn submissions must wait) and struct fiber_cond frozen_cond (signaled on unfreeze).

3. Define a new limbo function which captures vclock only when all sync txns entries are confirmed or rollbacked. It is important to mention that if some sync txn don't have time to be resolved we raise a new error to new replica.

   txn_limbo.cc

   int
   txn_limbo_snapshot_vclock_after_drain(struct txn_limbo *limbo,
   			      				      struct vclock *vclock)
   {
       /*
        * Only the leader persists CONFIRMs/ROLLBACKs and drives the queue.
        * A non-leader has nothing to drain.
        */
       if (limbo->state != TXN_LIMBO_STATE_LEADER) {
   	    vclock_copy(vclock, instance_vclock);
   	    return 0;
       }
       /*
        * Freeze new submissions so that the drained set is finite and no
        * new sync rows enter the WAL range being captured. Note, that
        * txn_limbo_flush() bypasses the freeze gate.
        */
       limbo->is_frozen = true;
       int rc = -1;
       if (txn_limbo_flush(limbo) != 0)
   	    goto unfreeze;
       /*
        * Wait for all the pending sync txns to reach a terminal state, so
        * that their CONFIRMs and ROLLBACKs appear within the captured
        * range.
        */
       double deadline = fiber_clock() + replication_synchro_timeout * 2;
       while (txn_limbo_last_synchro_entry(limbo) != NULL) {
   	    bool is_rollback;
   	    if (fiber_is_cancelled()) {
   		    diag_set(FiberIsCancelled);
   		    goto unfreeze;
   	    }
   	    if (txn_limbo_wait_last_txn(limbo, &is_rollback,
   				        deadline - fiber_clock()) != 0)
   		    break;
       }
       if (txn_limbo_last_synchro_entry(limbo) == NULL) {
   	    vclock_copy(vclock, instance_vclock);
   	    rc = 0;
       } else {
   	    diag_set(ClientError, ER_SYNC_LIMBO_DRAIN_TIMEOUT);
       }
   unfreeze:
       limbo->is_frozen = false;
       fiber_cond_broadcast(&limbo->frozen_cond);
       return rc;
   }

4. In txn_limbo_submit add a waiting loop in order to not allow sync txns appear on master:

   txn_limbo.cc

   int
   txn_limbo_submit(struct txn_limbo *limbo, uint32_t id, struct txn *txn,
           size_t approx_len)
   {
       // Only after all sync txns are resolved (during bootstrap) we can accept new sync txns.
       while (txn_limbo->is_frozen && !fiber_is_cancelled())
           fiber_cond_wait(&txn_limbo->frozen_cond);
       ...
       return txn_limbo_queue_submit(&limbo->queue, id == 0 ? instance_id : id,
                       txn, approx_len);
   }

5. In box_process_register and box_process_join insert a txn_limbo_capture_vclock_after_all_confirms waiting function

   box.cc

   void
   box_process_register(struct iostream *io, const struct xrow_header *header)
   {
       ...
       box_register_replica(&req.instance_uuid, req.instance_name);
       ...
       /* Remember master's vclock after the last request */
       struct vclock stop_vclock;
   if (txn_limbo_snapshot_vclock_after_drain(&txn_limbo, &stop_vclock) != 0)
   	diag_raise();
       ...
       relay_final_join(replica, io, header->sync, &start_vclock, &stop_vclock);
       say_info("final data sent.");
   }

   void
   box_process_join(struct iostream *io, const struct xrow_header *header)
   {
       ...
       struct vclock start_vclock;
       relay_initial_join(io, header->sync, &start_vclock, req.version_id,
               NULL);
       ...
       /* Remember master's vclock after the last request */
       struct vclock stop_vclock;
   if (txn_limbo_snapshot_vclock_after_drain(&txn_limbo, &stop_vclock) != 0)
   	diag_raise();
       ...
       relay_final_join(replica, io, header->sync, &start_vclock, &stop_vclock);
       say_info("final data sent.");
   }

Pros & Cons:

• ❎ No iproto protocol changes
• ❎ A draft patch is ready and all replication tests are passed
• 📛 We freeze master until all pending transactions are completed (confirm or rollback)
• 📛 There are can be problems with deferred quorum recalculation
• 📛 If master can't resolve all pending sync txns in limbo, the bootstrap process fails

[sergepetrenko]

Thanks for the design!

There's one extra issue with synchronous join you haven't mentioned: when an entry with replica's ID is committed on master, master re-calculates its replication_synchro_quorum.
So the following is possible:

1. Master works as usual, its quorum is Q and it has enough live replicas to commit transactions with this quorum
2. Master processes a JOIN request: assigns an ID to a new replica, which in turn results in quorum recalculation. Quorum is now Q + 1 (note, the ID assignment itself is processed with the old quorum Q)
3. It happens so that there are not enough live replicas for the master to commit anything with quorum Q + 1.
4. The "no quorum" situation persists until the replica's final join step is complete, because, as we know, there are no heartbeats during the final join stage, so the new replica cannot ack the incoming transactions anyhow.

That's a serious problem. On large installations with high write load it's quite easy to achive situations where the final join stage will run for minutes. And for that whole time the cluster will be unable to serve writes given an unlucky enough replica topology.

For example:
There's a cluster with 3 nodes, quorum = 2, one of the nodes is down. As soon as the fourth node is joined, quorum = 3, but its unachievable unless the new replica finishes join.
Or: there's a single node, so quorum = 1. As soon as the new replica is joined, quorum becomes 2. Same problem as above.

The alternative proposal (with bootstrap protocol rework) handles this issue in the following way:

1. Only send the registration request as soon as the replica sees it has a small enough lag with the master, thus limiting the potential downtime
2. The registration flow allows heartbeats, so replica starts acking fresh master's transactions as soon as it receives its own registration.

Speaking of your proposal, I like that it turned out so concise and easy to implement, but I don't see how this issue can be handled there. Maybe you have any ideas?

[mrForza]

Now, the quorum recalculation (replication_synchro_quorum) is deferred until the join is over. A freeze counter (box_quorum_freeze_count) and a pending flag (box_quorum_update_pending) are introduced in box.cc. While the freeze is active, box_update_replication_synchro_quorum does not recalculate the quorum and only marks the update as pending. box_process_join and box_process_register freeze the recalculation right before box_register_replica and release it via a scoped guard at the end of the handler, i.e. after the final join data has been sent. On the last release the deferred recalculation is executed, so the quorum grows only when the new replica is about to subscribe and ACK

High-level code of box.cc

static int box_quorum_freeze_count;

static bool box_quorum_update_pending;

static void
box_quorum_freeze(void)
{
	box_quorum_freeze_count++;
}

static void
box_quorum_unfreeze(void)
{
	assert(box_quorum_freeze_count > 0);
	box_quorum_freeze_count--;
	if (box_quorum_freeze_count == 0 && box_quorum_update_pending) {
		box_quorum_update_pending = false;
		box_update_replication_synchro_quorum();
	}
}

void
box_update_replication_synchro_quorum(void)
{
	if (box_quorum_freeze_count > 0) {
		box_quorum_update_pending = true;
		return;
	}
	...
}

void
box_process_join(struct iostream *io, const struct xrow_header *header)
{
	...
	relay_initial_join(io, header->sync, &start_vclock, req.version_id, NULL);
	say_info("initial data sent.");

	box_quorum_freeze();
	auto quorum_guard = make_scoped_guard(box_quorum_unfreeze);

    ...
	box_register_replica(&req.instance_uuid, req.instance_name);
    ...
    relay_final_join(replica, io, header->sync, &start_vclock, &stop_vclock);
	say_info("final data sent.");
    ...
}

void
box_process_register(struct iostream *io, const struct xrow_header *header)
{
    ...
	say_info("registering replica %s at %s", ...);

	box_quorum_freeze();
	auto quorum_guard = make_scoped_guard(box_quorum_unfreeze);

	box_register_replica(&req.instance_uuid, req.instance_name);
    ...
	relay_final_join(replica, io, header->sync, &start_vclock, &stop_vclock);
	say_info("final data sent.");
    ...
}

After this mini-patch all replication tests are passed!

[Gerold103]

About the problem of the quorum being not achievable during join - I like the Sergey's proposal to delay the registration until the replica is caught up. This looks elegant and simple. Adding new "frozen" states only increases the complexity. We had such "frozen" and "fenced" states already in the limbo, and their removal only made things simpler.

Regarding the problem of CONFIRM mentioned in the RFC itself - I still don't see why box_checkpoint_build_on_disk() doesn't help 🤔.

[mrForza]

I still don't see why box_checkpoint_build_on_disk() doesn't help 🤔.

Your suggestion is in progress) I will reply soon.

[Serpentian]

Good RFC, we still have several major comments to discuss, but in general it's already in very good shape.

---

applier_subscribe gained a stop_on_follow parameter that makes the loop return as soon as the applier transitions into the APPLIER_FOLLOW state
add a waiting of APPLIER_REGISTERED then APPLIER_READY for named replicas in the end of this function.

In the scope of the RFC review I won't review the code descriptions you have there, I suppose it should be done in the PR when it's ready. Basically, in that specific case I'm ok with anon bounded subscribe, but I'm not sure introducing the flag stop_on_follow is the best way to implement that and propose to decide it later. And speaking of waiting for REGISTERED, I'm not sure this will work, this state is skipped, if replica is already registered.

So, if I were you I'd avoid using such specific wording of solutions in code, we're not discussing the implementation, only ideas. But up to you.

During this phase the replica is an anonymous subscriber: it receives the master's rows (including the confirm/rollback of synchronous transactions) and is therefore able to send ACKs, which lets the master drain its pending sync queue.
Because the replica is already caught up and ACK-capable, registering it no longer risks stalling the master.

I find this part of description a little bit confusing and just wanna make sure, we understand that: Replica is always "able to send ACKSs", there's only two questions:

1. Which ACKs does it send after the quorum is increased (registration happens) - the transactions, which were done 10 minutes ago or 0.1 second ago. We're basically trying to minimize the tail in the box_process_register -> relay_final_join, so that we send as small amount of data there as possible in order to start ACKing the non confirmed transactions already (and not some old confirmed ones).
2. Whether master cares about our ACKs: master ignores ACKs from anon replicas, which is ok, but the first sentense is not correct
3. Whether we send these ACKs.

Speaking of the last one: can we consider sending ACKs as early, as possible? Now we start the ACK machinery on replica only in SUBSCRIBE, but we can do that even earlier: in the register stage. But only because of the changes, proposed in the current RFC. There is an important durability condition: ACKs must represent persisted data. The proposed flow creates the initial checkpoint and enables WAL before registration, which supports this. Legacy JOIN applies its final stream through a recovery journal that performs no actual writes, so we cannot give that path the same quorum (see the recovery_journal_write, now we just accumulate data in memory and then write the snap with this data).

Catch up mechanism's implementation. ii. Bounded wait + error raising.
the whole long flow again — fetch_snapshot → initial checkpoint creation → anon subscribe → catch-up

If only it was just that flow) The problem here is that recovery happens a lot earlier than we subscribe: the connections are created, we check ballots, then appliers are on pause and we recover snaps/xlogs (which can take hours) and only after that appliers are resumed. So, we'll just exit with error after user waited for 5 hours for tarantool to start. I agree, this is not an option for sure.

Catch up mechanism's implementation. iii. Bounded wait + background completion.

And this solution looks like the breaking change to me: it changes more than box.info reporting: startup currently requires a registered identity, and its subsequent Raft initialization calls a function which requires a nonzero ID. Applications may also use the ID immediately after box.cfg() returns. And since the replication_sync_timeout is 0 by default, almost any replica startup will end up in that new anon state. So, we either should introduce the compat and make sure user knows, that from now on it's possible that replica can return with anon state from box.cfg, without any id, or we shoulldn't introduce the breaking cahnge at all.

And in order not to introduce the breaking change we should use the solution, Vladislav proposed: register after bounded catch-up wait. But using the replication_sync_timeout will leave the absolute majority of users in the same state as before the patch: possible inavailability on register of new replica, which I don't like. And I'm not sure it's good idea to use the replication_sync_lag/timeout in two places. Maybe we should consider introducing ad new independent catch-up lag, whiich could inherit replication_sync_lag when unspecified (we already have the replication_reconnect_timeout working like that with replication_timeout). And separate non-zero catchup timeout for it.

Restart between the checkpoint and register stage and box.cfg

It seeems I'm ok with automatic register if no replication_anon is set, but we then should also describe the case, when master has commited registration, but replica didn't receive it and crashed. The now proposed anon SUBSCRIBE will fail. And the 4th part also doesn't decscribe the ER_INSTANCE_NAME_MISMATCH error (compare empty name with configured one).

---

---

@Gerold103, please, resolve the comments, if they're not useful anymore, way too many text in the comments

[mrForza]

I'm not sure this will work, this state is skipped, if replica is already registered.

Right, applier_register() is skipped when the own _cluster row comes with the snapshot. Since this is again an implementation details, I will remove this from my RFC, but I will take it into account when I start writing PR (the same is for stop_on_follow parameter).

So, if I were you I'd avoid using such specific wording of solutions in code, we're not discussing the implementation, only ideas. But up to you.

Yes, agree. I'm still learning how to write short and meaningful RFCs without implementation details.

Can we consider sending ACKs as early, as possible? ... in the register stage.

I think we need master's side changes: relay_final_join has no relay_reader_f, so the master wouldn't read those ACKs. Since the register tail is small in the new flow, I'd skip it.

And this solution looks like the breaking change to me.

Sadness, I'll write it down in rejected alternatives of "catch up mechanism". After our Monday's discussion I will stop at the first option (infinite wait).

we then should also describe the case, when master has commited registration, but replica didn't receive it and crashed

It will be done!

[Gerold103]

v4 is a big step, thanks. Part 1 is now the minimal fix of #11039 and part 2 is the #6281 work on top of it, which is the split we asked for. Below I go through Nikita's points and the updated RFC.

Part 1, the #11039 fix

Agreed as written, with one correction which Nikita also spotted. Point 1.2 says bootstrap_from_master() should wait for APPLIER_REGISTERED and APPLIER_READY at the end. That wait never completes for a registered replica which rebootstraps: its own _cluster row arrives inside the snapshot, replica_set_id() restores the ID while it is applied, need_id is false, and applier_register() is never called, so APPLIER_REGISTERED is never set. The wait must be conditional on box_is_anon(), and once part 2 lands it moves out of bootstrap_from_master() anyway, see the box_cfg_xc() sequence below.

I assume the part 1 is just for the sake of completeness of the doc, and we will be implementing right away the part 2.

Part 2, the #6281 fix

Anon acks. The text says the anonymous subscriber "is able to send ACKs, which lets the master drain its pending sync queue". The problem is that anonymous replica acks are ignored by everything, Raft and limbo both. This fact and things based on it won't survive. Nikita also spotted that. In fact, during this anon part of the subscribe the replica will be probably catching up on some old txns which are already acked and confirmed a while ago.

Lets correct that in the text.

ACKs in the register stage (Nikita's idea). With part 2 the register stream is a handful of rows, since otherwise we would be stalling the quorum for too long while we catch up. Thus there won't be much to ack in this window. And this probably makes the acks during register-catch-up not very useful, while also requiring a master-side change in the code. I would suggest to not do that.

Part 3, the catch-up variants

• Variant iii is a breaking change, Nikita is right.
• Variant ii is rejected for the right reason, an error after a long recovery helps nobody.

What is missing is the fourth variant, as Nikita also mentioned. The one from my previous comment. A bounded wait, after which the replica registers anyway. Its worst case is today's behaviour minus the deadlock, box.cfg{} always returns with a real ID, and nothing becomes observable that is not observable today.

Part 4, the restart in the window

The lost-registration case Nikita describes. The master committed the _cluster row, the replica died before receiving it. On restart the replica is anonymous locally and registered on the master, and both doors are shut: an anonymous SUBSCRIBE is rejected with ER_PROTOCOL ("Can't subscribe an anonymous replica having an ID assigned"), and REGISTER with ER_REPLICA_NOT_ANON. Both errors are in the applier's unrecoverable branch, so the applier stops and box.cfg{} fails with that error.

Not a hang, but a dead end, and the same dead end exists today for box.cfg{replication_anon = false} if the process dies in that window.

The replica can see this case coming before it subscribes: the ballot it receives at connect time lists every UUID registered on the master (registered_replica_uuids, which replicaset_connect() already walks during a join). If its own UUID is in the join master's ballot while it is anonymous locally, there is nothing to register and nothing to catch up for: a named subscribe is accepted as is. If the replica would send REGISTER, then it would be rejected anyway though. Here I would suggest to find a way to not even send it, when we know that we want to be not anon, and we are already registered on the master. We just need to wait for this _cluster row to get replicated to us somehow.

The ending of point 4 still says that on a catch-up timeout box.cfg{} returns with the instance anonymous, which is variant iii again. With the bounded wait it registers instead.

Registration after FOLLOW

This part is about how to register the instance once the applier is in FOLLOW and not right after the snapshot fetch.

I tried addressing this before, but the box_cfg_xc() snippet from my previous comment, the one replacing today's panic, misses one thing.

Today applier_f() decides REGISTER versus SUBSCRIBE from box_is_anon() && !cfg_replication_anon, on the same connection which just fetched the snapshot. So a named replica (named in config, anon in fact) resumed to APPLIER_FOLLOW would register first and catch up second. Not what we will want.

My thinking now is that this exposes a bigger and older design problem. The applier decides an external matter there. Not only there, but here it bites right now. Specifically, it decides if we should register and when, and even looks at the config for that. Instead of being a dumb state machine which is supposed to simply fetch transactions and ack them.

This is also why your v1-4 keep trying to squeeze in new flags, like stop_on_follow, anon_catchup and the rest. My point is that this doesn't scale. At least the cognitive load doesn't scale. We must not add even a single +1 flag or a +1 new check there. It only makes things worse when a next ticket needs to touch the applier.

I suggest to think about moving the registration decision out of the applier entirely. It will register, but only if told so externally. For that we could give the applier a start parameter is_anon. The applier's owner (box.cc) will decide if we must register (is_anon = false) or not (is_anon = true).

Based on the ballot (showing whether we are registered) and this applier param the applier would make the decisions like subscribe anon, or register firstly, or we are already registered, but simply need to wait for our _cluster row.

You will then be able to drop box_is_anon(), cfg_replication_anon, and replace them with a parameter passed to the applier. Alternatively, it could be a new global flag visible to all appliers at once, but this would be more ugly I think.

Raft learner

There is another solution which I think still wasn't considered. The actual Raft learners which the ticket #6281 is about. Instances which can be non-anonymous, fully functional on their own and looking like normal replicas, but they don't affect the quorum. Non-anonymous nodes could switch between learner and normal quorum participant at will, without affecting their replica ID or rebootstrapping anything.

The effect on the quorum, I guess, means that this role needs to be persistent somehow, and committed using the other non-learner nodes (same as we treat _cluster changes now).

Instead, currently we only suggest to continue going down the path of anon replica = learner, and no other way of having the learners.

In theory our problems would be solved if we just implement Part 1 of this RFC + introduce the learners + bootstrap all instances as learners until they catch up enough (regardless of their election_mode, learner or not). This would be a new election_... or replication_synchro_... timeout/lag setting then. Which sounds right - we do this exclusively for synchro replication and leader election. Nothing else needs a late register. We above suggest to use replication_sync_lag, but it really concerns only synchro and election sub-systems.

Hm?

Solutions 2 and 3

I suggest, like before, to fold them into a drop-down, marked as rejected. Atm they only take space on the page.

Pros and cons of solution 1

Still the stale ones: "changes of iproto protocol" (there are none) and the three version scenarios (there is no version gate anymore). The real cons are the handful of tests around the removed final join, and the new catch-up option.

[mrForza]

Part 1, the 11039 fix

Thanks for the explanation, I'll keep that in mind when I do PR. In the meantime, I will remove all implementation details from my RFC.

Part 2, 6281 fix

As for anon acks, agree, my bad. As for acks in register stage I also think it will hard to implement because it requires master's side changes (we don't want to do it).

Part 4, the restart in the window

OK, I'll make these changes in the last part of my RFC.

Raft learner

I didn't quite understand why we should move away from the original decision towards Learners. What is the motivation?

[Gerold103]

Remarks on v5

Part 2 contradicts itself on anonymous ACKs. The first paragraph is now correct: the master ignores ACKs from anonymous replicas, the catch-up only makes the lag small for the moment of registration. The "when some replicas die" paragraph below it still says the replica "is already participating as an ACKer (anonymously)" before registration. That sentence has to go; a dead voter cannot be substituted by an anonymous subscriber, only by a registered one, so this case is helped exactly as much as the first one, no more.

Part 2 still describes the applier-driven variant. "applier_f disconnects that connection and reconnects on a fresh one, and only now sends REGISTER", "coordinated between applier_f", while point 4 says the admission is "driven by box.cfg itself rather than by the applier inspecting its own state". The second is the external-control idea from my previous comment (the applier registers only when told to, via a start parameter and the ballot); you did not reply to that section, so I take point 4 as the answer. Then part 2 should say the same, and the applier_f sentences go.

Part 3, the infinite wait. The four reasons hold, and reason 2 is the strongest: a replica which cannot get within replication_sync_lag of the master will never be a useful quorum member, and registering it only moves the stall from box.cfg{} to the whole cluster. If this is the choice, then the con "a replica may be in anon-mode if it can't catch up" is no longer what happens: with variant 1 the replica is not in any mode, box.cfg{} does not return.

Besides, one correction to the rejection of variant 4: "it can degrade the today's behavior" is not right. Today the replica is registered before it can ACK at all, and stays unable to ACK for the whole final join and checkpoint; after that it ACKs while lagging, exactly the state variant 4 ends in on a timeout. So variant 4's worst case is today's behaviour minus the no-ACK window, never worse. What the rejection actually says is that variant 4 is worse than variant 1 when the replica keeps lagging, and that is true. Lets reword this statement about being worse than today's behaviour. See also the learner section, under which the whole choice becomes moot.

A deferred registration can fail, and the text does not say what happens then. One more point I have just realized today.

JOIN runs three checks before it sends a single snapshot row, with a comment saying why: "fail early ... so as not to waste time on the long join process". The master must be writable, the replication user must have write access to _cluster, and the requested name must not belong to a replica which is currently connected.

In the new flow all these requirements surface only at REGISTER: after the snapshot transfer and, with part 2, after a catch-up which can take hours. The replica join then might fail or has to be reconfigured. Rebootstrap would take hours again, and it means somebody has to reconfigure it.

I somehow can't see a nice solution to this. We can't do these checks at FETCH_SNAPSHOT. First, this request is read-only data-fetch process, and it doesn't concern itself with anything else, which is good. Second, even if we do check these things early, they might become not true by the time we are caught up.

The late registration then must fail box.cfg{}, I guess. And leave the collected data in a recoverable state so a reconfig could retry from where we stopped and not have to redownload everything from scratch.

Smaller things. The rejected solution 2 still has "The master uses as stop_vclock" with the word missing, and "rebootstra". Changelog: "veriant"; part 2: "devide"; part 3: "applicer".

Raft learners, the motivation

You asked why "move away from the original decision". The thing is that we do not have yet an original decision. RFC stands for "Request For Comments", and we are discussing here which solution would be the best on multiple fronts. The one have designed so far looks promising. We also had some considered and rejected alternatives.

And just now I realized, that we didn't consider what #6281 literally says in plain test: "harness raft 'learner' feature". This RFC, as I have missed until now, didn't try to look into the Raft learners feature. We have been discussing so far the hack about delaying replica registration. So here I am trying to bring the learners back to the table, to either find why it doesn't work, or why it is worse than the delayed registration, or to realize that it might be better.

In the current RFC every difficulty in parts 2 to 4 comes from one conflation, and learners are the way to undo it rather than to work around it.

Today registration bundles three things:

1. having an ID, which a named subscribe, raft messages and writes need;
2. being counted in the synchronous quorum;
3. being counted in the election quorum and being allowed to vote.

A node which is not yet caught up must not have the second and the third. The anonymous catch-up delays all three together, because anonymous is the only "not counted" state we have, and pays for that with everything part 2 to 4 are about: an anonymous phase, a reconnect, the restart-in-window state, the lost-registration path through the ballot, the deferred identity checks, a catch-up threshold on the replica, and now an unbounded wait inside box.cfg{}. The list of crutches keeps growing.

The next crutch to come we didn't write as a 'crutch' yet. The fact that the lag config option is controlled by replicas. That seems inverted. Master is supposed to decide when it wants to accept a replica into the quorum. It doesn't look like a big difference when this option is probably configured the same on all nodes, but still it is master's authority to have the final say on such things.

A learner is a node which has the first thing from above and not the other two: an ID, a name, a _cluster row, raft messages, ACKs flowing, but it is not counted in any quorum and its vote is not counted. That is what #6281 asks for, and it is what etcd does, for example. With it the design collapses to part 1:

• The replica does FETCH_SNAPSHOT, writes its checkpoint, sends REGISTER, subscribes as named. It is registered as a learner from the first second by default. box.cfg{} returns with a real ID at once, raft is configured, the identity checks are not deferred, there is no anonymous phase, no reconnect, no restart-in-window state, no lost-registration special case, no catch-up option on the replica.
• The promotion to voter happens when the node has caught up, and the master decides that, not the replica: the relay already tracks each replica's transaction lag, so the leader compares it with a threshold and commits the role change through _cluster, the way every membership change is committed. The quorum grows at the exact moment the node can ACK, which is the whole of Harness raft 'learner' feature #6281.
• The threshold is then a setting on the master and belongs to the synchro/election subsystem, which is the only subsystem which cares about a late admission. The replica has no policy at all, and nothing waits in box.cfg{}.
• The objection to variant 4 (a lagging node enters the quorum) does not arise, since a learner is never in the quorum. Several nodes can join at once, each as a learner. And a voter which later falls far behind could be demoted by the same mechanism, which is a problem we have today with no answer at all.

What it costs, so the comparison is fair:

• A role in _cluster, one more field, added with a schema upgrade the way the name field was added in 3.0.
• The quorum arithmetic counts voters instead of registered replicas in three places: the replication_synchro_quorum formula, the election quorum and cluster size handed to raft, and the healthy quorum.
• A learner does not vote and is not a candidate: its election mode is held at off until the promotion, the way anonymous replicas are kept out of elections today, and its configured mode is applied at the promotion. A node with elections off sends no votes, so the other nodes need no filter. It still applies the PROMOTE and term rows like any replica, so it is current when promoted.
• A learner's ACK is not counted in the limbo, in the same place where anonymous ACKs are dropped today, because an ACK from a node which cannot be elected must not make a transaction committed, or a later leader could lack it.
• A mixed-version cluster would disagree on the quorum, which is the Raft violation from the drain discussion, so the role must be gated behind the schema version and activate only after the upgrade, the way instance names in _cluster are gated today (SCHEMA_FEATURE_PERSISTENT_NAMES).

So it is a bigger change than part 2, touching schema, raft and the limbo. The split I would suggest, if we would decide to follow this path: part 1 now, as the fix of #11039, which it already is on its own, and learners as the fix of #6281.

The pros list says the current flow "makes it possible to solve the problem of Raft learners". It does indeed solve one problem at the registration stage, but we see how the pile of hacks is growing, and it gets concerning whether anon replicas with delayed registration are the right way.