[Resharding] pool is shut down

[vitabaks]

PgDog version
v0.1.36 veb5ab65

Description
We have finally completed the data sync stage for a large database, and we received an error when completing the COPY execution:

2026-04-18T02:31:14.592188Z ERROR [task: 1] pool: pool is shut down

The error in the log is not informative, it is not clear what exactly the problem is.

logical replication then did not start, the slot is not active:

db_name=# select now(), active, slot_name, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) as lag
from pg_replication_slots
where slot_type = 'logical';
              now              | active |             slot_name              |   lag   
-------------------------------+--------+------------------------------------+---------
 2026-04-18 10:25:02.026172+00 | f      | __pgdog_repl_200ydsmxvvvbtco6kfu_0 | 2855 GB
(1 row)

Perhaps this is due to the absence of PK in the two sharded tables. We used REPLICA IDENTITY FULL as a workaround, but judging by this article, this is not enough due to the rewriting of the query in upsert.

Based on this, I have a few questions:

1. Why didn’t PgDog perform pre-checks on the source before starting the data copy?
   • If a primary key is a strict requirement, it seems the process should not start without it.
2. Would it be sufficient to create the primary key on the shards after copying the schema, but before starting data replication?
   • Adding a primary key on a high-load source with tens of terabytes of data is quite challenging.
3. Is a primary key strictly required, or would a unique index be sufficient?

Logs

...
2026-04-18T02:30:44.103640Z  INFO synced 180164.520 MB for table "public"."table_name" position 8E06A/B8684DD0 [40.066 MB/sec]
2026-04-18T02:30:49.104770Z  INFO synced 180385.689 MB for table "public"."table_name" position 8E06A/B8684DD0 [44.234 MB/sec]
2026-04-18T02:30:53.089845Z DEBUG running healthcheck ";" [app_user@10.*.**.***:5441/db_name]
2026-04-18T02:30:53.089854Z DEBUG running healthcheck ";" [postgres@10.*.**.***:5434/db_name]
2026-04-18T02:30:53.089840Z DEBUG running healthcheck ";" [app_user@10.*.**.***:5442/db_name]
2026-04-18T02:30:53.089870Z DEBUG running healthcheck ";" [postgres@10.*.**.***:5440/db_name]
2026-04-18T02:30:53.089885Z DEBUG running healthcheck ";" [postgres@10.*.**.***:5441/db_name]
2026-04-18T02:30:53.089902Z DEBUG running healthcheck ";" [app_user@10.*.**.***:5440/db_name]
2026-04-18T02:30:53.089924Z DEBUG running healthcheck ";" [app_user@10.*.**.***:5434/db_name]
2026-04-18T02:30:53.089840Z DEBUG running healthcheck ";" [postgres@10.*.**.***:5442/db_name]
2026-04-18T02:30:53.090000Z DEBUG [cleanup] no cleanup needed, server in "idle" state [postgres@10.*.**.***:5441/db_name]
2026-04-18T02:30:53.090044Z DEBUG [cleanup] no cleanup needed, server in "idle" state [postgres@10.*.**.***:5442/db_name]
2026-04-18T02:30:53.090066Z DEBUG [cleanup] no cleanup needed, server in "idle" state [postgres@10.*.**.***:5434/db_name]
2026-04-18T02:30:53.090094Z DEBUG [cleanup] no cleanup needed, server in "idle" state [app_user@10.*.**.***:5440/db_name]
2026-04-18T02:30:53.090122Z DEBUG [cleanup] no cleanup needed, server in "idle" state [app_user@10.*.**.***:5442/db_name]
2026-04-18T02:30:53.090154Z DEBUG [cleanup] no cleanup needed, server in "idle" state [app_user@10.*.**.***:5434/db_name]
2026-04-18T02:30:53.091155Z DEBUG [cleanup] no cleanup needed, server in "idle" state [app_user@10.*.**.***:5441/db_name]
2026-04-18T02:30:53.125369Z DEBUG [cleanup] no cleanup needed, server in "idle" state [postgres@10.*.**.***:5440/db_name]
2026-04-18T02:30:54.105642Z  INFO synced 180726.993 MB for table "public"."table_name" position 8E06A/B8684DD0 [68.261 MB/sec]
2026-04-18T02:30:59.106681Z  INFO synced 181131.943 MB for table "public"."table_name" position 8E06A/B8684DD0 [80.990 MB/sec]
2026-04-18T02:31:04.107634Z  INFO synced 181511.931 MB for table "public"."table_name" position 8E06A/B8684DD0 [75.998 MB/sec]
2026-04-18T02:31:09.108759Z  INFO synced 181905.449 MB for table "public"."table_name" position 8E06A/B8684DD0 [78.704 MB/sec]
2026-04-18T02:31:14.110437Z  INFO synced 182238.426 MB for table "public"."table_name" position 8E06A/B8684DD0 [66.595 MB/sec]
2026-04-18T02:31:14.553397Z  INFO closing server connection [postgres@10.*.**.***:5440/db_name, state: idle, reason: other]
2026-04-18T02:31:14.553549Z  INFO closing server connection [postgres@10.*.**.***:5441/db_name, state: idle, reason: other]
2026-04-18T02:31:14.553579Z  INFO closing server connection [postgres@10.*.**.***:5442/db_name, state: idle, reason: other]
2026-04-18T02:31:14.554684Z DEBUG streaming replication on [postgres@10.*.**.***:5434/db_name]
2026-04-18T02:31:14.554691Z DEBUG replication from slot "__pgdog_l7bmggqbvfgdyaoc192yzx3q" started [postgres@10.*.**.***:5434/db_name]
2026-04-18T02:31:14.554696Z DEBUG confirmed 2498548778683856 flushed [postgres@10.*.**.***:5434/db_name]
2026-04-18T02:31:14.558873Z DEBUG slot "__pgdog_l7bmggqbvfgdyaoc192yzx3q" drained [postgres@10.*.**.***:5434/db_name]
2026-04-18T02:31:14.558888Z  INFO data sync for "public"."table_name" finished at lsn 8E06A/B8684DD0 [postgres@10.*.**.***:5434/db_name]
2026-04-18T02:31:14.558898Z  INFO closing server connection [postgres@10.*.**.***:5434/db_name, state: idle, reason: other]
2026-04-18T02:31:14.564756Z  INFO table sync for 276 tables complete [db_name, shard: 0]
2026-04-18T02:31:14.592129Z  INFO closing server connection [postgres@10.*.**.***:5434/db_name, state: idle, reason: other]
2026-04-18T02:31:14.592188Z ERROR [task: 1] pool: pool is shut down
2026-04-18T02:31:23.090132Z DEBUG running healthcheck ";" [postgres@10.*.**.***:5440/db_name]
2026-04-18T02:31:23.090136Z DEBUG running healthcheck ";" [app_user@10.*.**.***:5440/db_name]
2026-04-18T02:31:23.090234Z DEBUG running healthcheck ";" [postgres@10.*.**.***:5441/db_name]
2026-04-18T02:31:23.090142Z DEBUG running healthcheck ";" [postgres@10.*.**.***:5442/db_name]
2026-04-18T02:31:23.090161Z DEBUG running healthcheck ";" [app_user@10.*.**.***:5434/db_name]
2026-04-18T02:31:23.090178Z DEBUG running healthcheck ";" [app_user@10.*.**.***:5441/db_name]
2026-04-18T02:31:23.090190Z DEBUG running healthcheck ";" [postgres@10.*.**.***:5434/db_name]
2026-04-18T02:31:23.090466Z DEBUG [cleanup] no cleanup needed, server in "idle" state [postgres@10.*.**.***:5441/db_name]
2026-04-18T02:31:23.090471Z DEBUG [cleanup] no cleanup needed, server in "idle" state [postgres@10.*.**.***:5442/db_name]
2026-04-18T02:31:23.090478Z DEBUG [cleanup] no cleanup needed, server in "idle" state [postgres@10.*.**.***:5440/db_name]
2026-04-18T02:31:23.090484Z DEBUG [cleanup] no cleanup needed, server in "idle" state [app_user@10.*.**.***:5440/db_name]
2026-04-18T02:31:23.090219Z DEBUG running healthcheck ";" [app_user@10.*.**.***:5442/db_name]
...

Configuration
pgdog.toml

[general]
workers = 20

[[databases]]
name = "db_name"
host = "10.*.**.***"
port = 5434
role = "primary"
database_name = "db_name"

[[databases]]
name = "db_name_sharded"
database_name = "db_name"
host = "10.*.**.***"
port = 5440
role = "primary"
shard = 0

[[databases]]
name = "db_name_sharded"
database_name = "db_name"
host = "10.*.**.***"
port = 5441
role = "primary"
shard = 1

[[databases]]
name = "db_name_sharded"
database_name = "db_name"
host = "10.*.**.***"
port = 5442
role = "primary"
shard = 2

[[sharded_tables]]
database = "db_name_sharded"
column = "s_id"
data_type = "bigint"

[[sharded_tables]]
database = "db_name_sharded"
name = "table3"
column = "ss"
data_type = "varchar"

[[sharded_tables]]
database = "db_name_sharded"
name = "table4"
column = "ss"
data_type = "varchar"

[admin]
password = "******"

users.toml

[[users]]
name = "postgres"
password = "********"
database = "db_name"
schema_admin = true

[[users]]
name = "postgres"
password = "********"
database = "db_name_sharded"
schema_admin = true

[[users]]
name = "app_user"
password = "********"
database = "db_name"

[[users]]
name = "app_user"
password = "********"
database = "db_name_sharded"

[levkk]

The "pool is shutdown" error is curious; it indicates the config was reloaded (RELOAD, RECONNECT admin commands). We implemented connection pools as "immutable" rust objects. TLDR: when we reload the config, we create new pools and move connections over. This is not noticeable for clients because we do the old->new pool hand-off in the query engine code.

How are you guys running pgdog? Was this a prod deployment that you're also using for re-sharding? (e.g., by running the RESHARD admin command or COPY_DATA, etc.). I bet we don't handle reloading the config that well in the resharding code.

Perhaps this is due to the absence of PK in the two sharded tables. We used REPLICA IDENTITY FULL as a workaround, but judging by this article, this is not enough due to the rewriting of the query in upsert.

Our docs are a little bit vague on this one. The code is actually closer to reality: we only check for a replica identity, we don't care if it's a primary key or a unique index or replica identity full. So I don't think that's the root cause here.

Edit: Actually I think I'm wrong here. Replica identity full is recorded separately (not a regular identity), so we need to adjust our query.

[vitabaks]

It is unlikely that anyone could have performed the reload except me, and I did not do it. I just start the copy on Friday and checked the result on Monday.

Maybe I should attach the full log?

[levkk]

Maybe I should attach the full log?

That would be great. In the meantime, we'll add some fixes to make sure the pool is always "fresh", especially for these long-running jobs.

[vitabaks]

Was this a prod deployment that you're also using for re-sharding?

At this stage, this is a staging database with a dataset and workload close to production.

we only check for a replica identity, we don't care if it's a primary key or a unique index or replica identity full.

If you rewrite INSERT statements to UPSERT, then at a minimum a unique index is required. I’d like to try creating unique indexes on the shards after the schema is synchronized and before starting the data copy.

Maybe I should attach the full log?

I have sent the log to hi@pgdog.dev