Skip to content

[Resharding] COPY_DATA: reliability of the data sync stage #894

@vitabaks

Description

@vitabaks

PgDog version
v0.1.36 veb5ab65

Description
Not all tables were copied:

sudo docker logs pgdog-proxy 2>&1 | grep "data sync" | grep -c started
276
sudo docker logs pgdog-proxy 2>&1 | grep "data sync" | grep -c finished
63

The issue occurs because the target shard (port 5441) was restarted during the table copy process and became temporarily unavailable. Instead of retrying the table copy, PgDog continues processing the list of tables without waiting for the database to become available.

I also think we could implement a retry mechanism: if table copying is interrupted, the target shards could be truncated and the procedure retried. This would avoid having to start over and repeatedly copy large of data.

Logs

Configuration
.

Metadata

Metadata

Assignees

Labels

bleeding edgeExciting new frontier stuff.

Type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions