Skip to content

[PHP] Share resumable file sync patch processing #581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adamziel wants to merge 2 commits into from
Closed

[PHP] Share resumable file sync patch processing #581

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
959ba60
[PHP] Share resumable file sync patch processing
adamziel Aug 13, 2026
59615be
[PHP] Keep file sync processor cursors JSON-safe
adamziel Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 16 additions & 18 deletions docs/PUSH-TERMINOLOGY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,9 @@ local relative path to a document-root-relative path.
patch result. To copy only source changes, the source snapshots from the
last sync and now are the patch base and result. The planner owns the index
diff and the active deletion roots file.
- A **file sync patch processor** builds a fresh local index, then runs a file
sync patch planner in the direction selected by its start method. Its cursor
owns both phases, so a caller can store it unchanged.
- The **pull index WAL** records completed pull mutations awaiting application
to the remote and local indexes.
- A **pull plan** lists remote absolute paths still scheduled for download or
Expand Down Expand Up @@ -306,18 +309,19 @@ The local index contains the fresh filesystem-root scan the sender saved after
a target-confirmed files-push commit, advanced path by path by later completed
files-pull mutations.
Files-pull does not scan unrelated paths or accept their pending local changes.
PushPlan diffs its fresh local index against the local index its caller
supplies. Its FileSyncPatchPlanner owns the FileIndexDiffProcessor and the
active deletion roots file. That file remembers directory deletions which
cover index paths the planner has not processed yet.
PushPlan starts FileSyncPatchProcessor with the supplied local index as the
patch base and the fresh local index as the patch result. The processor owns
FreshLocalIndexProcessor and FileSyncPatchPlanner. The planner owns
FileIndexDiffProcessor and the active deletion roots file. That file remembers
directory deletions which cover index paths the planner has not processed yet.

The PushPlan cursor is stored in `sender.json`. It contains `plan_directory`,
`local_index_file`, and the current
planning position. During `indexing`, that position contains the complete
FreshLocalIndexProcessor cursor. During `diffing`, it contains the output
offsets and the complete FileSyncPatchPlanner cursor. PushPlan stores either
nested cursor without unpacking or rebuilding it. The active
deletion roots file is append-only; each entry links to the preceding active
`local_index_file`, and the current planning position. While processing, that
position contains the complete FileSyncPatchProcessor cursor and both output
offsets. PushPlan stores the processor cursor without unpacking or rebuilding
it. The processor cursor contains the complete FreshLocalIndexProcessor cursor
while scanning and the complete FileSyncPatchPlanner cursor while planning.
The active deletion roots file is append-only; each entry links to the preceding active
directory. The exclusions have a maximum of 100 paths. The `sender.json`
phases are `creating`, `finishing_previous_commit`,
`starting_plan`, `planning`, `pushing_paths`, `pushing_deletes`, `committing`,
Expand Down Expand Up @@ -523,15 +527,9 @@ Use these names verbatim inside `PushPlan`:
| Index entry and shape | `$index_entry`, `$local_index_entry`, `$local_index_entry_shape`, `index_entry_shape()` |
| Cursor | `$cursor`, `get_cursor()` |
| Plan-owned excluded paths | `$excluded_paths_file` |
| Fresh local index processor | `FreshLocalIndexProcessor`, `$fresh_local_index_processor` |
| Fresh local indexing cursor | `fresh_local_index_cursor`, `$fresh_local_index_cursor` |
| Fresh local index byte offset | `$fresh_local_index_byte_offset` |
| Open fresh local index | `$fresh_local_index_handle` |
| Combined index bytes | `$index_bytes_total` |
| File-sync planner cursor | `file_sync_planner_cursor`, `$file_sync_planner_cursor` |
| File sync patch processor | `FileSyncPatchProcessor`, `$file_sync_patch_processor` |
| File sync patch processor cursor | `file_sync_patch_processor_cursor`, `$file_sync_patch_processor_cursor` |
| Plan progress | `get_progress()` |
| Start index diff | `start_index_diff()` |
| Seek an index file | `seek_index_file_to_byte_offset()`, `$index_file_handle` |
| Open push-plan output file | `open_push_plan_output_file_at_byte_offset()`, `$push_plan_output_file_handle` |

## Protocol names
Expand Down
116 changes: 88 additions & 28 deletions packages/reprint-client/src/lib/index/class-file-sync-patch-planner.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@
* interruption. resume() ignores those bytes.
*
* @phpstan-type IndexDiffCursor array{old_index_byte_offset:int,new_index_byte_offset:int,preceding_new_index_entry_path_b64:string|null}
* @phpstan-type Cursor array{patch_base_index_file:string,patch_result_index_file:string,active_deletion_roots_file:string,included_index_path_roots:list<string>,excluded_index_path_roots:list<string>,index_diff_cursor:IndexDiffCursor,active_deletion_root_byte_offset:int|null}
* @phpstan-type Cursor array{patch_base_index_file_b64:string,patch_result_index_file_b64:string,active_deletion_roots_file_b64:string,included_index_path_roots_b64:list<string>,excluded_index_path_roots_b64:list<string>,index_diff_cursor:IndexDiffCursor,active_deletion_root_byte_offset:int|null}
* @phpstan-type ActiveDeletionRoot array{path:string,previous_byte_offset:int|null}
* @phpstan-type ExpectedSource array{type:string,size:int,ctime:int}
* @phpstan-type DeleteOperation array{action:'delete',path:string}
Expand Down Expand Up @@ -152,11 +152,23 @@ public static function create(
}
return self::resume(
[
"patch_base_index_file" => $patch_base_index_file,
"patch_result_index_file" => $patch_result_index_file,
"active_deletion_roots_file" => $active_deletion_roots_file,
"included_index_path_roots" => $included_index_path_roots,
"excluded_index_path_roots" => $excluded_index_path_roots,
"patch_base_index_file_b64" => base64_encode(
$patch_base_index_file
),
"patch_result_index_file_b64" => base64_encode(
$patch_result_index_file
),
"active_deletion_roots_file_b64" => base64_encode(
$active_deletion_roots_file
),
"included_index_path_roots_b64" => array_map(
"base64_encode",
$included_index_path_roots
),
"excluded_index_path_roots_b64" => array_map(
"base64_encode",
$excluded_index_path_roots
),
"index_diff_cursor" => [
"old_index_byte_offset" => 0,
"new_index_byte_offset" => 0,
Expand All @@ -176,11 +188,11 @@ public static function create(
* @param array $cursor {
* Cursor returned by get_cursor().
*
* @type string $patch_base_index_file Tree state before the patch.
* @type string $patch_result_index_file Tree state described by the patch.
* @type string $active_deletion_roots_file State for active directory deletions.
* @type list<string> $included_index_path_roots Roots within which changes may be planned.
* @type list<string> $excluded_index_path_roots Roots which changes must not affect.
* @type string $patch_base_index_file_b64 Base64-encoded path to the tree state before the patch.
* @type string $patch_result_index_file_b64 Base64-encoded path to the tree state described by the patch.
* @type string $active_deletion_roots_file_b64 Base64-encoded path to the active directory-deletion state.
* @type list<string> $included_index_path_roots_b64 Base64-encoded roots within which changes may be planned.
* @type list<string> $excluded_index_path_roots_b64 Base64-encoded roots which changes must not affect.
* @type array $index_diff_cursor File-index diff cursor.
* @type int|null $active_deletion_root_byte_offset Active deletion-root offset.
* }
Expand All @@ -191,24 +203,40 @@ public static function resume(array $cursor): self
{
$planner = new self();
$planner->cursor = $cursor;
$planner->included_index_path_roots =
$cursor["included_index_path_roots"];
$planner->excluded_index_path_roots =
$cursor["excluded_index_path_roots"];
$patch_base_index_file = self::decode_cursor_path(
$cursor["patch_base_index_file_b64"],
"patch base index file"
);
$patch_result_index_file = self::decode_cursor_path(
$cursor["patch_result_index_file_b64"],
"patch result index file"
);
$active_deletion_roots_file = self::decode_cursor_path(
$cursor["active_deletion_roots_file_b64"],
"active deletion roots file"
);
$planner->included_index_path_roots = array_map(
[self::class, "decode_index_path_root"],
$cursor["included_index_path_roots_b64"]
);
$planner->excluded_index_path_roots = array_map(
[self::class, "decode_index_path_root"],
$cursor["excluded_index_path_roots_b64"]
);
$planner->index_diff = FileIndexDiffProcessor::resume(
$cursor["patch_base_index_file"],
$cursor["patch_result_index_file"],
$patch_base_index_file,
$patch_result_index_file,
$cursor["index_diff_cursor"]
);
$planner->active_deletion_roots_handle = fopen(
$cursor["active_deletion_roots_file"],
$active_deletion_roots_file,
"a+b"
);
if (!is_resource($planner->active_deletion_roots_handle)) {
$planner->index_diff->close();
throw new RuntimeException(
"Failed to open the active deletion roots file: "
. $cursor["active_deletion_roots_file"]
. $active_deletion_roots_file
);
}
$planner->active_deletion_root =
Expand Down Expand Up @@ -314,9 +342,13 @@ public function next_path(): bool
);

// Find the highest patch-base directory without a patch-result
// entry below it. Only the adjacent result entries can neighbor
// each parent in byte order, so no index rescan is needed.
$candidate_path_to_delete = $index_path;
// entry below it which path selection allows us to delete. A
// higher directory may sit outside an included root or contain an
// excluded root, so keep looking below it. Only the adjacent
// result entries can neighbor each parent in byte order, so no
// index rescan is needed.
$candidate_path_to_delete =
$this->path_may_change($index_path) ? $index_path : null;
$index_path_components = wp_unix_path_segments($index_path);
$candidate_path_components = [];
for (
Expand All @@ -335,14 +367,15 @@ public function next_path(): bool
$this->index_diff->get_preceding_path_in_new_index(),
$this->index_diff->get_following_path_in_new_index()
)
&& $this->path_may_change($candidate_path)
) {
$candidate_path_to_delete = $candidate_path;
break;
}
}
if (
!$patch_base_empty_directory_is_implied_by_patch_result_descendant
&& $this->path_may_change($candidate_path_to_delete)
&& $candidate_path_to_delete !== null
&& !$this->active_deletion_root_covers_path($index_path)
) {
$path_to_delete = $candidate_path_to_delete;
Expand Down Expand Up @@ -460,11 +493,11 @@ public function get_operation(): ?array
* @return array {
* Cursor for resume().
*
* @type string $patch_base_index_file Tree state before the patch.
* @type string $patch_result_index_file Tree state described by the patch.
* @type string $active_deletion_roots_file State for active directory deletions.
* @type list<string> $included_index_path_roots Roots within which changes may be planned.
* @type list<string> $excluded_index_path_roots Roots which changes must not affect.
* @type string $patch_base_index_file_b64 Base64-encoded path to the tree state before the patch.
* @type string $patch_result_index_file_b64 Base64-encoded path to the tree state described by the patch.
* @type string $active_deletion_roots_file_b64 Base64-encoded path to the active directory-deletion state.
* @type list<string> $included_index_path_roots_b64 Base64-encoded roots within which changes may be planned.
* @type list<string> $excluded_index_path_roots_b64 Base64-encoded roots which changes must not affect.
* @type array $index_diff_cursor File-index diff cursor.
* @type int|null $active_deletion_root_byte_offset Active deletion-root offset.
* }
Expand Down Expand Up @@ -668,6 +701,33 @@ private function path_may_change(string $index_path): bool
return true;
}

/** Decodes one arbitrary-byte path root stored in a JSON cursor. */
private static function decode_index_path_root(
string $index_path_root_b64
): string {
$index_path_root = base64_decode($index_path_root_b64, true);
if ($index_path_root === false) {
throw new InvalidArgumentException(
"File sync patch planner cursor contains an invalid base64 path root."
);
}
return $index_path_root;
}

/** Decodes one arbitrary-byte file path stored in the JSON cursor. */
private static function decode_cursor_path(
string $encoded_path,
string $field_name
): string {
$path = base64_decode($encoded_path, true);
if ($path === false) {
throw new InvalidArgumentException(
"File sync patch planner cursor contains an invalid base64 {$field_name}."
);
}
return $path;
}

/** Rejects calls after close(). */
private function assert_open(): void
{
Expand Down
Loading
Loading