Skip to content

[PHP] Add exact local cleanup to file sync patches #587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adamziel wants to merge 1 commit into
base: codex/share-file-index-path-inspection
Choose a base branch
from
Open

[PHP] Add exact local cleanup to file sync patches #587

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
744 changes: 744 additions & 0 deletions packages/reprint-client/src/lib/index/class-file-sync-cleanup-processor.php
View file
Open in desktop

Large diffs are not rendered by default.

290 changes: 172 additions & 118 deletions packages/reprint-client/src/lib/index/class-file-sync-patch-planner.php
View file
Open in desktop

Large diffs are not rendered by default.

83 changes: 58 additions & 25 deletions packages/reprint-client/src/lib/index/class-file-sync-patch-processor.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@
* @phpstan-type FreshIndexPosition array{phase:'indexing',file_index_cursor:FileIndexCursor,fresh_local_index_byte_offset:int}|array{phase:'sorting'}|array{phase:'complete'}
* @phpstan-type FreshIndexCursor array{fresh_local_index_file_b64:string,filesystem_root_b64:string,storage_path_b64:string,include_caches:bool,position:FreshIndexPosition}
* @phpstan-type PlannerIndexDiffCursor array{old_index_byte_offset:int,new_index_byte_offset:int,preceding_new_index_entry_path_b64:string|null}
* @phpstan-type PlannerCursor array{patch_base_index_file_b64:string,patch_result_index_file_b64:string,active_deletion_roots_file_b64:string,included_index_path_roots_b64:list<string>,excluded_index_path_roots_b64:list<string>,index_diff_cursor:PlannerIndexDiffCursor,active_deletion_root_byte_offset:int|null}
* @phpstan-type FreshTreePosition array{phase:'indexing'|'sorting'|'starting_patch',patch_base_index_file_b64:string,patch_result_index_file_b64:string,included_index_path_roots_b64:list<string>,excluded_index_path_roots_b64:list<string>,fresh_local_index_cursor:FreshIndexCursor}
* @phpstan-type PlannerCursor array{patch_base_index_file_b64:string,patch_result_index_file_b64:string,active_deletion_roots_file_b64:string,included_index_path_roots_b64:list<string>,excluded_index_path_roots_b64:list<string>,deletion_policy:'collapsed'|'exact',index_diff_cursor:PlannerIndexDiffCursor,active_deletion_root_byte_offset:int|null}
* @phpstan-type FreshTreePosition array{phase:'indexing'|'sorting'|'starting_patch',patch_base_index_file_b64:string,patch_result_index_file_b64:string,included_index_path_roots_b64:list<string>,excluded_index_path_roots_b64:list<string>,deletion_policy:'collapsed'|'exact',fresh_local_index_cursor:FreshIndexCursor}
* @phpstan-type Position FreshTreePosition|array{phase:'planning',file_sync_patch_planner_cursor:PlannerCursor}|array{phase:'complete'}
* @phpstan-type Cursor array{fresh_local_index_file_b64:string,position:Position}
* @phpstan-type SyncOperation array{action:'copy'|'delete'|'replace',path:string,expected_source?:array{type:string,size:int,ctime:int}}
* @phpstan-type SyncOperation array{action:'copy'|'delete'|'replace',path:string,expected_source?:array{type:string,size:int,ctime:int},expected_base?:array{type:string,size:int,ctime:int}}
*/
final class FileSyncPatchProcessor {
/** @var Cursor */
Expand Down Expand Up @@ -100,15 +100,18 @@ public static function start_to_fresh_local_tree(
$storage_path,
$included_index_path_roots,
$excluded_index_path_roots,
$include_caches
$include_caches,
"collapsed"
);
}

/**
* Plans the patch which changes the current local tree into a saved index.
*
* Copy and replace operations read their expected source state from the
* supplied patch-result index.
* supplied patch-result index. Deleted local entries keep their exact
* paths. A caller can remove them with unlink() or rmdir() without walking
* through skipped or excluded children.
*
* @param string $work_directory Existing directory for the fresh index and planner state.
* @param string $filesystem_root Filesystem root scanned for the fresh index.
Expand Down Expand Up @@ -136,7 +139,8 @@ public static function start_from_fresh_local_tree(
$storage_path,
$included_index_path_roots,
$excluded_index_path_roots,
$include_caches
$include_caches,
"exact"
);
}

Expand Down Expand Up @@ -241,22 +245,41 @@ public function next_step(): bool
}
$excluded_index_path_roots[] = $excluded_index_path_root;
}
$this->patch_planner = FileSyncPatchPlanner::create(
self::decode_cursor_path(
$position["patch_base_index_file_b64"],
"patch base index file"
),
self::decode_cursor_path(
$position["patch_result_index_file_b64"],
"patch result index file"
),
wp_join_unix_paths(
dirname($this->fresh_local_index_file),
"deleted_directories_stack.jsonl"
),
$included_index_path_roots,
$excluded_index_path_roots
$patch_base_index_file = self::decode_cursor_path(
$position["patch_base_index_file_b64"],
"patch base index file"
);
$patch_result_index_file = self::decode_cursor_path(
$position["patch_result_index_file_b64"],
"patch result index file"
);
$active_deletion_roots_file = wp_join_unix_paths(
dirname($this->fresh_local_index_file),
"deleted_directories_stack.jsonl"
);
if ($position["deletion_policy"] === "exact") {
$this->patch_planner =
FileSyncPatchPlanner::create_with_exact_deletions(
$patch_base_index_file,
$patch_result_index_file,
$active_deletion_roots_file,
$included_index_path_roots,
$excluded_index_path_roots
);
} elseif ($position["deletion_policy"] === "collapsed") {
$this->patch_planner = FileSyncPatchPlanner::create(
$patch_base_index_file,
$patch_result_index_file,
$active_deletion_roots_file,
$included_index_path_roots,
$excluded_index_path_roots
);
} else {
throw new InvalidArgumentException(
"File sync patch processor cursor has an invalid deletion policy: "
. $position["deletion_policy"]
);
}
$this->cursor["position"] = [
"phase" => "planning",
"file_sync_patch_planner_cursor" =>
Expand Down Expand Up @@ -285,9 +308,10 @@ public function next_step(): bool
/**
* Returns the operation selected by the latest planning step.
*
* Delete operations contain only `action` and `path`. Copy and replace
* operations also contain the result index entry which must be copied.
* Non-planning steps and processed paths which need no change return null.
* Copy and replace operations contain the result index entry which must
* be copied. Exact delete and replace operations also contain the base
* entry which will be removed. Non-planning steps and processed paths
* which need no change return null.
*
* @return array|null {
* @type string $action `copy`, `delete`, or `replace`.
Expand All @@ -299,6 +323,13 @@ public function next_step(): bool
* @type int $size Expected size.
* @type int $ctime Expected inode change time.
* }
* @type array $expected_base {
* Base index entry removed by an exact `delete` or `replace`.
*
* @type string $type Expected `file`, `link`, or `dir` type.
* @type int $size Expected size.
* @type int $ctime Expected inode change time.
* }
* }
* @phpstan-return SyncOperation|null
*/
Expand Down Expand Up @@ -416,7 +447,8 @@ private static function create(
string $storage_path,
array $included_index_path_roots,
array $excluded_index_path_roots,
bool $include_caches
bool $include_caches,
string $deletion_policy
): self {
if (!is_dir($work_directory)) {
throw new LogicException(
Expand Down Expand Up @@ -456,6 +488,7 @@ private static function create(
"base64_encode",
$excluded_index_path_roots
),
"deletion_policy" => $deletion_policy,
"fresh_local_index_cursor" =>
$processor->fresh_local_index_processor->get_cursor(),
],
Expand Down
95 changes: 95 additions & 0 deletions packages/reprint-client/src/lib/index/file-sync-path-functions.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

namespace Reprint\Importer;

// phpcs:disable WordPress.Security.EscapeOutput.ExceptionNotEscaped -- Index paths and byte offsets are CLI values, never HTML output.

use function WordPress\Filesystem\wp_join_unix_paths;
use function WordPress\Filesystem\wp_unix_path_segments;
use function WordPress\Reprint\Exporter\path_is_same_as_or_descendant_of;
Expand Down Expand Up @@ -144,3 +146,96 @@ function file_sync_result_contains_path_or_descendant(
$index_path
);
}

/**
* Appends one path to a linked stack stored in a file.
*
* Each entry points to the preceding stack entry by byte offset. Entries from
* an interrupted step can remain at the end of the file without joining the
* saved stack. A caller which needs to reclaim those bytes may truncate the
* file to its separately stored output byte offset before appending again.
*
* @param resource $stack_handle Open file used by this stack.
* @param string $path Arbitrary-byte path to push.
* @param int|null $previous_byte_offset Byte offset of the preceding entry.
* @param int|null $expected_ctime Expected directory ctime, when the
* caller will remove this path later.
* @return int Byte offset of the appended entry.
*/
function append_file_sync_path_stack_entry(
$stack_handle,
string $path,
?int $previous_byte_offset,
?int $expected_ctime = null
): int {
if (fseek($stack_handle, 0, SEEK_END) !== 0) {
throw new \RuntimeException(
"Failed to seek to the end of the file sync path stack."
);
}
$byte_offset = ftell($stack_handle);
if (!is_int($byte_offset)) {
throw new \RuntimeException(
"Failed to determine the file sync path stack byte offset."
);
}
$entry = [
"path_b64" => base64_encode($path),
"previous_byte_offset" => $previous_byte_offset,
"expected_ctime" => $expected_ctime,
];
$line = json_encode(
$entry,
JSON_UNESCAPED_SLASHES | JSON_THROW_ON_ERROR
) . "\n";
if (fwrite($stack_handle, $line) !== strlen($line)) {
throw new \RuntimeException(
"Failed to append to the file sync path stack."
);
}
return $byte_offset;
}

/**
* Reads one linked path-stack entry at its byte offset.
*
* @param resource $stack_handle Open file used by this stack.
* @return array{path:string,previous_byte_offset:int|null,expected_ctime:int|null} Decoded stack entry.
*/
function read_file_sync_path_stack_entry(
$stack_handle,
int $byte_offset
): array {
if (fseek($stack_handle, $byte_offset) !== 0) {
throw new \RuntimeException(
"Failed to seek to file sync path stack byte {$byte_offset}."
);
}
$line = fgets($stack_handle);
if (!is_string($line)) {
throw new \RuntimeException(
"Failed to read the file sync path stack at byte {$byte_offset}."
);
}
try {
$entry = json_decode($line, true, 512, JSON_THROW_ON_ERROR);
} catch (\JsonException $exception) {
throw new \RuntimeException(
"Failed to decode the file sync path stack at byte {$byte_offset}.",
0,
$exception
);
}
/** @var array{path_b64:string,previous_byte_offset:int|null,expected_ctime:int|null} $entry */
$path = base64_decode($entry["path_b64"], true);
if ($path === false) {
throw new \RuntimeException(
"Failed to decode the file sync path at byte {$byte_offset}."
);
}
return [
"path" => $path,
"previous_byte_offset" => $entry["previous_byte_offset"],
"expected_ctime" => $entry["expected_ctime"],
];
}
Loading
Loading