Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .agent/rules/consensus-and-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -2201,6 +2201,40 @@ right, because the wire now trusts it.
point opened as itself), never through what they point at, and the node writes none. The histories answer
`StartWithout` -- no state of a history file may keep a node from starting -- and are set
aside where they are read (`SetAsideForeignHistory`): not read, never written over, WARNED.
**Whose files count as the node's own is a SET, and it depends on WHO ASKS AND WHAT FOR** (owner
ruling, 2026-10-06; limited by the controller to verbs that write nothing). Ownership is judged
from the account asking, so an elevated `--print-surfaces` over the running service's own
directory met every file the service wrote as "another account's" -- MEASURED on a healthy node,
rc=2, `mode: none` -- while the service judged the same files its own. An ADMINISTRATOR
(`IsAdministratorProcess`) running a verb that only INSPECTS the directory (`StateUseRows`, a
COLUMN: `--print-surfaces` alone; `--print-identity` MINTS an absent key and id, so it `Runs`, as
does anything else) counts a file -- and the
directory -- owned by the node's own per-service virtual account, `NT SERVICE\<--service-name>`
(`NodeServiceAccountOf`, off the `NodeWindowsLogon` the registration uses), as the node's. The
argument is NOT "only that account can create one": a holder of the restore privilege can set
any owner -- it is that whoever can set this owner could plant an Administrators-owned file
anyway, which is trusted already. **And the direction is an elevated process reading what a
deprivileged, network-facing account wrote**, the way an escalation travels -- accepted for a
verb that writes nothing, and for nothing that RUNS: a start, foreground or supervised, and an
install keep the strict set, so a second process never writes into a live service's directory
on the strength of it. One row per (caller, use) pair (`OwnStateAccountRows`), one decorator
every reader of the directory in `main` takes (`OwnStateAccountsGuard`, a directory's owner
included through `WritersBeyondOwner`), compared by SID (`FileOwner::id`, `AccountIdOf`) and
never by name, and only a PER-SERVICE SID (`IsPerServiceSid`: `S-1-5-80-0` is the group `ALL
SERVICES`). **Blind spot, and the direction it fails in: OPEN, for an administrator alone** -- a
per-service SID is not checked to be THIS node's service (no `QueryServiceConfig` binary-path
check), so an administrator who deliberately types `--service-name=<another service>` has
`--print-surfaces` trust that service's files. Accepted: it takes an administrator naming another
service on the command line (`--service-name` is argv-only), and `--print-surfaces` writes nothing. An empty `--service-name` names no account; an account that does not resolve trusts
no file. **POSIX is unchanged**: `AccountIdOf` resolves nothing there, so `sudo --print-surfaces`
over a dedicated service user's directory is still refused; extending it would be a row naming
that user (`getpwnam`) and a uid comparison, nothing else. **A refusal names the owner and the
SEAT that would judge it as the service**: removing or moving a file or the directory aside is
the remedy for a PLANTED one only, never for one merely owned by another account. And a reading
that refused ends a start, and `--print-surfaces`, ONCE and FIRST by its own ARM
(`FormationUnread::stage`, `UnreadStateStage`): an I/O arm is `Failed` and retried by the
supervisor, a verdict `Refused`; the startup row (`StateDirectoryUnreadRefusal`) claims only the
permanent ones. AGENT.md tripwires it in one bullet, beside the state-directory one.
**The table is the ONLY author of a state file's name**: `StateFile` (`Core/StateFiles.hpp`,
header-only because four layers write these files) names every one, every writer's constant
is a lookup into it, and the node's table is an `EnumTable` keyed by the same enum -- so a new
Expand Down
8 changes: 4 additions & 4 deletions AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -304,16 +304,14 @@ launcher's cache key is made of. Before `apps/fastcache-cc/`, `CompileCache/`.
pasted `RunsWorker(c) &&`; the merged listener reads `FamilyRoutes`.
- What a node holds back from compiles is what its tier **built**, never what a flag asked for, so
capacity is derived *below* the tier startup. Which tiers cost RAM is a `StorageTierTable`
column, a present zero is *unbounded*, and a disk tier's key index is **a working set rather
than a store** (`storage.md`).
column, a present zero is *unbounded*, and a disk tier's key index is **a working set rather than a store** (`storage.md`).
- A node SERVES while it identifies its toolchains: the cheap half at startup, the walk on the
heartbeat thread's first round, registering NOTHING until the fingerprint is real;
`ToolchainSurvey` travels beside the map with a deleted default constructor.
- A probe that did not RUN is not one that answered nothing, and an identity built on one is
neither served nor cached: the guard is `exitCode == NotSpawned`, never the count.
- A reply's codec is chosen from what the OTHER end said it accepts, never from this end's list
against itself — `AvailableCodecs()`, never a literal — and a test must separate the two ends
**disagreeing**.
against itself — `AvailableCodecs()`, never a literal — and a test must separate the two ends **disagreeing**.
- A stored value's text regions are canonicalized by **every** server on this wire, through the
one `CanonicalStoredValue` beside `CompileValue`; retirement is a schema bump, never a sniff.
- And by every **VERSION** of them, or the rule holds at no moment a fleet is actually in.
Expand Down Expand Up @@ -514,6 +512,8 @@ launcher's cache key is made of. Before `apps/fastcache-cc/`, `CompileCache/`.
forever, never derived from the machine** — not the hostname, not the OS machine-id even as a
SEED. `--node-id` overrides and is recorded, `--raft-self=<host>` states the address, and the
resolved value reaches every configuration this process builds.
- **An administrator counts the node's OWN service account's state files as the node's only in a verb that WRITES
NOTHING** (`StateUseRows`, `OwnStateAccountRows`); a start keeps the strict set, and no remedy removes for an owner alone.
- **The hostname is a fleet-page LABEL and decides nothing**, and it still passes the UTF-8 gate
at `SchedulerService::Register`. **No prefix matching on ids.**
- **A replicated setting must not decide where a node sends a CREDENTIAL** — refused BY NAME
Expand Down
23 changes: 20 additions & 3 deletions docs/tools/fastcache-compile-node.md
Original file line number Diff line number Diff line change
Expand Up @@ -1389,8 +1389,21 @@ The same holds for every file the node keeps in its state directory: the id, the
formation record, the consensus store (`raft-state`, `raft-log`, `raft-snapshot`), the
remembered cluster endpoints (`fleet-endpoints`) and the roster. One owned by another
account refuses the start, naming the file, what it holds and its owner, and is left as it
is -- remove it, or move the consensus store aside, as the refusal says. So does any entry
the node does not keep. The history files are the exception, because a history may never
is. Ownership is judged from the account asking. On Windows an elevated administrator
running `--print-surfaces`, which writes nothing, counts the
files and the directory of the node's own service account, `NT SERVICE\<--service-name>`, as
the node's own, so `--print-surfaces` over a running service's directory judges the line as
the service will run it. A start, foreground or as a service, `--print-identity` (which mints a
key and an id where they are absent) and `--install-service` keep the strict rule: a foreground
start by an administrator over the service's files is refused, as before. On POSIX nothing changes: root does not count a service user's files as the node's,
so `sudo fastcache-compile-node --print-surfaces` over a dedicated user's directory is still
refused -- judge it as that user. A file of any other account is refused; judge it as the
service it belongs to, and remove it, or move the consensus store aside, only when nobody but
this node should have written it. A directory that cannot be read ends the command, or the
start, once and with that reason, before any setting is judged against a mode the node could
not read: a read that failed (a listing refused, an access list that would not answer) exits
1 and is retried by the service manager, a verdict on what was found exits 78. Any entry the
node does not keep is refused too. The history files are the exception, because a history may never
keep a node from starting: one owned by another account is set aside with a warning, not
read and not written over, and the node keeps no history there until it is removed. The
key is the only file there created readable by its owner and the administrators alone;
Expand Down Expand Up @@ -1484,7 +1497,11 @@ nobody reads:
| `--listen-raft` names a usable port | That is where every peer dials it. A value that is not an address is refused with the text you typed. |

The reverse holds too: `--raft-self` **without** consensus is refused rather than ignored,
since there is no port to pair the host with and nothing would say so. (`--node-id` and
since there is no port to pair the host with and nothing would say so. Whether a node runs
consensus is its mode, kept in its state directory, so the refusal names one of two causes:
an empty `--listen-raft=` that closed the port its mode opens, or a state directory whose
formation record could not be read -- the mode is then unknown, and the refusal carries the
reading's own refusal, the same reason `--print-surfaces` prints on its `mode: none` line. (`--node-id` and
`--cluster-dir` are not refused there — the node keeps its identity in that directory,
and the dashboard its history file, so a node with no consensus still has a use for
both.)
Expand Down
12 changes: 7 additions & 5 deletions packaging/config/fastcache-compile-node.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -165,13 +165,15 @@
# this node's identity in the cluster, and what every vote is counted
# against. An OVERRIDE: a node with a consensus port mints one into
# --cluster-dir on its first start and reads it back forever, so this is only
# needed for a fleet already named by hand. --listen-raft is what turns
# consensus on; this names the node that runs it.
# needed for a fleet already named by hand. Its mode, kept in --cluster-dir,
# decides whether it runs consensus; this names the node that runs it.
#node_id:

# where peers reach this node's consensus port; default 6680. The port is what
# turns consensus ON, so a node with no settings is a cluster of one, and an
# empty value runs none -- on a cache-only node (slots: 0) alone: a worker with
# where peers reach this node's consensus port; default 6680. The mode kept in
# --cluster-dir decides whether the node runs consensus, and every mode but a
# learner's opens this port: a node with no settings is a cluster of one, and an
# empty value closes the port and runs none -- on a cache-only node (slots: 0)
# alone: a worker with
# no consensus never joins a fleet, so it would register nowhere and is refused
# at startup. A bare port binds the WILDCARD: peers are on other
# machines by definition, so loopback would silently not work.
Expand Down
70 changes: 61 additions & 9 deletions src/FastCache/Platform/FileTrust.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -560,6 +560,18 @@ namespace
return Utf8FromWideText(text).value_or("an account this machine cannot name");
}

/// @p sid in its string form (`S-1-5-...`), which is what `FileOwner::id` compares.
/// @param sid The SID.
/// @return Its string form, or empty when it cannot be spelled.
[[nodiscard]] std::string SidTextOf(PSID sid)
{
wchar_t* text = nullptr;
if (::ConvertSidToStringSidW(sid, &text) == FALSE)
return {};
auto const owned = LocalBlock { text };
return Utf8FromWideText(text).value_or(std::string {});
}

/// Build a security descriptor from @p sddl, for a create that applies it from the first instant.
/// @param sddl The access list.
/// @return The descriptor, owned, or the error that refused it.
Expand Down Expand Up @@ -1044,23 +1056,50 @@ FileOwner FileOwnerOf(std::filesystem::path const& path)
return FileOwner {};

auto name = AccountNameOf(owner);
auto id = SidTextOf(owner);
if (IsThisProcessUser(owner))
return FileOwner { .standing = FileOwnerStanding::ThisProcess, .name = std::move(name) };
return FileOwner { .standing = FileOwnerStanding::ThisProcess, .name = std::move(name), .id = std::move(id) };
if (MatchesWellKnownSid(owner, AdministrativeOwners))
return FileOwner { .standing = FileOwnerStanding::Administrative, .name = std::move(name) };
return FileOwner { .standing = FileOwnerStanding::Another, .name = std::move(name) };
return FileOwner { .standing = FileOwnerStanding::Administrative, .name = std::move(name), .id = std::move(id) };
return FileOwner { .standing = FileOwnerStanding::Another, .name = std::move(name), .id = std::move(id) };
#else
struct ::stat info {};

// `lstat`: the entry, never what a link points at.
if (::lstat(path.c_str(), &info) != 0)
return FileOwner {};
auto name = std::format("uid {}", info.st_uid);
auto id = name;
if (info.st_uid == ::geteuid())
return FileOwner { .standing = FileOwnerStanding::ThisProcess, .name = std::move(name) };
return FileOwner { .standing = FileOwnerStanding::ThisProcess, .name = std::move(name), .id = std::move(id) };
if (info.st_uid == 0)
return FileOwner { .standing = FileOwnerStanding::Administrative, .name = std::move(name) };
return FileOwner { .standing = FileOwnerStanding::Another, .name = std::move(name) };
return FileOwner { .standing = FileOwnerStanding::Administrative, .name = std::move(name), .id = std::move(id) };
return FileOwner { .standing = FileOwnerStanding::Another, .name = std::move(name), .id = std::move(id) };
#endif
}

std::optional<std::string> AccountIdOf(std::string const& account)
{
#if defined(_WIN32)
std::array<std::byte, SECURITY_MAX_SID_SIZE> sid {};
if (account.empty() || !ResolveAccountSid(account, sid))
return std::nullopt;
auto text = SidTextOf(sid.data());
if (text.empty())
return std::nullopt;
return text;
#else
(void) account;
return std::nullopt;
#endif
}

bool IsAdministratorProcess()
{
#if defined(_WIN32)
return EffectiveTokenIsIn(WinBuiltinAdministratorsSid);
#else
return ::geteuid() == 0;
#endif
}

Expand All @@ -1085,12 +1124,20 @@ DirectoryWriters DirectoryWritersOf(std::filesystem::path const& directory)
case FileOwnerStanding::Undetermined:
case FileOwnerStanding::Last:
return DirectoryWriters::Undetermined;
// `NamedAccount` is never answered by `FileOwnerOf`, so it cannot reach here; were it to, a
// directory is judged as its owner left it, never by a caller's trust in an account.
case FileOwnerStanding::NamedAccount:
case FileOwnerStanding::Another:
return DirectoryWriters::ForeignOwner;
case FileOwnerStanding::ThisProcess:
case FileOwnerStanding::Administrative:
break;
}
return DirectoryWritersBeyondOwner(directory);
}

DirectoryWriters DirectoryWritersBeyondOwner(std::filesystem::path const& directory)
{
#if defined(_WIN32)
auto const nobody = NoBroadPrincipalMay(directory, EntryPlantingRights);
if (!nobody.has_value())
Expand Down Expand Up @@ -1256,9 +1303,14 @@ std::string DirectoryWritersHint(std::filesystem::path const& directory, Directo
case DirectoryWriters::Undetermined:
return std::format("could not determine who may create or delete entries in {}", directory.string());
case DirectoryWriters::ForeignOwner:
return std::format("{} is owned by an account that is neither this process's nor an administrator's, which "
"can grant itself anything in it: another account created it. Remove it, with what is "
"in it, and this node creates its own, its owner's alone",
// Never "remove it" for its owner alone: the account a service runs as owns the
// directory that service created, and a judgement made from another seat says only
// that the seat differs. Removal is the remedy for a planted directory.
return std::format("{} is owned by an account that is neither this process's nor an administrator's, and "
"an owner can grant itself anything in it. Ownership is judged from the account asking: "
"if that account is the one the service using it runs as, judge it as that account. "
"Only a directory nobody else should have created -- a planted one -- is removed, with "
"what is in it",
directory.string());
case DirectoryWriters::Others: {
auto commands = std::string {};
Expand Down
Loading
Loading