Repository navigation
Discovery: services from ports, self-improving names, and symmetric identity - #348
Merged
Merged
Conversation
A sweep of a multi-VLAN homelab produced 26 anonymous "host-<hash>" cards out of 27: ARP is link-local so remote subnets yield no MAC, and few home networks have PTR records. The sweep found everything and recognised nothing. Three sources close most of that gap without a new dependency. Service banners. Once a host is known alive it is asked what it is: a TLS certificate's common name (443/8006/8443), an HTTP page title or Server header, then an SSH greeting. This is the identification half of `nmap -sV` in about 150 lines of BCL sockets. Only open ports are asked — a handshake with a closed port buys nothing but a timeout. The three answers are different claims, so they are treated differently. An X.509 common name is a host name by construction, so a certificate names the machine. A page title names an application, and a host may run several, so each becomes a Service resource hanging off the card rather than renaming it. An SSH greeting names only the daemon — every Linux box on a subnet answers "OpenSSH" — so it annotates and never names. An appliance's own management page is not a service on itself, so a title matching the host's own name is skipped. Open ports. Liveness stops at the first answer, which left pingable hosts with no port evidence at all. A living host is now checked against a wider list (554 cameras, 1883 brokers, 8123, 9000, 3000, 32400 and friends) and the result lands in an open-ports label. These are observations, not conclusions: "554 is open" is a fact, "this is a camera" is an inference the reader is better placed to draw, and a port number is a convention rather than a guarantee — so nothing is named from one. They are, however, the best possible targets for the banner probes. MAC vendor. Where a MAC is known the card gains the organisation IEEE assigned that OUI to, from a curated 10,868-entry subset of the registry generated by generate-oui-table.py. A hypervisor's own prefix beats the locally-administered bit so a KVM guest reads as QEMU/KVM rather than anonymous, while an address a phone invented for itself is reported as randomised instead of attributed to whoever owns the block. Services are also anchored by address: one whose runsOn resolves to nothing takes the system at its own IP, which fixes the dangling link a docker collector leaves when it can only guess its host's name. Narrow by design — only an unresolvable link, only when exactly one system claims that address, and never for systems, where a shared address means the same machine rather than a parent. Cards stay sparse and identity stays put: nothing learned here writes an OS or a core count, and a name from a service never moves a discoveryId. --no-identify restores a pure liveness sweep. All names, addresses and MAC suffixes in tests and docs are invented; the OUI prefixes are public IEEE registry data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Proxmox only records a guest's address when someone set one statically, so on a DHCP estate every guest arrived with no address at all. That cost more than the field: a network sweep of another subnet has no ARP entry to work from, so it identifies a host by address alone — and with the guests carrying no address there was nothing to match, leaving a second, emptier card beside every guest the hypervisor had already described in full. Guests are now asked where they are. A VM answers through its qemu-guest-agent, a container through its running interfaces; a stopped guest is never asked, since it has nothing to report and the call is one round trip per guest. Choosing among the answers is the hard part. A guest that runs containers reports several interfaces — docker0 and its per-network bridges, Home Assistant's hassio, any VPN tunnel — and recording 172.17.0.1 as the machine's address would be worse than recording nothing, because every container host on the estate reports the same one. The NIC MACs Proxmox assigned are the discriminator: an interface carrying one is a NIC the hypervisor gave the guest, anything else the guest invented. Where the MACs cannot be read, nothing is claimed. A statically configured address still wins, being what the administrator asked for. With addresses in hand a sweep's find can be matched to the guest it actually is, so the resolver gains an address bridge beside the MAC one. Narrow on purpose: only a scan-grade card that produced no MAC of its own, only against an agent-grade card of the same kind, and only when exactly one stored system claims that address. Services anchor to the stored card in preference to a sweep's stand-in for the same reason. On a live two-node estate this turned nine scanned cards on the guest subnet into six guests updated in place and two genuinely unknown hosts, with the discovered web services landing on the real VMs rather than on stand-ins. The read orchestration moves into the domain on the way past. The CLI and the MCP tool each had a copy, and the copies had already drifted — the MCP one dropped the guests' MACs, silently costing every guest its chance of unifying with a scan. Now there is one, and it is tested. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A Proxmox number written as a string took the whole run down. Its perl backend quotes numeric fields inconsistently — "vmid":"100" and "maxdisk":"512110190592" both turn up across versions — and the TryGetInt32 family does not return false for a string, it throws. The kind is now checked first and a quoted number parsed, which is what the caller meant either way. Before this, one such field produced "Unexpected error occurred" plus a stack trace on the CLI, and MCP forwarded the BCL's "requires an element of type 'Number'" as though the user had done something wrong. An ssh:// or npipe:// DOCKER_HOST crashed. `docker context` sets the first for a remote host and the second is the Windows default, so both are ordinary values to find in the environment. HttpClient accepts either URI and only throws NotSupportedException on the first request, which was past the command's catch list. They are now refused where the client is built, through the UriFormatException both front ends already turn into "not a usable Docker endpoint" — one phrasing for a bad endpoint rather than two — and the message says how to forward the socket instead. A remote engine with no IPv4 address had its services recorded at this machine's address. Services are recorded at their host's address and the inventory holds IPv4 only, so when the endpoint resolves to IPv6 alone the address is genuinely unknown; substituting whatever machine ran the command gave every service a confident, wrong address that then flowed into the ansible, ssh and hosts exports. There is no fallback now, and the run stops with an explanation. Each fix has tests that fail without it: quoted numbers across the guest list, both unreachable endpoint schemes, and an engine answering on an IPv6-only socket. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The collector for everything a sweep can see but not identify. ARP is link-local. Sweeping from one machine yields a MAC for that machine's own segment and nothing but an address for every other subnet — and an address alone cannot survive a DHCP re-lease or be matched against anything already documented. The firewall routes every subnet, so its neighbour table carries the MAC for all of them, the name it handed out, and which leg each machine answered on. Cards are seeded exactly as the network sweep seeds its own, keyed on the MAC, because both describe the same thing by the same evidence: a machine observed on the network rather than asked about itself. So a host the firewall knows and a host a sweep found are one card, whichever ran first, with no special case anywhere to say so. On a live estate this turned 21 resources into 13 updated in place and 8 machines no sweep had ever seen — devices that answer no port and no ping but sit in the firewall's table. What it leaves out, on purpose: the firewall's own addresses (every routed subnet contributes one and they are all the same box, which is a Firewall rather than the handful of Systems this would invent), entries that have aged out, broadcast and multicast, and neighbours on public addresses. That last one is the ISP's equipment on the WAN leg — not the user's infrastructure, and recording it would put a public address into a file people commit. --include-public asks for them. Tolerant of the endpoint rename in OPNsense 25.7, which moved these actions from camelCase to snake_case and broke integrations pinned to either spelling; both are tried. Both response shapes are read too, since the search wrapper wraps the same rows in an object. A key that lacks the Diagnostics: ARP Table privilege gets a redirect to the login page rather than a 401, so that is reported as the permission problem it is. Verified against a live two-firewall estate; the fixtures and tests use invented addresses and MAC suffixes throughout. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An open port was landing in an `open-ports` label: a comma-separated string nothing could link to, filter on, or hang a note from. RackPeek already has a resource for "a thing listening on an address and a port", so each open port now becomes a Service with its own stable id, running on the host that serves it — nebula-ssh, nebula-https, nebula-proxmox. What a service said about itself still beats what its port number implies, because a port is a convention and an answer is evidence. Where nothing answered the name falls back to the host plus the port's usual service, and an unrecognised port keeps its number as nebula-tcp-9987 rather than guessing. An appliance whose management page only says its own name back is named for the port instead, so a firewall called opnsense gains an opnsense-https rather than a second card called opnsense. Two labels change with it. `vendor` becomes `nic-vendor`, because the OUI identifies whoever owns the network interface, which is not the same claim as who made the machine — a Proxmox guest's virtual NIC reads Proxmox while the box underneath it is a Dell. And `segment` goes: it described the firewall's wiring rather than the machine, and changed whenever anything was re-cabled. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A collector names a card from whatever it could see, and when it could see nothing the name falls back to a slug of the card's own id — host-1a2b3c4d says only that something is there. A later run, or a collector that can see more, often does know the machine's name: a firewall knows what it handed out over DHCP, a hypervisor knows what its guest is called. Those placeholders now give way to real names, and everything pointing at the old name follows: stored runsOn, stored connections, services named after their host, and the incoming payload's own references. A new `userNamed` flag draws the line discovery must not cross. Renaming sets it, every rename path goes through one use case, and nothing in discovery touches the name again. A resource with no discoveryId is user-named whatever the flag says — nothing but a person could have written it. The upgrade is one-way, placeholder to real, and one real name never replaces another, or two collectors that each knew a different name would rename a box back and forth on every run. The address bridge is now symmetric. It fired only when the incoming card was the scan and the stored one was agent-grade, so sweeping before running the hypervisor left both cards behind where the other order produced one. Which collector ran first is an accident of what someone typed and must not decide what the inventory holds. Two scan cards can bridge as well, but only when exactly one of them saw a MAC: a firewall's neighbour table names the interface answering at an address where a sweep of a subnet it does not sit on only knows something replied, and those are not two stand-ins. Cards of equal standing never bridge. Measured on a nine-subnet lab, sweeping first: fourteen machines were in the inventory twice, and are now in it once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bugs, one cause: three places each decided for themselves what a service's URL was, and all three were wrong. The card derived every link as http:// regardless of the port, so SSH on 22 rendered as http://10.0.0.5:22/ — a link that looks real, invites a click and cannot load. It was reading Network.Protocol as though it named a scheme, but discovery writes the transport there (TCP), which says nothing about what rides on top of it. The dependency trees had it worse: they fed NetworkString() into an href, and that was display text — "Ip: 10.0.0.5:3000", prefix and trailing space included — so the link was dead on arrival. Both rules now live in one place. A scheme is only inferred where the convention is genuinely a web one; ssh, smb, mqtt, dns and anything uncurated get no link at all, because a dead link is worse than none. A URL someone typed by hand still always wins, and an explicit http or https in the protocol field is still believed, which is the only thing that can answer on an uncurated port. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BrowsableUrl built its result with UriBuilder, which throws UriFormatException on a host it cannot parse — "a/b", a name with spaces, a bare run of dots. The address field holds whatever a person typed or a collector read off a device, and neither is obliged to produce something a URL can be built from. That throw was already latent on the service card, where it would have spoiled one card. It matters more now: the dependency trees call this for every service they render, so a single malformed address would blank the entire hardware or system page instead of costing one link. A missing link costs a click. Letting this escape costs the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on the network-host-identity work with four changes to how discovery names and identifies things, plus a UI fix.
Every open port becomes a Service
The
open-portslabel is gone. It was a comma-separated string nothing could link to, filter on, or hang a note from, and RackPeek already has a resource for "a thing listening on an address and a port". Each open port now gets its own Service with a stable id, running on its host.What a service announced about itself still beats what its port implies. Where nothing answered, the name falls back to host plus the port's usual service, and an unrecognised port keeps its number rather than guessing.
Labels named for what they are
vendorbecomesnic-vendor— the OUI identifies whoever owns the network interface, which is not the same claim as who made the machine.segmentis dropped: it described the firewall's wiring rather than the machine.Neither label has ever been in a release, so no migration is needed.
Names improve until a person chooses one
A placeholder like
host-1a2b3c4dnow gives way to a real name when a collector turns up knowing it, and every reference follows — storedrunsOn, stored connections, and services named after their host. A newuserNamedflag records a human choice and discovery never touches the name again.The upgrade is one-way, and one real name never replaces another, or two collectors knowing different names would rename a box back and forth every run.
The address bridge is symmetric
It previously fired only when the incoming card was the scan and the stored one was agent-grade — so sweeping before running the hypervisor left duplicates where the other order produced one card. Two scan cards can now bridge too, but only when exactly one of them saw a MAC.
Measured on a nine-subnet lab, sweeping first: 14 machines were in the inventory twice, and are now in it once.
Service links that work
Every service link was derived as
http://regardless of port, so SSH on 22 rendered ashttp://host:22/. The dependency trees were worse — they fed display text (Ip: 10.0.0.5:3000) straight into anhref. Both rules now live in one place, and ports a browser cannot open get no link at all.Testing
64 new tests (Discovery 475, MCP 71, main 378, E2E 106). Each new behaviour has a test proven to fail without its change. Verified end to end against a live nine-subnet lab.
🤖 Generated with Claude Code