Support git remote-helper schemes (entire://) as a transport#96
Open
Soph wants to merge 5 commits into
Open
Conversation
Implement a gitproto.Conn backed by a git-remote-<scheme> binary, bridging git-sync's smart transport onto the helper's stateless-connect capability. This lets git-sync drive URL schemes it has no native transport for (e.g. entire://) by delegating auth and network I/O to the helper while still running the wire protocol itself. Each Conn operation spawns its own helper process: the helper services one stateless-connect session per process, and its receive-pack path reads the pushed pack until stdin EOF, so a fresh process per RPC is the simplest correct model and naturally supports batched (multi-request) pushes. The v2 upload-pack response is framed with a trailing response-end (0002) packet, which is stripped so the consumer sees exactly the byte stream it would over HTTP; receive-pack responses run to EOF. The helper strips the smart-HTTP "# service=" banner, so advertisements match the bannerless SSH shape that the existing decoders already accept. Tests drive a fake helper via the os/exec re-exec pattern, covering the v2 advertisement, response-end stripping, receive-pack read-to-EOF, and the fallback path, plus unit tests for the pkt-line framing helpers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Entire-Checkpoint: 1be8d21b867a
newConn now checks for a git-remote-<scheme> binary when the endpoint scheme isn't one git-sync handles natively, and builds a HelperConn when one exists. http/https are excluded so they always use the optimized native HTTP transport (git ships its own git-remote-http(s)); ssh is already handled above. This is what makes entire:// URLs work, given git-remote-entire on PATH. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Entire-Checkpoint: b8f20408aa93
Add a "Remote-helper schemes" section to docs/usage.md and a README FAQ entry explaining that git-sync falls back to git-remote-<scheme> for non-native URL schemes, with entire:// as the motivating example, and noting the --stats throughput caveat shared with SSH. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Entire-Checkpoint: 91460e76a6df
Post-review cleanup of the remote-helper transport, no behavior change: - Extract a shared readRawPktLine primitive in pktline.go and route both readAdvertisement and responseEndReader through it, replacing two hand-rolled copies of pkt-line header parsing. This also adds the pktline.MaxSize bound both copies were missing, and lets responseEndReader reuse one grow-only buffer instead of allocating per packet — relaying a multi-GB pack now costs no per-packet allocation. - Drop the unreachable os.ErrClosed guard in helperProcess.finish (the sync.Once already makes a redundant close impossible) and collapse it to a single errors.Join. - Merge RequestInfoRefs's twin error branches into one errors.Join. - Fold newConn's scheme dispatch into a single switch so the native scheme set is named once instead of split between a switch and a trailing if. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Entire-Checkpoint: de61fb4754bb
Before opening a stateless-connect session, dial now issues the remote helper's `capabilities` command and checks that stateless-connect is advertised. A helper that lacks it (e.g. one offering only the legacy `connect` capability) fails with an actionable error naming the helper and its advertised capabilities, instead of a cryptic mid-protocol failure. The probe is a local exchange — no network round trip happens until stateless-connect — so it adds no meaningful cost. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Entire-Checkpoint: 53d1e0ac66e6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a git remote-helper transport to git-sync so it can drive URL schemes it has no native transport for. When the scheme isn't one git-sync handles natively, it looks for a
git-remote-<scheme>binary onPATH(the same conventiongituses) and bridges itsstateless-connectcapability — delegating auth and network I/O to the helper while git-sync still runs the wire protocol.The motivating case is Entire's own
entire://URLs:Previously this failed with
unsupported protocol scheme "entire"because the URL fell through to Go's HTTP transport.entire://is backed by thegit-remote-entirehelper (whichgit clone entire://…invokes), so git-sync now delegates to it the same way.How
internal/gitproto/helper.goaddsHelperConn, agitproto.Connbacked by a helper process. Key protocol details (verified againstgit-remote-entire's source + live probing):0002response-end packet (stripped so the consumer sees the same byte stream it would over HTTP).report-statusback.DecodeV2Capabilities/decodeV1AdvRefsalready accept.dialprobes the helper'scapabilitiesand fails with a clear error ifstateless-connectis absent (noconnectfallback).http/https/sshare never routed to a helper — git-sync keeps its optimized native transports even thoughgitshipsgit-remote-http(s).Scope / limitations
stateless-connect-capable helper (entire, modern git-remote-curl). Helpers offering only the legacyconnectcapability are rejected with a clear error, not supported.--statsomits helper-side throughput.Testing
helper_test.godrives a fake helper via the os/exec re-exec pattern: v2 advertisement, response-end stripping, receive-pack read-to-EOF,fallback, missing-stateless-connect error, plus unit tests for the pkt-line framing helpers.probenegotiates v2, and the realreplicatefrom GitHub →entire://pushed 55,008 refs (protocol=v2, relay-mode=bootstrap).go test ./...,go vet, andgolangci-lintall clean.🤖 Generated with Claude Code
Note
Medium Risk
New transport path touches core sync connection setup and subprocess/protocol bridging; mistakes could break pushes or mis-handle schemes, but native http/https/ssh paths are unchanged and helpers are opt-in via PATH.
Overview
Adds a remote-helper transport so non-native URL schemes (e.g.
entire://) work whengit-remote-<scheme>is onPATH, matching howgitresolves custom remotes.newConnin the syncer now routes ssh and http/https to existing native transports; other schemes useHelperConnwhen a helper exists, otherwise behavior stays as before (unsupported scheme from HTTP). http/https are never delegated togit-remote-http(s).HelperConnimplementsgitproto.Connby spawning a helper per RPC, probing forstateless-connect(no legacyconnectfallback), driving upload-pack/receive-pack over the helper bridge, stripping v20002response-end framing, and passing the user URL verbatim to the helper for auth (e.g. Entire CLI /git-remote-entire).Pkt-line handling gains
readRawPktLinefor verbatim on-wire frames used when reading helper advertisements and responses. README and usage docs describe helper schemes,entire://examples, and that--statsomits helper-side byte counts (like SSH).Reviewed by Cursor Bugbot for commit 4ff9494. Configure here.