Skip to content

Support git remote-helper schemes (entire://) as a transport#96

Open
Soph wants to merge 5 commits into
mainfrom
feat/remote-helper-transport
Open

Support git remote-helper schemes (entire://) as a transport#96
Soph wants to merge 5 commits into
mainfrom
feat/remote-helper-transport

Conversation

@Soph

@Soph Soph commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

What

Adds a git remote-helper transport to git-sync so it can drive URL schemes it has no native transport for. When the scheme isn't one git-sync handles natively, it looks for a git-remote-<scheme> binary on PATH (the same convention git uses) and bridges its stateless-connect capability — delegating auth and network I/O to the helper while git-sync still runs the wire protocol.

The motivating case is Entire's own entire:// URLs:

git-sync replicate --all-refs \
  https://github.com/source-org/source-repo.git \
  entire://cluster-host/et/project/repo

Previously this failed with unsupported protocol scheme "entire" because the URL fell through to Go's HTTP transport. entire:// is backed by the git-remote-entire helper (which git clone entire://… invokes), so git-sync now delegates to it the same way.

How

internal/gitproto/helper.go adds HelperConn, a gitproto.Conn backed by a helper process. Key protocol details (verified against git-remote-entire's source + live probing):

  • Fetch (upload-pack) uses the v2 stateless framing: ack, advertisement, per-request POST with a trailing 0002 response-end packet (stripped so the consumer sees the same byte stream it would over HTTP).
  • Push (receive-pack) is a connect-style raw proxy that reads the pushed pack until stdin EOF and streams report-status back.
  • A fresh helper process is spawned per RPC: the helper services one stateless-connect session per process, and receive-pack reads the pack to EOF — so per-RPC processes are the simplest correct model and naturally support batched (multi-request) pushes.
  • The advertisement comes back banner-stripped (like SSH), which the existing DecodeV2Capabilities/decodeV1AdvRefs already accept.
  • Before opening a session, dial probes the helper's capabilities and fails with a clear error if stateless-connect is absent (no connect fallback).

http/https/ssh are never routed to a helper — git-sync keeps its optimized native transports even though git ships git-remote-http(s).

Scope / limitations

  • Requires a stateless-connect-capable helper (entire, modern git-remote-curl). Helpers offering only the legacy connect capability are rejected with a clear error, not supported.
  • Like SSH, the helper bridge has no per-request byte accounting, so --stats omits helper-side throughput.

Testing

  • New helper_test.go drives a fake helper via the os/exec re-exec pattern: v2 advertisement, response-end stripping, receive-pack read-to-EOF, fallback, missing-stateless-connect error, plus unit tests for the pkt-line framing helpers.
  • Validated end-to-end against the live server: probe negotiates v2, and the real replicate from GitHub → entire:// pushed 55,008 refs (protocol=v2, relay-mode=bootstrap).
  • go test ./..., go vet, and golangci-lint all clean.

🤖 Generated with Claude Code


Note

Medium Risk
New transport path touches core sync connection setup and subprocess/protocol bridging; mistakes could break pushes or mis-handle schemes, but native http/https/ssh paths are unchanged and helpers are opt-in via PATH.

Overview
Adds a remote-helper transport so non-native URL schemes (e.g. entire://) work when git-remote-<scheme> is on PATH, matching how git resolves custom remotes.

newConn in the syncer now routes ssh and http/https to existing native transports; other schemes use HelperConn when a helper exists, otherwise behavior stays as before (unsupported scheme from HTTP). http/https are never delegated to git-remote-http(s).

HelperConn implements gitproto.Conn by spawning a helper per RPC, probing for stateless-connect (no legacy connect fallback), driving upload-pack/receive-pack over the helper bridge, stripping v2 0002 response-end framing, and passing the user URL verbatim to the helper for auth (e.g. Entire CLI / git-remote-entire).

Pkt-line handling gains readRawPktLine for verbatim on-wire frames used when reading helper advertisements and responses. README and usage docs describe helper schemes, entire:// examples, and that --stats omits helper-side byte counts (like SSH).

Reviewed by Cursor Bugbot for commit 4ff9494. Configure here.

Soph and others added 5 commits June 19, 2026 18:24
Implement a gitproto.Conn backed by a git-remote-<scheme> binary, bridging
git-sync's smart transport onto the helper's stateless-connect capability.
This lets git-sync drive URL schemes it has no native transport for (e.g.
entire://) by delegating auth and network I/O to the helper while still
running the wire protocol itself.

Each Conn operation spawns its own helper process: the helper services one
stateless-connect session per process, and its receive-pack path reads the
pushed pack until stdin EOF, so a fresh process per RPC is the simplest
correct model and naturally supports batched (multi-request) pushes. The v2
upload-pack response is framed with a trailing response-end (0002) packet,
which is stripped so the consumer sees exactly the byte stream it would over
HTTP; receive-pack responses run to EOF. The helper strips the smart-HTTP
"# service=" banner, so advertisements match the bannerless SSH shape that
the existing decoders already accept.

Tests drive a fake helper via the os/exec re-exec pattern, covering the v2
advertisement, response-end stripping, receive-pack read-to-EOF, and the
fallback path, plus unit tests for the pkt-line framing helpers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 1be8d21b867a
newConn now checks for a git-remote-<scheme> binary when the endpoint scheme
isn't one git-sync handles natively, and builds a HelperConn when one exists.
http/https are excluded so they always use the optimized native HTTP transport
(git ships its own git-remote-http(s)); ssh is already handled above. This is
what makes entire:// URLs work, given git-remote-entire on PATH.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: b8f20408aa93
Add a "Remote-helper schemes" section to docs/usage.md and a README FAQ entry
explaining that git-sync falls back to git-remote-<scheme> for non-native URL
schemes, with entire:// as the motivating example, and noting the --stats
throughput caveat shared with SSH.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 91460e76a6df
Post-review cleanup of the remote-helper transport, no behavior change:

- Extract a shared readRawPktLine primitive in pktline.go and route both
  readAdvertisement and responseEndReader through it, replacing two hand-rolled
  copies of pkt-line header parsing. This also adds the pktline.MaxSize bound
  both copies were missing, and lets responseEndReader reuse one grow-only
  buffer instead of allocating per packet — relaying a multi-GB pack now costs
  no per-packet allocation.
- Drop the unreachable os.ErrClosed guard in helperProcess.finish (the
  sync.Once already makes a redundant close impossible) and collapse it to a
  single errors.Join.
- Merge RequestInfoRefs's twin error branches into one errors.Join.
- Fold newConn's scheme dispatch into a single switch so the native scheme set
  is named once instead of split between a switch and a trailing if.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: de61fb4754bb
Before opening a stateless-connect session, dial now issues the remote helper's
`capabilities` command and checks that stateless-connect is advertised. A helper
that lacks it (e.g. one offering only the legacy `connect` capability) fails
with an actionable error naming the helper and its advertised capabilities,
instead of a cryptic mid-protocol failure. The probe is a local exchange — no
network round trip happens until stateless-connect — so it adds no meaningful
cost.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 53d1e0ac66e6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant