Push streaming requires PHP 8.1+: PHP's curl binding couldn't pause uploads before 8.1

[adamziel]

TL;DR

reprint push streams file bytes to the remote site as it reads them from disk — the request body is never buffered in memory. Doing that through PHP's curl extension requires a mechanism (CURL_READFUNC_PAUSE) that the extension only supports since PHP 8.1. On PHP 7.4 and 8.0 the exact same code doesn't error — it silently truncates the upload, which is the worst possible failure mode for a sync tool. So the push client refuses to construct on PHP older than 8.1 and points at this issue. Pull/import is unaffected and still supports PHP 7.4+.

What push needs from HTTP

A push session sends one staged_push HTTP request whose body carries framed chunks of many files. The sender works like this:

1. Read a few MiB of a file into memory (one chunk — the only thing ever buffered).
2. Write that chunk into the open HTTP request, so it leaves the machine now.
3. Return control to the caller, which may persist a resume cursor, check its budgets, or pause.
4. Repeat until the request's byte or time budget is spent, then finish the request, read the server's response, and open the next request where the cursor left off.

Step 2 + 3 is the crucial shape: the HTTP client must let us write part of a request body, get control back, and write more later. If the client only supports "give me the whole body, then I'll send it", we'd have to hold the entire request body (up to 1 GiB) in memory, and a crash would lose track of what the server actually received. The server commits each frame as it arrives, so interrupted requests resume cheaply — but only if bytes actually flow while the request is open.

How curl uploads actually work

curl is pull-based. You don't hand it bytes; you register a read callback (CURLOPT_READFUNCTION), and libcurl calls it whenever the network socket can accept more data:

curl_setopt($handle, CURLOPT_READFUNCTION, function ($handle, $stream, int $length) {
    // libcurl is asking: "give me up to $length bytes of the request body"
    return $more_bytes;   // a string; returning '' means "the body is finished"
});

With the classic curl_exec(), the entire transfer happens inside one blocking call — your code cannot run between chunks. curl has an escape hatch designed exactly for our situation: the read callback may answer "I have nothing right now, but the body is not finished" by returning the constant CURL_READFUNC_PAUSE. The transfer's upload side then pauses. When you have more data, you call:

curl_pause($handle, CURLPAUSE_CONT);          // wake the upload up
curl_multi_exec($multi_handle, $running);     // drive the transfer forward (in a loop)

Combined with the curl_multi_* API (which, unlike curl_exec(), lets you drive a transfer in small non-blocking steps), this produces exactly the push shape: feed bytes → pump until libcurl consumed them → return to the caller → repeat → signal end-of-body → read the response.

The PHP 7.4 / 8.0 problem

PHP's curl extension sits between your callback and libcurl: it invokes your PHP function, takes the return value, and copies it into libcurl's buffer. Here is the relevant code from PHP 8.0's ext/curl/interface.c (curl_read()), lightly trimmed:

if (Z_TYPE(retval) == IS_STRING) {
    length = MIN((int) (size * nmemb), Z_STRLEN(retval));
    memcpy(data, Z_STRVAL(retval), length);
}
/* no other return type is handled — length stays 0 */
return length;

Only string returns are handled. CURL_READFUNC_PAUSE is an integer, so it matches no branch and length stays 0 — and per libcurl's contract, a read callback returning 0 means end of body. So on PHP ≤ 8.0, "please pause" is delivered to libcurl as "the upload is complete": curl finishes the chunked body, the server sees a clean end-of-stream mid-file, and nobody gets an error. The file is just short.

PHP 8.1 added the missing branch (php/php-src ext/curl/interface.c, PHP-8.1 branch):

if (Z_TYPE(retval) == IS_STRING) {
    length = MIN((int) (size * nmemb), Z_STRLEN(retval));
    memcpy(data, Z_STRVAL(retval), length);
} else if (Z_TYPE(retval) == IS_LONG) {
    length = Z_LVAL_P(&retval);   /* CURL_READFUNC_PAUSE reaches libcurl */
}

Compare with the PHP-8.0 branch of the same file, where the IS_LONG branch does not exist. PHP 7.4 is identical to 8.0 in this respect.

Verified, not assumed

Two things were checked empirically while building this:

1. The pause mechanism truly streams. We pointed a curl-multi + pause upload at a raw TCP listener (no HTTP server, so byte arrival timing is directly observable) and asserted: a 100 KB window — larger than libcurl's 64 KB upload buffer — arrives at the listener while the chunked body is still open; the transfer sits idle through a 300 ms gap without ending; a second window arrives after unpausing; the chunked terminator appears on the wire only after end-of-body is signaled. Result on PHP 8.4.5 / libcurl 8.20.0: every assertion held. So on PHP 8.1+, send_chunk() means "these bytes left for the network before I returned" — at most libcurl's 64 KB upload buffer sits between our code and the socket.
2. The 7.4/8.0 gap is real. The missing IS_LONG branch is visible in the extension source linked above, and a probe on the pause path shows the int return being treated as end-of-body.

Why not work around it?

• Buffer the body, send with one curl_exec() — works on any PHP, but needs request-body-sized memory (up to 1 GiB), makes mid-request pacing impossible, and turns "resume from cursor" into guesswork. This is the design push exists to avoid.
• Raw TCP sockets — we built this first and it worked, but it hand-rolls HTTP/1.1 forever: no HTTP/2 when available, no ALL_PROXY support, our own TLS and response parsing to maintain. libcurl gives all of that for free.
• Both, version-gated — two transports to keep correct instead of one.

Push is a brand-new feature, so requiring PHP 8.1+ for it takes nothing away from anyone. (PHP 8.0 reached end of life in November 2023; 7.4 in November 2022.)

What you'll see on PHP 7.4 / 8.0

Constructing the push client throws immediately:

reprint push requires PHP 8.1 or newer: streaming request bodies through curl needs CURL_READFUNC_PAUSE support, which PHP's curl extension added in 8.1 — on older PHP the pause return is misread as end-of-body and the upload silently truncates. Current PHP is X.Y.Z. See for the full story.

Everything else — pull, db-apply, apply-runtime — keeps its existing PHP 7.4+ support.