Back off QIT 429 retries in CLI

[zhongruige]

Summary

Fixes the qit-cli pieces of QIT-991:

• Parses Retry-After consistently for HTTP 429 responses.
• Uses exponential backoff plus jitter for 429 retries.
• Fails immediately with a clear wait message when the server asks for a longer Retry-After than the CLI retry budget allows.
• Retries signed test-package ZIP downloads that receive HTML 429 responses from the platform edge.
• Deletes partial HTML error responses so they cannot be mistaken for downloaded ZIP files.
• Resets the per-request 429 retry budget so reused upload builders do not exhaust later chunk retries.
• Adds jittered backoff for remote test status polling after failed polls instead of retrying once per second.

Root Cause

Vendor runs could burst multiple Manager API requests and signed ZIP downloads through the WordPress.com/Atomic edge. The edge can return HTTP 429 with an HTML body before QIT PHP runs. The CLI previously retried some Manager 429s with a flat wait, did not retry static ZIP 429s cleanly, could keep retrying polls every second after failures, and reused upload retry state across chunks.

Manual Testing

Recommended setup:

• Test this branch against a QIT Manager environment that includes the matching QIT-991 manager changes.
• Use a connected partner account that can run a command requiring a test package download.
• Use --verbose for CLI runs so retry/wait messages are visible.
• If a real edge 429 is hard to trigger, use a local proxy, staging fixture, or temporary endpoint override that returns HTTP 429 for the first request and then succeeds.

1. Normal package download still works

   • Run a command that downloads a test package ZIP, such as an environment/test-package workflow for an accessible package.
   • Expected: the command completes successfully.
   • Expected: no signed URL query parameters are printed in normal output.

2. Transient ZIP download 429 recovers

   • Arrange for the signed ZIP download URL to return HTTP 429 once, then HTTP 200.
   • Run the same CLI command with --verbose.
   • Expected: the CLI prints a clear “download rate limited” wait/retry message.
   • Expected: the command retries and then completes successfully.
   • Expected: the final cached file is the ZIP, not the intermediate 429 HTML response.

3. Persistent ZIP download 429 fails cleanly

   • Arrange for the signed ZIP download URL to keep returning HTTP 429 with an HTML body.
   • Run the same CLI command.
   • Expected: the CLI exits with a clean HTTP 429/rate-limit message.
   • Expected: raw HTML is not shown to the user.
   • Expected: signed URL query secrets are not shown.
   • Expected: no partial HTML file remains in the ZIP cache.

4. Manager API Retry-After is respected

   • Arrange for a Manager API request made by the CLI to return HTTP 429 with Retry-After: 120, then succeed.
   • Expected: the CLI waits/retries instead of failing immediately.
   • Arrange for the same API request to return HTTP 429 with a Retry-After value greater than 180 seconds.
   • Expected: the CLI exits quickly with a clear message telling the user roughly how long to wait before trying again.

5. Remote test polling backs off after failures

   • Start a remote test command that waits for completion.
   • Arrange for /wp-json/cd/v1/get-single polling to fail or return HTTP 429 for a few attempts.
   • Expected: the CLI does not retry every second.
   • Expected: retries back off with jitter, then normal polling resumes after a successful response.

6. Chunked upload keeps a fresh retry budget per chunk

   • Upload a large ZIP, or lower UPLOAD_CHUNK_KB in a test run so the upload uses several chunks.
   • Arrange for one chunk to return HTTP 429 and then succeed.
   • Expected: that chunk retries and succeeds.
   • Expected: later chunks can still retry their own HTTP 429 responses instead of failing because an earlier chunk consumed the retry budget.