Fix: ds4-server rejects HTTP requests using Transfer-Encoding: chunked

[moritzburgard]

Bug: ds4-server rejects HTTP requests using Transfer-Encoding: chunked

The ds4-server HTTP API (/v1/chat/completions, /v1/messages, etc.) rejects POST requests that use chunked transfer encoding with a 400 Bad Request and {"error":{"message":"invalid JSON request"}}. Requests with an explicit Content-Length header work correctly.

This affects any client that sends chunked requests — notably the Roo Code VS Code extension, which uses Chromium's networking stack and may send chunked requests even when the OpenAI SDK would normally set Content-Length.

Root cause

read_http_request() in ds4_server.c uses a custom HTTP parser that determines the request body length solely from the Content-Length header. When Content-Length is absent and the client uses Transfer-Encoding: chunked, content_length() returns 0. The parser reads zero bytes of body, leaving the request body empty. The JSON parser then fails, producing the "invalid JSON request" error.

Per RFC 7230 section 3.3.3, a client may send Transfer-Encoding: chunked without Content-Length. A compliant server must handle this.

Fix

Added has_chunked_transfer_encoding() — scans headers for Transfer-Encoding with value chunked (case-insensitive, correctly matching the final encoding token to avoid false positives on values like gzip, chunked). Modified read_http_request() to detect chunked encoding and, when present, incrementally read and decode the chunked body in a single linear pass.

Key properties of the implementation:

• O(N) single-pass decoding — no quadratic re-scan on each new recv
• RFC 9112 compliant — consumes the trailing CRLF and trailer section before closing the connection, preventing TCP RST
• Request smuggling mitigation — rejects requests that set both Transfer-Encoding: chunked and Content-Length
• Robust chunk size parsing — validates strtol output; invalid hex or out-of-range sizes produce a 400 Bad Request immediately
• Handles LF-only terminators — detects \r\n vs \n chunk terminators dynamically

Verification

Before the fix, a Node.js POST without Content-Length returned 400:

const http = require('http');
const body = JSON.stringify({model:'deepseek-v4-flash',messages:[{role:'user',content:'hi'}],max_tokens:50});
const req = http.request('http://127.0.0.1:8000/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' }
}, (res) => {
    let data = '';
    res.on('data', c => data += c);
    res.on('end', () => console.log(res.statusCode));
});
req.write(body); req.end();
// Before fix → 400
// After  fix → 200

A raw chunked request via netcat also works:

printf "POST /v1/chat/completions HTTP/1.1\r\nHost: 127.0.0.1:8000\r\nContent-Type: application/json\r\nTransfer-Encoding: chunked\r\n\r\n%x\r\n%s\r\n0\r\n\r\n" \
  $(echo -n '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hi"}],"max_tokens":50}' | wc -c) \
  '{"model":"deepseek-v4-flash","messages":[{"role":"user","content":"hi"}],"max_tokens":50}' \
| nc -N 127.0.0.1 8000
# → HTTP/1.1 200 OK

Backward compatibility

Requests with Content-Length continue to use the original fast path. Chunked requests incur a small scan-and-copy overhead to reassemble the body. No existing functionality is affected.

---

Note: The initial bug diagnosis was developed with ds4-agent. The implementation was reviewed for RFC compliance, memory safety, and correctness by Antigravity (Google DeepMind), which identified and fixed additional edge cases including TCP packet boundary bugs, trailer handling, and request smuggling.