Support RESP3 replies in redis.setresp(3)

[fatal10110]

Summary

• Add RESP3 ABI tags and TypeScript ReplyValue shapes for booleans, doubles, maps, sets, big numbers, and verbatim strings.
• Make redis.setresp(3) enable RESP3 Lua conversions for the current script instead of rejecting it.
• Decode RESP3 host replies for redis.call / redis.pcall, and document the expanded ABI.
• Clarify that RESP3 push frames are not a Lua return type; they are server-initiated transport messages outside this script-engine surface.

Root Cause

redis.setresp(3) updated a protocol flag, but neither the C return encoder, host reply decoder, TypeScript codec, nor public ReplyValue type could represent RESP3 values. The first PR draft avoided the silent no-op by rejecting RESP3; this update implements the missing representable RESP3 shapes instead.

Real Redis Lua scripts can switch command reply conversion with redis.setresp(3), but push replies are not ordinary values returned by redis.call or by the script itself. They are out-of-band server-to-client protocol frames, so this PR models the RESP3 value types scripts can actually consume/return and documents push as out of scope.

Validation

• Red before fix: node --test --import tsx --test-name-pattern "redis\\.setresp" test/engine.test.ts failed because RESP3 conversions were not represented.
• Red before fix: node --test --import tsx --test-name-pattern "RESP3 values roundtrip" test/codec.test.ts failed because RESP3 values fell through to bulk encoding.
• node --test --import tsx --test-name-pattern "redis\\.setresp|RESP3 values roundtrip" test/engine.test.ts test/codec.test.ts passed: 5/5.
• npm run build passed.
• npm test passed: 175/175.
• Docs-only follow-up: clarified docs/limits-compat.md wording for RESP3 push scope.

Fixes #9

[fatal10110]

RESP3 support looks solid and symmetric across the C encoder, C/TS decoders, and the type surface. The idx absolutization in encode_table correctly accounts for the new lua_getfield probes. Two minor findings below.

[fatal10110]

REPLY_BIG_NUMBER decode skips the bounds check that every sibling RESP3 decoder here performs (BOOL/DOUBLE/VERBATIM all throw new Error("ERR reply decoding failed")), and that the C decoder does too (*offset + count_or_len > len). On a truncated/malformed buffer, buffer.subarray(cursor, cursor + countOrLen) silently clamps and you get a short big_number instead of an error. Add:

if (cursor + countOrLen > buffer.length) {
  throw new Error("ERR reply decoding failed");
}

[fatal10110]

Redis's redis.setresp raises "RESP version must be 2 or 3." with no ERR prefix (src/script_lua.c luaSetResp). The added "ERR " here yields a spurious ERR code, deviating from the Redis-7 error fidelity the project targets. Drop the prefix to match.

[fatal10110]

Correction on my earlier review: the second finding (drop the ERR prefix from redis.setresp's error) was wrong — retracted. The codec splits the leading [A-Z][A-Z0-9]* token as code, so without ERR the message "RESP version must be 2 or 3." would parse RESP as the code and mangle the message. The ERR prefix correctly yields code=ERR + the Redis message body, matching how real Redis surfaces the error to clients. No change there.

The first finding (big-number bounds check) is fixed in e07e4c1.