Skip to content

feat: download an MCQA sub query's result over the instance tunnel - #8

Merged
dingxin-tech merged 1 commit into
dingxin-tech:mainfrom
TonyStack2026:tony/em-mcqa-instance-tunnel
Sep 29, 2026
Merged

dingxin-tech merged 1 commit into
dingxin-tech:mainfrom
TonyStack2026:tony/em-mcqa-instance-tunnel

Conversation

@TonyStack2026

Copy link
Copy Markdown
Contributor

Why

SQLExecutor(INTERACTIVE) reads a sub query through the instance tunnel unless the caller opts out, and that is also the path JDBC's MaxQA mode takes. Before this, an interactive consumer fell back to re-running its statement as an offline instance — so #6's own acceptance test had to set useInstanceTunnel(false) to avoid being greened by that fallback. This closes B3, the last item standing between the emulator and a local MaxQA regression run.

Stacked on #6 (this branch is #6 + 1 commit): the diff here shows #6's session commit as well. If you merge #6 first, this PR collapses to a single commit; if you'd rather review them together, review #6 and this one in order.

Contract

Derived from the public Java SDK at the pinned release used by tests/java (0.61.2-public, 1c11317):

GET /projects/{p}/instances/{id}?data&cached&schema_in_stream&taskname=..[&queryid=N]
     [&rowrange=(start,count)][&sizelimit=bytes][&instance_tunnel_limit_enabled]
  • No download session exists to ask for a schema, so the stream carries its own: field 1 JSON (TunnelTableSchema shape) → SCHEMA_END_TAG 33553920 → that frame's CRC32C (over the field number as a 4-byte LE int, then the payload — not the tag or the length). The reader resets its checksum there, so the records after the frame are byte-identical to wire.Protobuf — pinned by TestProtobufSchemaTailMatchesPlainStream.
  • odps-tunnel-record-count = rows still available from this read's start offset, not the size of the result. SessionRecordSetIterator remembers the first answer's count and pages openRecordReader(offset+cursor, fetchSize) while cursor < count; reporting the total makes an offset read walk off the end. (This is the one number here I could not read off the SDK, only deduce from its loop — the Java case offsetReadPagesTheSubQueryResult is the check.)
  • rowrange windows; sizelimit shortens a response by halving (never to zero rows, so a reader always advances); instance_tunnel_limit_enabled applies READ_TABLE_MAX_ROW (10k) to that read only — the sub query keeps its full result.
  • Still-running sub query → the read waits (that is what cached asks for), and past 30s answers the SDK's own retryable OdpsTaskTimeout / "Wait for cache data timeout" instead of an empty stream.
  • Distinct structured answers, never an empty 200: failed sub query, cancelled (Job is cancelled), unknown queryid, wrong taskname, one-shot instance (InvalidState, points at ?downloads), and a statement with no result set → InstanceTypeNotSupported / "Non select query not supported", the pair SQLExecutorImpl#checkIsSelect looks for. Tunnel errors are JSON now (fail() learned to treat ?cached as a tunnel endpoint) — the XML body left the SDK with no error code to route on.
  • Sub queries keep their typed result beside the CSV text, in the same retention window and the same instance byte budget, so both paths of one statement serve the same rows with the same types.

Verification (source 9a372fa, Linux amd64, 2026-09-22 09:3x CST)

Check Result
go test -race -count=1 ./internal/... engine / server / wire all ok; 10 new cases in mcqa_download_test.go
go vet -unreachable=false ./..., gofmt -l clean
McqaSessionTest vs a locally built binary 6/6, incl. defaultFetchDownloadsTheSubQueryResult (typed bigint back from the stream) and offsetReadPagesTheSubQueryResult
Same two cases vs the previous build (#6) both fail with NoSuchObject: unsupported endpoint — they are not passing via the offline fallback
EmulatorTest 26/26, UdfMetadataTest 4/4 unchanged, no offline-plane regression
tests/python/run.py PyODPS contract probe, fresh process 30 passed / 0 failed
ReliabilityTest not run locally: testcontainers can't inspect the docker bridge network on this host; it runs in CI

Replay: go test -race ./internal/..., then go build -o /tmp/em ./cmd/emulator && /tmp/em -listen 127.0.0.1:18221 & and mvn -o -f tests/java/pom.xml -Demulator.endpoint=http://127.0.0.1:18221 -Dtest=McqaSessionTest test.

Still not simulated (said so in README / /capabilities)

Compressed or Arrow-encoded sub-query downloads (Protobuf only), named-session attach, MaxQA v2 (/mcqa), per-statement statistics, session-scoped SQL state isolation. subquery-instance-tunnel was added to the mcqa capability list.

M-B's B3: `SQLExecutor(INTERACTIVE)` reads a sub query through the instance
tunnel unless the client opts out, and that is the path JDBC's MaxQA mode
takes. Without it an interactive consumer silently re-runs its statement as an
offline instance, so the session plane could not be accepted end to end —
the previous increment had to set `useInstanceTunnel(false)` in its own
acceptance test to avoid being fooled by that fallback.

- `GET .../instances/{id}?data&cached&taskname=..[&queryid=N]` answers one sub
  query as a Protobuf record stream. There is no download session behind it, so
  the stream carries its own schema frame (field 1 JSON, then SCHEMA_END_TAG
  33553920 with the CRC32C of the field number and payload); the records after
  it are byte-identical to the ones the offline download path writes.
- `odps-tunnel-record-count` reports the rows still available from the read's
  start offset, not the size of the result: `SessionRecordSetIterator` keeps the
  first answer's count and pages `openRecordReader(offset+cursor, fetchSize)`
  while the cursor is below it, so reporting the total sends a client reading
  past the end.
- `rowrange` windows, `sizelimit` shortens a response by halving (at least one
  row, so a reader always advances) and `instance_tunnel_limit_enabled` applies
  READ_TABLE_MAX_ROW to that read only.
- A sub query that is still running makes the read wait — long polling is what
  `cached` asks for — and a wait past the bound answers with the SDK's own
  retryable tunnel timeout instead of an empty stream. Failed, cancelled,
  unknown and dropped sub queries, and one-shot instances, each get their own
  structured JSON error; a statement with no result set is refused with
  `InstanceTypeNotSupported`, which is how the service sends the client to the
  session API instead of an empty download.
- Sub queries now retain their typed result next to the CSV text, inside the
  same retention window and the same instance byte budget, so both read paths of
  one statement serve the same rows and the same types.

Evidence: 10 new Go cases (framing and CRC checked by decoding the response with
our own record decoder, paging, size limit, the 10k cap, non-select, structured
failures, long poll, idle-TTL activity, retention) — `go test -race ./internal/...`
green. Against a locally built binary `McqaSessionTest` is 6/6, including two new
SDK cases: the default fetch, and an offset read that pages the same result; both
fail on the previous build with `NoSuchObject: unsupported endpoint`, so they are
not passing by falling back. `EmulatorTest` 26/26 and `UdfMetadataTest` 4/4 stay
green, and the PyODPS contract probe stays at 30/0 on a fresh process.
@TonyStack2026
TonyStack2026 force-pushed the tony/em-mcqa-instance-tunnel branch from 51edb8c to 8d6607b Compare September 22, 2026 12:50
@dingxin-tech
dingxin-tech merged commit c2b410f into dingxin-tech:main Sep 29, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants