Repository navigation
feat: download an MCQA sub query's result over the instance tunnel - #8
Merged
dingxin-tech merged 1 commit intoSep 29, 2026
Conversation
This was referenced Sep 22, 2026
TonyStack2026
force-pushed
the
tony/em-mcqa-instance-tunnel
branch
from
September 22, 2026 03:22
9a372fa to
51edb8c
Compare
M-B's B3: `SQLExecutor(INTERACTIVE)` reads a sub query through the instance
tunnel unless the client opts out, and that is the path JDBC's MaxQA mode
takes. Without it an interactive consumer silently re-runs its statement as an
offline instance, so the session plane could not be accepted end to end —
the previous increment had to set `useInstanceTunnel(false)` in its own
acceptance test to avoid being fooled by that fallback.
- `GET .../instances/{id}?data&cached&taskname=..[&queryid=N]` answers one sub
query as a Protobuf record stream. There is no download session behind it, so
the stream carries its own schema frame (field 1 JSON, then SCHEMA_END_TAG
33553920 with the CRC32C of the field number and payload); the records after
it are byte-identical to the ones the offline download path writes.
- `odps-tunnel-record-count` reports the rows still available from the read's
start offset, not the size of the result: `SessionRecordSetIterator` keeps the
first answer's count and pages `openRecordReader(offset+cursor, fetchSize)`
while the cursor is below it, so reporting the total sends a client reading
past the end.
- `rowrange` windows, `sizelimit` shortens a response by halving (at least one
row, so a reader always advances) and `instance_tunnel_limit_enabled` applies
READ_TABLE_MAX_ROW to that read only.
- A sub query that is still running makes the read wait — long polling is what
`cached` asks for — and a wait past the bound answers with the SDK's own
retryable tunnel timeout instead of an empty stream. Failed, cancelled,
unknown and dropped sub queries, and one-shot instances, each get their own
structured JSON error; a statement with no result set is refused with
`InstanceTypeNotSupported`, which is how the service sends the client to the
session API instead of an empty download.
- Sub queries now retain their typed result next to the CSV text, inside the
same retention window and the same instance byte budget, so both read paths of
one statement serve the same rows and the same types.
Evidence: 10 new Go cases (framing and CRC checked by decoding the response with
our own record decoder, paging, size limit, the 10k cap, non-select, structured
failures, long poll, idle-TTL activity, retention) — `go test -race ./internal/...`
green. Against a locally built binary `McqaSessionTest` is 6/6, including two new
SDK cases: the default fetch, and an offset read that pages the same result; both
fail on the previous build with `NoSuchObject: unsupported endpoint`, so they are
not passing by falling back. `EmulatorTest` 26/26 and `UdfMetadataTest` 4/4 stay
green, and the PyODPS contract probe stays at 30/0 on a fresh process.
TonyStack2026
force-pushed
the
tony/em-mcqa-instance-tunnel
branch
from
September 22, 2026 12:50
51edb8c to
8d6607b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
SQLExecutor(INTERACTIVE)reads a sub query through the instance tunnel unless the caller opts out, and that is also the path JDBC's MaxQA mode takes. Before this, an interactive consumer fell back to re-running its statement as an offline instance — so #6's own acceptance test had to setuseInstanceTunnel(false)to avoid being greened by that fallback. This closes B3, the last item standing between the emulator and a local MaxQA regression run.Stacked on #6 (this branch is
#6 + 1 commit): the diff here shows #6's session commit as well. If you merge #6 first, this PR collapses to a single commit; if you'd rather review them together, review #6 and this one in order.Contract
Derived from the public Java SDK at the pinned release used by
tests/java(0.61.2-public,1c11317):TunnelTableSchemashape) →SCHEMA_END_TAG33553920 → that frame's CRC32C (over the field number as a 4-byte LE int, then the payload — not the tag or the length). The reader resets its checksum there, so the records after the frame are byte-identical towire.Protobuf— pinned byTestProtobufSchemaTailMatchesPlainStream.odps-tunnel-record-count= rows still available from this read's start offset, not the size of the result.SessionRecordSetIteratorremembers the first answer's count and pagesopenRecordReader(offset+cursor, fetchSize)whilecursor < count; reporting the total makes an offset read walk off the end. (This is the one number here I could not read off the SDK, only deduce from its loop — the Java caseoffsetReadPagesTheSubQueryResultis the check.)rowrangewindows;sizelimitshortens a response by halving (never to zero rows, so a reader always advances);instance_tunnel_limit_enabledappliesREAD_TABLE_MAX_ROW(10k) to that read only — the sub query keeps its full result.cachedasks for), and past 30s answers the SDK's own retryableOdpsTaskTimeout/ "Wait for cache data timeout" instead of an empty stream.Job is cancelled), unknownqueryid, wrongtaskname, one-shot instance (InvalidState, points at?downloads), and a statement with no result set →InstanceTypeNotSupported/ "Non select query not supported", the pairSQLExecutorImpl#checkIsSelectlooks for. Tunnel errors are JSON now (fail()learned to treat?cachedas a tunnel endpoint) — the XML body left the SDK with no error code to route on.Verification (source
9a372fa, Linux amd64, 2026-09-22 09:3x CST)go test -race -count=1 ./internal/...ok; 10 new cases inmcqa_download_test.gogo vet -unreachable=false ./...,gofmt -lMcqaSessionTestvs a locally built binarydefaultFetchDownloadsTheSubQueryResult(typedbigintback from the stream) andoffsetReadPagesTheSubQueryResultNoSuchObject: unsupported endpoint— they are not passing via the offline fallbackEmulatorTest26/26,UdfMetadataTest4/4tests/python/run.pyPyODPS contract probe, fresh processReliabilityTestbridgenetwork on this host; it runs in CIReplay:
go test -race ./internal/..., thengo build -o /tmp/em ./cmd/emulator && /tmp/em -listen 127.0.0.1:18221 &andmvn -o -f tests/java/pom.xml -Demulator.endpoint=http://127.0.0.1:18221 -Dtest=McqaSessionTest test.Still not simulated (said so in README /
/capabilities)Compressed or Arrow-encoded sub-query downloads (Protobuf only), named-session attach, MaxQA v2 (
/mcqa), per-statement statistics, session-scoped SQL state isolation.subquery-instance-tunnelwas added to themcqacapability list.