fix: consume the parts a failed merge assembled from

[TonyStack2026]

What

A chunked upload whose merge fails verification (digest mismatch, declared x-odps-resource-merge-total-bytes mismatch) used to leave every part resource it had just assembled from. Now the merge consumes the parts it read in that case too, the same way it already consumes them on success.

Refusals decided before any part is read still keep the parts, deliberately:

merge attempt	published?	parts afterwards
success	yes	consumed (unchanged)
digest / declared-size mismatch	no	consumed (this PR)
no parts named, manifest not <md5>|<parts>	no	kept (nothing was read)
a declared part does not exist	no	kept
create while the target resource already exists	no	kept — the client may retry as an update

Why it matters to clients

The Java SDK names its chunks deterministically (<schema>.<resource>.part.tmp.<index>), so a refused duplicate create legitimately leaves a temp resource, and until now a refused merge leaked one per attempt with no way for a client to notice. Any check that sweeps a project for .part.tmp. then blames an unrelated upload.

That is exactly how the PyODPS contract probe started failing stream write (part+merge) lands byte-exact whenever the Java acceptance suite had run first against the same emulator instance: the leftover part belonged to the Java suite's duplicateResourceCreateIsRejected, not to the merge being asserted.

Changes

• internal/server/resources.go — one consumeParts() helper, called on the two post-read failures and on success.
• internal/server/resources_test.go — pins both sides of the line: a refused merge consumes its part (404 after), a refused create keeps it (200 after, target payload untouched).
• tests/python/run.py — scope the leftover sweep to the resource the case uploaded (it was project-wide, which made the case order-dependent); assert a refused merge consumes its own part (the old assertion pinned the opposite, and it was a guess, not an observed contract); cleanup() now also sweeps parts left by an aborted run of the probe itself.

Verification

• go test -race -count=1 ./internal/... — ok (server 18.5s, engine 3.1s, wire 1.2s).
• Negative control: with resources.go reverted, the new assertion fails as refused merge must consume its part: 200 x, so the test does pin the change.
• go build ./cmd/emulator, then the PyODPS probe (pyodps 0.13.2) against the running binary with a Java-style leftover seeded (POST ?rIsPart as default.dup_seedcheck.py.part.tmp.000000): 30 passed, 0 failed with this branch's probe; the pre-change probe against the same server fails only the two expectations this PR moves together.

Scope

Base is main @ 31af5c5. Touches resources.go / resources_test.go / tests/python/run.py — disjoint from the files in the currently open MCQA and REST-fault branches, except that all three append one line under Unreleased in CHANGELOG.md.

Not included, spotted while reading this path: a merge that declares a total size differing from the payload is refused here, while the signature is what actually decides the payload — this emulator is stricter than the service on that one header. And refused-merge / refused-create errors are answered 400 InvalidParameter, whereas the service distinguishes a save failure from an authorization failure. Both are error-semantics changes and want the same contract comparison before touching them, so they are left for a separate decision.

[TonyStack2026]

Re-based onto f7b2e4f and re-verified today; the order dependence this PR fixes is now measured on current main, in one server process, Java suite first then the probe:

server binary	tests/python/run.py	result
main f7b2e4f	main (project-wide .part.tmp. scan)	Java 30 run/0 fail, probe 29 passed / 1 failed
this PR ffb1ec9	this PR	Java 30 run/0 fail, probe 30 passed / 0 failed
this PR ffb1ec9	main's version	probe 28 passed / 2 failed

Failure on main:

FAIL stream write (part+merge) lands byte-exact :: AssertionError: temp parts removed after merge:
  got ['default.dup_55b63a5a9cf9.py.part.tmp.000000'] want []

The third row is the reason this PR changes both files, and it cuts against "the assertions were loosened to go green":

• the leftover part comes from UdfMetadataTest.duplicateResourceCreateIsRejected: the Java SDK uploads deterministic part names, and the merge is refused before it reads any part (target already exists). This PR deliberately keeps parts in that case, so the project-wide residue check still fails on this server — only narrowing the check to the resources the case itself created fixes that row.
• the main probe asserted "a refused merge keeps its parts", which is the behavior this PR changes. Against this server that old expectation fails with part kept after refusal: got False want True — the two halves are paired, not one-sided.

Still not verified, unchanged from the PR description: whether a real service endpoint consumes the parts of a failed merge. The merge consumes what it read rule comes from comparing the documented failure semantics, not from a live measurement.

CI on the rebased head: acceptance success (3m44s, run 35682893201). Main's own push run 35678348028 also went green with the probe step executing (30 passed, 0 failed) in a fresh container — which is exactly why CI cannot see this order dependence: the Java suite and the probe never share a process there.

[TonyStack2026]

Re-checked the server-side premise of this PR tonight against a live MaxCompute service
instead of only an implementation read. Same script, same cells, three subjects — the
service, the emulator at current main (700181e), and this PR's head (43bb9232):

cell	live service	main 700181e	this PR 43bb923
merge whose manifest MD5 does not match (part uploaded first)	refused — 500 InternalServerError, ODPS-0421213: Save resource error - Merge part temp files failed! Message: The merged file's signature does not match! — the part it had read is gone afterwards	refused — 400 InvalidParameter — part kept	refused — 400 InvalidParameter — part consumed
merge onto an already-existing target, no overwrite	refused — 409 ObjectAlreadyExists, ODPS-0421121 — part kept, original payload intact	refused — 400 ResourceAlreadyExists — part kept, payload intact	same as main
manifest MD5 correct, x-odps-resource-merge-total-bytes one byte too high	accepted, and it publishes the correct bytes	refused (400)	refused (400)
TABLE resource whose referenced table does not exist	refused — 404 NoSuchObject, ODPS-0422111: Table not found	accepted	accepted

So the line this PR draws is the line the service draws: a merge that reads parts and
then fails validation consumes them; a merge refused before reading anything
(duplicate target, bad manifest) leaves them alone. e1_part_consumed is the only cell
that flips between main and this head — the other three are unchanged, which is also
why this is not an assertion relaxed to make a test go green.

Two consequences worth recording, both outside this PR's scope and deliberately not
changed here:

1. Status/error codes differ. The service answers a digest mismatch with 500 InternalServerError + ODPS-0421213 and a duplicate create with 409 ObjectAlreadyExists + ODPS-0421121; the emulator answers both with 400. A client
   that branches on status (the Java SDK's RestClient only raises NoSuchObjectException
   on 404, and 5xx goes into its retry path) takes a different branch locally than on the
   service. Fixing this changes the probe and the consumer suites on both sides, so it
   needs its own contract discussion rather than riding along here.
2. The emulator is stricter on declared total bytes and looser on table existence.
   The service does not compare x-odps-resource-merge-total-bytes against what it
   assembled (it decides by digest), so a client that mis-declares it succeeds on the
   service and fails locally. Conversely the service refuses a TABLE resource that points
   at a missing table, while the emulator accepts it — a local pass where the service
   fails. These two go in opposite directions; aligning them is a decision, not a fix.

One operational note for anyone writing a residual-part check: after a refused duplicate
create the service keeps the part but does not list it — list(prefix=...) returns
nothing while a meta request for the exact name returns 200. Probing by name is the only
way to see that leftover; a prefix sweep silently misses it (it missed one for me before
I switched to deleting by name and re-probing).

Scope of the measurement, so it is not over-read: single part per merge, 2 KiB payloads,
one schema, default project. Multi-part, large-payload and concurrent-retry behaviour were
not measured. CHANGELOG/error-message text is treated as corroboration of "the service
did refuse this way", not as a citable contract.

[TonyStack2026]

Follow-up to my previous comment: yesterday's live-service run used a single part, and that cannot
distinguish the two readings of "a failed merge consumes the parts it read":

• A — consume every part the manifest declares.
• B — consume only the parts whose content was actually read before the failure.

With one part A and B give the same answer, so that measurement did not actually pin down which
semantics this PR implements. It implements A (read all declared parts into memory, validate, then
delete the whole declared list; if a listed part cannot be read at all, return before deleting
anything). Multi-part is also the path the Java SDK really uses for anything above 64 MiB, so this
needed a separate measurement.

Same script, same cells, only PROBE_ENDPOINT changed (700181e = current main, 43bb9232 = this head):

cell	live service	main 700181e	this PR 43bb9232
P1 three parts present, manifest digest wrong	refused (ODPS-0421213 … signature does not match); all three parts consumed	refused; all three parts kept	refused; all three parts consumed
P2 manifest lists a part that was never uploaded, listed last	refused (NoSuchObject); the two uploaded parts kept	refused; both kept	refused; both kept
P3 same, missing name listed first	refused (NoSuchObject); both kept — ordering changes nothing	refused; both kept	refused; both kept
P4 digest correct, x-odps-resource-merge-total-bytes one byte too high	accepted, publishes the correct 768 bytes	refused; parts kept	refused; parts consumed
P5 target exists, three valid parts, no overwrite	refused (409 ObjectAlreadyExists); all three kept, original payload intact	refused; all kept	refused; all kept

So the service is A as well: P1 clears every declared part, and in P2/P3 it keeps everything — if it
were B, P3 would have consumed the two parts it could read. That closes the gap my earlier comment
left open: this PR's line is the service's line, not just a plausible reading of it.

One finding worth re-rating: P4. The service ignores the declared byte count on the multi-part path
too, and after this PR a local run loses both the payload and the parts when a client over-declares
by one byte, while the same request succeeds against the service. That is a stronger argument than the
single-part case for aligning that cell (it is still outside this PR — I did not touch it here).
Status/error codes differ on the multi-part path as well: 500 / 409 / NoSuchObject on the service
versus 400 / 400 / 404 here.

Scope, so this is not over-read: 3 × 256 B parts (768 B merged), default schema, sequential requests.
Real 64 MiB chunks, large part counts, concurrent retries and server-side GC timing were not measured.

[dingxin-tech]

This changes the externally visible contract of a failed merge by deleting uploaded part resources after a size or MD5 mismatch. The PR says the previous retention assertion was a guess, but the new consumption behavior also lacks a real MaxCompute service observation. Because the parts cannot be recovered after deletion, please pin the service's behavior for both failure types (and document any intentional emulator divergence) before changing this contract. The Python probe should follow that verified behavior rather than define it.

[TonyStack2026]

这条我按你说的顺序做了：先去钉服务端的真实行为，再让探针跟着结果走。结论跟这个分支之前的说法两处不一致，两处都改了。

在真实 MaxCompute 项目上实测（三层项目、临时资源由探针自己创建并删除，2026-10-04 19:23–19:26），合并请求被拒之后把分片读回来：

合并请求	服务端答	之后分片还在吗
摘要与字节数都对	接受，正式资源发布	被消费
摘要与拼出来的 payload 不符	ODPS-0421213 Save resource error - Merge part temp files failed!	被消费
manifest 点名一个从没上传的分片	ODPS-0421111 Resource not found	保留（那个真分片还在）
目标资源已存在	ODPS-0421121 The resource has already existed	保留
申报的 x-odps-resource-merge-total-bytes 报错数、摘要正确（304 字节申报成 4400）	接受，目标就是正确的 304 字节	被消费

两个结论：

1. MD5 不符的合并消费分片，这个是对的——服务端真的会连着分片一起拒掉。所以 SDK 用同一套确定性分片名重试时必须重传，模拟器不该是那个悄悄留着旧分片的地方。这一条我保留。
2. 申报字节数与实配合并长度不符，服务端根本不检查。它只拿申报值去比项目的上限，完整性判断在 MD5 上。也就是说这条 400 是本模拟器的本地严格检查，服务端没有对应失败，我原来在这个路径上顺手消费分片就没有依据了——现在这条路径改成拒绝但保留分片。这个分歧写进 docs/protocol.md，不再靠猜。

一处我明确没测也没改：目标已存在时服务端给的错误码是 ODPS-0421121，模拟器回 400 InvalidParameter，这个状态差异是既有行为；我没回读服务端的 HTTP 状态，所以探针里这一条只断言"被拒 + 分片保留"，不断言状态码，注释里写清了原因。

测试两边都钉住：internal/server/resources_test.go 里摘要失败的合并继续断言分片被消费，申报字节数不符的合并改成断言分片仍可读（200）；tests/python/run.py 加三条（摘要拒/消费、读数前就定案的拒/保留、申报不符/保留），探针的残留清扫按用例自己的资源名限定，一个客户端中断的上传不会再被算到另一个客户端头上。

验证：go test -race -count=1 ./internal/...（engine 3.03s、server 35.81s、wire 1.28s）全 ok；用这棵树新构建的二进制跑 PyODPS 探针 33 passed / 0 failed（原来 30 条）。新 head 1eb8111。

没合并、没 tag，等你复审。