test(cose): publish the COSE negative conformance vectors (#243 phase 3) - #302
Conversation
Phase 3 of the ADR-0011 migration, tracked in agentrust-io#243. Adds AM-VEC-COSE-002 through 008 alongside the positive encoding vector published in phase 2. The first four are the cases named in section 9 of the envelope specification: a tampered protected header, an alg substituted between the protected and unprotected headers, a typ mismatch, and an unprotected header injected before signature verification. The last three cover places a CBOR implementation genuinely differs, so they catch cross-language divergence rather than restating v0.1 behaviour: an untagged structure, trailing bytes after the object, and a detached payload. WHY NEGATIVES CARRY NO PINNED ENCODING The positive vector pins every element, because an implementation has to produce those exact bytes. A negative cannot: its bytes are malformed by construction, so pinning their decomposition would assert that a verifier can parse something it is being told to reject. Negatives therefore carry envelope_hex and an expected result, and no expected.cose block. The consequence is that the schema records that a manifest is rejected, not why. Every structural rejection maps to MISMATCH, so a verifier that rejects one of these for the wrong reason still passes. Rather than add an expected reason, which would change a published contract, each vector names the rule under test in its description and spec_refs, and a new test asserts that every mutated envelope still decodes as a four-element CBOR array. Without that a vector could quietly degrade into "rejected because it was garbage" and still look like it was testing its rule. ONE VECTOR EXPECTS VALID AM-VEC-COSE-005 injects an unprotected header after signing and expects VALID. That is the same rule as 003 from the other side: nothing in the unprotected header is covered by the signature, and section 6 step 7 evaluates it last, so an injection must not change the verdict. A verifier that merged the two halves, or read kid from the malleable one, fails it. NO POST-QUANTUM VECTOR ML-DSA-65 signing is hedged. cryptography exposes sign(data, context=None) with no deterministic mode, and signing the same bytes twice produces different signatures, so a post-quantum envelope cannot be pinned the way AM-VEC-COSE-001 is. A vector whose bytes changed on every regeneration would be a snapshot rather than a contract, and shipping a private seed for consumers to regenerate against would break the suite's rule that no private key material is written to disk. The README records this rather than leaving the omission to be rediscovered. NOT INCLUDED Regenerating the suite also updates all twenty v0.1 vectors, because intent joined SIGNED_FIELDS in 0.11.0 after those vectors were last generated. That drift is present on main independently of this change and is left out here to keep the diff scoped. VERIFICATION 874 passed, 6 skipped. 37 conformance tests. mypy strict, bandit, and ruff under the rule set CI applies are all clean. Every vector is byte-identical on regeneration from the fixed seed. Refs agentrust-io#243 Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
|
🟡 Contributor Check: MEDIUM
Automated check by AgenTrust Contributor Check. |
github-actions
Bot
added
the
needs-review:MEDIUM
imran-siddique
left a comment
imran-siddique
left a comment
There was a problem hiding this comment.
The seven new envelope-shape vectors are well constructed, but this cannot yet serve as the phase-3 portable contract described by the PR and issue. The phase-2 security follow-up on #243 explicitly assigns five additional negative vectors to phase 3: (1) unauthorized issuer/signing-key pairing, (2) duplicate JSON member names, (3) non-finite JSON numbers, (4) manifest-version/envelope mismatch, and (5) excessive payload nesting. None is present in the generator or index.
These are not generic v0.1 repetitions: each covers a COSE-path defect found during phase 2 that another-language implementation following the earlier envelope document could reproduce. In particular, outcome conformance without the issuer/key vector does not enforce the authorization boundary, and ordinary JSON parsers commonly diverge on duplicates/non-finite values.
Please add deterministic COSE vectors for all five cases, list them in the index/README, and include them in the regeneration and reference-verifier tests. Each vector should isolate its named defect where feasible (valid signature over the malformed/unauthorized payload rather than incidental signature failure). CI is green for the current set, but the claimed cross-language security contract is incomplete without the cases already recorded in #243.
…ase 3 Issue agentrust-io#243's phase 2 security follow-up names five negative cases that the first submission did not carry, and without them the suite is not the portable contract phase 3 is supposed to publish. It could prove an implementation accepted a valid envelope and rejected a malformed one, but said nothing about the authorization boundary or the two places JSON parsers are known to disagree. AM-VEC-COSE-009 is a trusted key that is not authorized for the manifest's issuer. Its envelope is byte-identical to AM-VEC-COSE-001 and only context.trusted_key_issuers differs, so nothing about the object can explain the rejection. A verifier that stops at "the signature verifies under a trusted key" returns VALID and has no authorization boundary at all. It is the one negative that reports signature_verified true. AM-VEC-COSE-010 and 011 are the parser divergences RFC 8259 leaves open: a duplicate member name, where section 4 says the behaviour is unpredictable and the second value is attacker-chosen, and the literal NaN, which section 6 admits no form of but several parsers accept as an extension, Python's own among them unless told otherwise. AM-VEC-COSE-012 puts a 0.1 payload in a 0.2 envelope, which section 6 step 3 routes on the payload rather than the envelope, so it is the one negative expecting INCOMPATIBLE_VERSION. 013 nests past the depth bound, which has to produce a verdict rather than exhaust the stack. All five sign over the payload under test rather than swapping bytes into an already-signed envelope, so each carries a valid signature and the rule it names is the only thing left to reject on. 010, 011 and 013 place their defect inside attestation, which the schema types as free-form, so the rest of the document is valid too. Every negative now records signature_valid, and two tests hold it to account: one asserts it is true except on the two vectors that declare false by design, and one re-derives it from the bytes on disk using only the published public key and the RFC 9052 Sig_structure, the way a foreign implementation would. That is what keeps a vector from quietly degrading into a signature-failure test. Also adds a regeneration test that rebuilds every vector in memory and diffs it against the committed copy. It found real drift on main: the v0.1 vectors have published a signed_fields list without intent since 0.11.0 added it, and AM-VEC-018 carried the manifest hash that followed. No signature or expected result moves, because none of those manifests declares an intent and signing_pre_image omits absent fields, so the signed bytes are the same either way. What was wrong is what the suite told other languages to build their pre-image from. Refs agentrust-io#243, agentrust-io#302 Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
The review asks that each vector isolate its named defect rather than being rejected incidentally. signature_valid already rules out one way that can go wrong, an incidental signature failure, but it says nothing about a second defect elsewhere in the payload. This asserts the property directly: strip only the defect a vector names, and the envelope must verify VALID. If anything else about it were also wrong, the repaired object would not verify and the test says so. It matters most for 011 and 013, which hang their defect on an attestation object. The carrier is asserted benign on its own, so neither vector can be passing on the carrier rather than on the rule it names. Refs agentrust-io#243, agentrust-io#302 Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
Re-reading the phase 2 findings on agentrust-io#243 rather than the summary of them surfaced two things the five vectors did not cover. The first is fixed here. agentrust-io#274 made the version gate bidirectional, and the follow-up records both directions: a 0.1 payload must not be verified under 0.2 rules, and a 0.2 manifest must not fall back to the v0.1 detached signature block. AM-VEC-COSE-012 was only the first. A one-way gate is not a gate, since anyone unable to produce a valid COSE envelope would present the manifest in the envelope still accepted, so AM-VEC-COSE-014 pins the second. It is the only vector in the COSE series carrying `manifest` rather than `envelope_hex`, because the rule under test is precisely that this document must not be accepted outside a COSE envelope. It expects MISMATCH rather than INCOMPATIBLE_VERSION: the verifier does support 0.2, and what it declines is verifying 0.2 through the v0.1 path, so an unsupported-version result would state something untrue about its capabilities. The same document at version 0.1 is AM-VEC-001 and verifies VALID, which the isolation test asserts. The second cannot be fixed and is documented instead. The hybrid authorization case, one authorized component key alongside one unauthorized one, needs a COSE_Sign carrying an ML-DSA-65 signature. That signing is hedged and cryptography 49 exposes no deterministic mode, so the bytes differ on every regeneration, and exempting a vector from the regeneration check would remove the guarantee that makes the rest of the suite worth trusting. It stays covered by a per-run test in the Python suite, and the vector README now states the rule as binding on other languages so it is not read as out of scope. Also corrects the AM-VEC-COSE-010 rationale, which described parser behaviour more specifically than had been established. RFC 8259 section 4 and RFC 8785 carry the argument without the claim. Refs agentrust-io#243, agentrust-io#302 Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
Auditing the vectors against the review rather than against my own summary of it showed 014 was the weakest of the six. It carried no signature_valid and picked up two of the reference-verifier tests instead of four, because both signature checks selected on the presence of envelope_hex and 014 is the one COSE-series vector whose subject is a manifest document. Its signature was in fact valid, so the vector was sound, but nothing said so and nothing checked it. That is the same gap signature_valid exists to close everywhere else: without it a verifier could pass 014 by rejecting the signature and never reaching the version rule. It now declares signature_valid over the RFC 8785 pre-image its envelope actually signs, and the two checks select on the vector id rather than on the key under test, so a vector that forgets to declare it fails instead of quietly dropping out of the check. Confirmed by removing the key and watching both tests fail. The CBOR decodability check still does not apply to 014, correctly: it asserts a mutated envelope remains a four element CBOR array, which says nothing about a manifest document. Refs agentrust-io#243, agentrust-io#302 Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
index.json declared spec_version 0.1 and described every vector as carrying a manifest, while fourteen of the thirty four target the v0.2 COSE envelope. It is the first file a consumer in another language reads, so it was the worst place for that to be wrong. Nothing reads the field today, so this is safe to correct now rather than after phase 4 has consumers. Refs agentrust-io#302 Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
…o#243 Finding 2 on agentrust-io#243 records NaN and Infinity as one class of defect. They are one class but not one code path: a parser hook that special-cases NaN passes AM-VEC-COSE-011 and still accepts Infinity, which is exactly the divergence these vectors exist to catch in another language. AM-VEC-COSE-015 closes it. Built the same way as 011, on the same free-form attestation carrier, so the literal is the only defect and the signature over it is valid. -Infinity travels the same path as Infinity in every parser checked, so it is covered here rather than given a third vector. Refs agentrust-io#243, agentrust-io#302 Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
Only CHANGELOG.md conflicted, and only because both sides opened a new ## Unreleased / ### Added block. Both entries are kept under one heading, the TPM attest parsing from agentrust-io#304 first since it landed first, then the COSE conformance vectors. No code conflict. The 35 vectors still regenerate byte-identically after the merge, so nothing on main moved the engine's normative behaviour. Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
|
All five added, plus two more the findings on #243 imply. Each is in the generator, the index and the README, and each is signed over the payload under test so the rule it names is the only reason to reject it.
009 is byte-identical to 014 and 015 are not in your list, but both are in the findings.
Isolation is asserted, not claimed. Every negative carries One case cannot be a vector. The hybrid half of finding 1, one authorized component key alongside one unauthorized one, needs an ML-DSA-65 signature. That signing is hedged and Two fixes found along the way. A new regeneration test caught pre-existing drift on One thing to decide separately. 35 vectors, 935 passed and 6 skipped, green across Python 3.11 to 3.13 on Linux, macOS and Windows. Merged |
It was appended before 014, so index.json listed the suite out of order. The index is the catalogue a consumer in another language reads first, and a catalogue that skips and doubles back reads like an oversight. Refs agentrust-io#302 Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
…io#306 agentrust-io#306 added `profile` and `unbound_artifacts` to SIGNED_FIELDS and touched no vector files, so the committed vectors again published a signed_fields list that disagreed with the SDK. Same shape as the `intent` drift from agentrust-io#288 that this branch already fixed once. The regeneration test added earlier in this branch caught it on the merge rather than after release, which is what it exists for. Nothing verifies differently: no signature, no envelope bytes and no expected result moves, because none of these manifests declares either field and signing_pre_image omits absent fields. What was wrong is the list other languages build their pre-image from. CHANGELOG conflicted for the same reason as the previous merge, both sides opening an Added block. Resolved keeping the composition-only entry from agentrust-io#306, the TPM entry once, and the COSE vectors. Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
|
Merged
Nothing verifies differently, which is why this is easy to miss. "crypto_profile",
+ "profile",
+ "unbound_artifacts",
"artifacts",What was stale is the list another language would read to build its pre-image, which is the part the vectors exist to publish. 944 passed and 6 skipped, vectors regenerating byte-identically. |
…io#307 agentrust-io#307 added `source_bundle` to SIGNED_FIELDS and touched no vector files, so the committed vectors again published a signed_fields list that disagreed with the SDK. Third time in this sequence: `intent` in agentrust-io#288, `profile` and `unbound_artifacts` in agentrust-io#306, `source_bundle` now. The regeneration test caught it on the merge again. Nothing verifies differently: no signature, envelope bytes or expected result moves, because none of these manifests declares the field and signing_pre_image omits absent ones. What goes stale is the list another language would read to build its pre-image. Three occurrences in three weeks is a property of the workflow rather than of any one change: adding to SIGNED_FIELDS breaks nothing, fails no test, and leaves the vectors quietly disagreeing with the SDK. Once this branch lands the test runs on main and the next one fails in its own PR. CHANGELOG conflicted for the same reason as before. Resolved keeping the Agent Plugins entry from agentrust-io#307, composition-only from agentrust-io#306, TPM once, and the COSE vectors. Signed-off-by: Mohammed Zoheb Shaik <zoheb.shaik7@gmail.com>
|
Merged
Nothing verifies differently in any of the three. Three times in three weeks looks like a property of the workflow rather than of any one change: adding to 958 passed and 6 skipped, vectors regenerating byte-identically. |
imran-siddique
left a comment
imran-siddique
left a comment
There was a problem hiding this comment.
The requested phase-3 coverage is now present: unauthorized issuer/key binding, duplicate JSON members, non-finite numbers (NaN and Infinity), version/envelope mismatch, and excessive nesting. Focused vector suite: 86 passed; exact CI lint rule passed. The full local suite is shadowed by an unrelated installed #256 checkout, so hosted CI remains the complete-suite authority. Approving.
imran-siddique
added
the
cosai-ws4
3 participants
Summary
Phase 3 of the ADR-0011 migration, tracked in #243. Publishes the COSE negative conformance vectors,
AM-VEC-COSE-002through008, alongside the positive encoding vector that landed with phase 2.The vectors are the portable contract, so these are what gate any other-language SDK.
What is covered
AM-VEC-COSE-002MISMATCHAM-VEC-COSE-003MISMATCHalgin the unprotected header is rejected, never readAM-VEC-COSE-004MISMATCHtypalias is refusedAM-VEC-COSE-005VALIDAM-VEC-COSE-006MISMATCHAM-VEC-COSE-007MISMATCHAM-VEC-COSE-008MISMATCHThe first four are the cases named in section 9 of the envelope specification. The last three cover places a CBOR implementation genuinely differs, so they catch cross-language divergence rather than restating v0.1 behaviour that the existing vectors already pin.
Three decisions, raised rather than assumed
Negatives carry no pinned encoding
The positive vector pins every element, because an implementation has to produce those exact bytes. A negative cannot: its bytes are malformed by construction, so pinning their decomposition would assert that a verifier can parse something it is being told to reject. Negatives therefore carry
envelope_hexand an expected result, and noexpected.coseblock.The consequence is that the schema records that a manifest is rejected, not why. Every structural rejection maps to
MISMATCH, so a verifier that rejects one of these for the wrong reason still passes.Two things mitigate that without changing the schema. Each vector names the rule under test in its
descriptionandspec_refs. Andtest_cose_negative_vector_is_not_silently_unparseableasserts that every mutated envelope still decodes as a four-element CBOR array, so a vector cannot quietly degrade into "rejected because it was garbage" while still appearing to test its rule.Where a vector can isolate its rule, it does.
AM-VEC-COSE-004is signed over the aliased header rather than havingtypedited into an already-signed one, so its signature is valid and thetypvalue is the only defect. Built the other way, an implementation that verified the signature and never implemented thetyprule would have passed it. Each negative was checked individually against the reference verifier to confirm it is refused for the reason it names rather than incidentally:An optional
expected.reasonwould close the gap properly, but that is an addition to a published contract, so I would rather raise it here than assume it. Happy to add it if you want the negatives to assert cause as well as outcome.One vector expects
VALIDAM-VEC-COSE-005injects an unprotected header after signing and expectsVALID. That is the same rule as003seen from the other side: nothing in the unprotected header is covered by the signature, and section 6 step 7 evaluates it last, so an injection must not change the verdict. A verifier that merged the two halves, or readkidfrom the malleable one, fails it.It sits with the negatives because it tests the same normative sentence, but it is worth knowing before someone reads a
VALIDin a file named for a negative case.No post-quantum vector
ML-DSA-65 signing is hedged.
cryptographyexposessign(data, context=None)with no deterministic mode, and signing identical bytes twice produces different signatures, so a post-quantum envelope cannot be pinned the wayAM-VEC-COSE-001is.A vector whose bytes changed on every regeneration would be a snapshot rather than a contract, and shipping a private seed for consumers to regenerate against would break the suite's own rule that no private key material is written to disk. The README records this so the omission is a decision rather than something rediscovered later.
Not included
Regenerating the suite also updates all twenty v0.1 vectors, because
intentjoinedSIGNED_FIELDSin 0.11.0 after those vectors were last generated. I confirmed that drift is present onmainindependently of this change, and left it out to keep this diff scoped. Happy to send it as its own PR.Consuming a negative vector
tests/vectors/README.mddocuments the shape. It is the same as a positive vector, decodeenvelope_hexand run the verifier over the bytes; only the assertion differs. The two properties above, that negatives state outcome rather than cause and that one of them expectsVALID, are called out there as well, since the README is what another language reads.Verification
--select E,F,W --ignore E501as CI runs itRefs #243