feat(xdna): optional explicit AMD XDNA2 lane for the GLM shared expert - #1261
Open
Kenneth-Javier wants to merge 20 commits into
Open
feat(xdna): optional explicit AMD XDNA2 lane for the GLM shared expert#1261Kenneth-Javier wants to merge 20 commits into
Kenneth-Javier wants to merge 20 commits into
Conversation
Colibri gains the host side of an optional AMD XDNA2 (Ryzen AI NPU) compute
lane. XDNA is a lane, never an engine: the model, router, expert identity,
weight ownership, scheduling and fallback all stay where they are.
c/backend_xdna.{h,c} resolve an optional coli_xdna.dll at runtime and link
nothing, the way backend_loader.c resolves the GPU backend. Neither file
includes or links XRT, so an ordinary build has no XRT header, no import
library and no DLL import, and a machine with no helper, no XRT and no NPU is
a normal machine rather than a broken one.
Binding is all-or-nothing. A helper is usable only when it loads, reports the
expected ABI generation and exports every entry point this host requires; any
failure clears the callable pointers and releases the module, so nothing can
outlive its rejection. The verdict is sticky, so a host without a helper pays
one lookup rather than one per operation, and the lookup is an absolute path
beside the executable -- no PATH search and no current-directory search.
A successful bind means HELPER_ABI_AVAILABLE and nothing more. Device
discovery, artifacts, weight preparation, dispatch, matmul interception and
scheduling policy are all absent by design, as is any user-facing switch:
there is no operation to offer yet. colibri does not link the object either,
so the default binary is byte-for-byte what it was; `make xdna-obj` builds the
host side on its own.
The whole contract is qualified without hardware. test_backend_loader.py, which
already builds synthetic DLLs and inspects imports for the GPU loader, gains
the XDNA cases: helper absent, good helper, wrong ABI generation, missing
required entry point, present-but-unloadable helper, repeated probes, three
shutdown orders, and a dependency inspection proving the ordinary binary
imports neither XRT nor the helper. The fixtures contain no XRT.
No real XRT-linked helper is built here: binding is fully covered without one,
and dead XRT-linked code would contaminate the build for no gain.
Colibri gains the engine-owned half of XDNA artifact selection: which artifact answers which operation, and whether that artifact can be trusted. The helper owns neither, and will eventually receive a choice already made and verified. A registry row records what research established about one artifact -- semantic family, the exact M bucket it was compiled for, K and N, activation, prepared weight and output dtypes, target device family, logical filenames for the xclbin and its instruction stream, the SHA256 of each, and four independent qualification facts. Lookup requires an explicit semantic family, so two operations with identical M/K/N cannot inherit each other's qualification: shape alone is never eligibility. Buckets are matched exactly, because research qualified specific M values and nothing here interpolates between them. Both artifact files must exist and both hashes must match, and presence and integrity stay distinct verdicts -- a missing file means this build does not ship that artifact, a hash mismatch means the bytes are not the bytes that were qualified. Qualification is checked before the filesystem, so a row that was never correctness-qualified declines whether or not its bytes are intact. Research measured a design that compiled, loaded, dispatched to completion, returned finite numbers and was numerically wrong, so a successful dispatch is not evidence of a correct one and is not an input here. SHA256 is implemented in backend_xdna.c rather than taken from a system library. BCrypt is available to the Windows toolchain, but using it would add -lbcrypt to the host link line for a lane the host does not link yet, and would fork otherwise portable registry logic along a platform boundary it has no other reason to have. The NIST vectors cover it, including the padding boundaries where naive implementations lose a block. The strongest verdict reachable is STATIC_ARTIFACT_QUALIFIED, and the name says static because that is all it is. No device is opened, no weight prepared, no pointer aligned, no memory checked and no economics consulted -- helper availability, artifact qualification, device readiness, prepared-weight validity and economic preference remain five independent concepts. The registry names the first family's qualified artifacts; the build does not contain them. Where they should live is a packaging decision that has not been made. An absent artifact yields ARTIFACT_UNAVAILABLE and the operation continues on its current path, which is the intended state rather than an error. Logical names resolve under a caller-supplied root, and a name that tries to escape that root is rejected during validation, before it can reach the filesystem. The default binary is unchanged and still imports no XRT.
The XDNA lane gains the engine-owned host state a later fmt4 conversion will publish into: an aligned allocation, an explicit validity state machine, a release path and host-byte accounting. Nothing is converted, no device is opened, no pointer is wrapped and the helper is never called. Three properties vary independently and are deliberately not collapsed into a flag: whether memory is allocated, whether its contents are valid, and how many host bytes are consumed. An invalid buffer still costs memory, and no amount of successful allocation makes contents valid, so allocation cannot imply validity anywhere in the API. PREPARED_VALID is reachable only by publishing success from PREPARING. An invalid image can never shortcut back to valid; it must go through a complete new cycle. A writable destination exists only while PREPARING, so a published image cannot be rewritten behind its own back, and a poisoned buffer that failed publication cannot be republished. Allocation reuses the repository's existing posix_memalign and compat_aligned_free pair rather than introducing a second aligned-allocation convention, and guarantees 4096-byte alignment for the qualified Windows userptr path. The payload size is never rounded up behind the caller: a 30-byte payload reports 30 bytes and still gets an aligned pointer. Sizes use checked arithmetic, because an unchecked product wraps into a small plausible number and the allocator then succeeds with a buffer far too small for what the caller will write, which is worse than failing. A defensive alignment validator sits beside the allocator guarantee, since a buffer arriving from a pool or at an offset does not carry it, and a misaligned pointer fails at the XRT boundary with a message about video memory that points at entirely the wrong subsystem. The bytes are host memory. They are not VRAM, not NPU memory and not an XRT device allocation. QT gains one opaque pointer under COLI_XDNA, starting NULL. Loading a model allocates none of it and a registry query allocates none of it; only an explicit request does. When an expert slot is reused for a different expert the derived image is dropped, exactly as the GPU tiers drop theirs, because a stale prepared image would be a silently wrong weight rather than a missing one. The default build defines nothing, so QT and the binary are unchanged. The loader now includes compat.h, which the project mandates be compiled with _FILE_OFFSET_BITS=64, so the standalone-compile test compiles it the way the build actually does rather than more leniently.
The XDNA lane can now build its prepared weight. The authoritative fmt=4 grouped-int4 tensor is converted directly into the BF16 B[K,N] image the qualified artifact expects, written straight into the aligned destination the previous slice owns. The source is read and never modified. Semantics were taken from the production kernel rather than from a description of it: row bytes (I+1)/2, groups (I+gs-1)/gs, weights stored [O][I], even input index in the low nibble and odd in the high, value (nibble - 8) times the group scale. Destination indexing is dst[i*O + o], so the [O,I] to [I,O] transform is inherent in the addressing and needs no intermediate matrix; values pass through one float scalar, so no full-sized FP32 image is ever allocated and the only allocation a conversion makes is the BF16 destination itself. Float to BF16 rounds to nearest even rather than truncating. The two differ only on exact ties, and every value this converter produces is exactly representable, but the rule still has to be the qualified one and is tested at the tie boundaries. Correctness is checked against two independent oracles: a reference transform written from the format contract, and the production matmul_i4_grouped kernel driven with one-hot activations so it yields the decoded weights directly. Both agree bit for bit, across odd I, partial trailing groups, group boundaries at 63/64/65 and a single-element tensor. Everything rejectable is rejected before the preparation cycle opens, so a bad source never strands an object mid-cycle, and no path returns while still preparing. A mid-conversion failure leaves genuinely partial bytes and forces the invalid state; those bytes carry no authority, because state decides validity rather than contents, and the partial image cannot be published. A failed buffer keeps its capacity and is reusable only through a complete re-preparation, which produces an image bit-identical to one prepared from scratch. The conversion is serial. The frozen contract permits parallelisation but does not require it, and publication semantics matter more here than throughput; a 6144x2048 weight converts in well under a tenth of a second. Conversion needs no helper, no artifact, no XRT and no device. The helper ABI is unchanged and still knows nothing about nibble packing or group scales, and the default binary continues to import no XRT.
Connect the optional XDNA2 lane to one real operation: the MoE shared-expert gate and up projections, at K=6144 N=2048, fmt=4 with group size 64, logical M 1..64 padded to the qualified M=64 artifact bucket. The engine keeps every decision that needs to know what the operation means -- semantic family, artifact selection, integrity, weight representation, hard eligibility and fallback. The helper is handed an already-selected, already-verified artifact and an already-prepared buffer, and executes. No engine type, XRT type or C++ exception crosses the ABI. Hard eligibility now combines the semantic, artifact, representation, alignment, helper, device and runtime gates, evaluated cheapest-first so an operation that could never run costs a few comparisons. It stays strictly separate from economic preference, which does not exist yet: there is no --xdna flag, no COLI_XDNA variable and no automatic policy, so an ordinary build runs exactly the path it runs today. The lane is reachable only from an internal test control. Any refusal or helper failure returns "not handled" and the caller runs its current matmul_qt path. The candidate never calls matmul_qt itself, and the output buffer is written only after successful completion, so no failure can leave a half-written result behind. sh_down and generic matmul_qt are not intercepted, and family is passed explicitly rather than inferred from shape. Helper ABI generation 1 -> 2 is a deliberate compatibility break; a generation-1 helper is refused outright rather than partially bound. The default build is unchanged: no XRT header, no XRT link, 12 imports, no XRT DLL, and coli_xdna.dll is not required to exist. Artifact bytes are still not shipped. Also fixes a latent registry defect: g_nrows initialised to 0 left the production table installed but empty, which was invisible while every caller was a test that installed its own registry.
Close the failure contract for the optional XDNA lane. The invariant is that optional XDNA work may fail but current Colibri operation semantics may not, so every decline and every runtime failure returns to the exact matmul_qt call that stood at the call site before the lane existed. No CPU-specific fallback, no partial-result salvage, no recursion, no double dispatch. Failure stages are now classified rather than collapsed. An absent artifact and a tampered one are different verdicts, as are a missing helper and an incompatible one; the sub-reason was already known and was being discarded exactly where an operator needs it. XDNA output is valid only after successful completion. A failure after the helper has already written a full, finite, plausible result still leaves the output invalid and the caller buffer untouched -- structurally, since the caller buffer is never handed to the helper at all. No NaN or finiteness scan defines correctness anywhere in the lane. Lane health is the narrowest model the failures justify: device-init failure is process-scoped, everything else is one shape or one operation, and a full lane teardown is the only thing that clears it. Runtime failure no longer implies anything about the prepared BF16 image, which stays VALID and reusable. Fixes a correctness defect in the I5 lane, confirmed on real XDNA2 hardware before and after: the userptr wrapper was keyed on the prepared pointer alone. Retained capacity is reused in place, so re-preparing a different weight kept the same address, the re-wrap was skipped, and the device -- which snapshots at wrap time -- computed against a view that no longer matched the engine image (46431 of 131072 elements outside the acceptance criterion). Every publication now bumps a generation, the wrapper is keyed on (pointer, generation), and the engine releases the wrapper before freeing or invalidating the memory it borrows, so it can never outlive or alias released engine memory. Also fixes stale output validity when the seam declined before running the core, and covers the compiled-in production registry with a regression that reads it with no test rows installed. Adds an internal explicit mode as a separate entry point rather than a mode flag, so no global state can leave the production seam in a no-fallback configuration. Both modes share one implementation. Default behaviour is unchanged and stays that way: with a helper present, a device available and valid artifacts staged, an ordinary build still runs the current path and dispatches zero XDNA operations. There is no economic policy and no public XDNA control. Whether BF16 activation semantics are acceptable at model level remains unqualified and gates any future automatic selection.
Integration adaptation for current dev. Not a new capability: it keeps the already-qualified prepared-weight converter applied only to the byte layout it was qualified against. fmt=4 now has two in-memory layouts. The classic pair layout puts elements 2j and 2j+1 in byte j. The K1 planar layout puts elements k and k+32 in byte k of each 64-element block, and qt_planarize() rewrites the tensor in place when the grouped planar IDOT path is opted into. It applies to exactly the gs>=64 tensors this lane qualifies, including the shared expert weights, and QT now carries a planar flag saying which layout the bytes are in. The converter reads the pair layout. Planar bytes would decode to nonsense rather than fail, which is the worst available outcome, so the hard-eligibility gate now refuses them and the operation falls back to the current path. The flag is passed explicitly from the call site, like the semantic family, so it cannot be inferred or forgotten. Adding planar support is a separate question with its own qualification; it is deliberately not answered here.
The supported-surface table said "group size 64" without saying which of the two fmt=4 byte layouts the lane accepts. Name it, explain why the distinction exists, and add the layout gate to the documented gate order and failure-class list.
The optional XDNA lane served logical M 1..64 against the F3 M64 artifact and
declined everything above it. The F3 M256 artifact for the same shape was
already listed in the production registry with verified hashes and all four
research qualifications, but no code path could select it.
Add a single deterministic selector:
M 1..64 -> F3 M64
M 65..256 -> F3 M256
M > 256 -> decline, exact current path
This is shape eligibility, not economics: the selector has no cost model, no
device-load input and no preference, and no caller can override it. There is
deliberately no heuristic of the form "this pads a lot, prefer the current
path" -- padding cost is real but acting on it would be an economic decision.
Buckets are not a range to interpolate across. An M bucket is a separately
compiled program; N6 saw one shape family fail to compile at M256 while its
M64 sibling compiled. So the selector returns the smallest COMPILED bucket that
holds the logical rows, and a bucket it names but the registry does not hold
declines rather than silently using a different one.
The lane was already generic in row->artifact_m, so the change is small:
the constant that meant both "largest logical M" and "the bucket" is split
into the two meanings it had conflated, and the two sites that hard-coded 64
now consult the selector. Staging, padding, copyback, artifact reopen, registry
lookup, prepared-state ownership and wrapper lifetime were already generic and
are untouched. The helper ABI already carried M, so it does not change.
Per-bucket counters are added because a single padded_ops total cannot say
which artifact ran and weights a 1-row pad the same as a 191-row one. They are
qualification diagnostics on stderr, not a product interface.
No new precision contract: both rows are BF16/BF16/F32 over the same fmt4 gs64
source with the same RNE preparation and the same fallback. Default behaviour
is unchanged -- there is still no public XDNA control, and the default build
contains no seam.
The optional XDNA lane has been internally qualified for some time -- real
GLM-5.2 execution on both artifact buckets, device correctness closed against
the project's own BF16 oracle -- but it had no product surface, and no way for a
shipped build to find its artifacts. Every qualification run supplied the
artifact root through test-only instrumentation, which hid the second problem
until the first uninstrumented product invocation returned ARTIFACT_UNAVAILABLE.
This adds both halves.
EXPLICIT INTENT
coli --xdna -> COLI_XDNA=1
Default off. Never inferred from hardware, a loadable helper, valid artifacts or
a successful probe -- the reduced-precision consequence means the user has to
ask. It is a distinct flag rather than a --policy value because --policy chooses
how the planner may spend memory and this chooses a compute device; someone who
wants quality planning ON the NPU must be able to say both. COLI_XDNA matches
the COLI_CUDA / COLI_METAL / COLI_VULKAN convention, reusing the compile macro
name exactly as those do.
Product intent is a separate switch from the qualification force seam. Sharing
one would give a user-facing option a path to the control that forces
unqualified behaviour.
Permission is not selection: enabling permits the qualified lane, it does not
push work onto the device. Every hard gate still decides, so an unsupported
shape or family still runs the current path however explicit the request was.
PACKAGED ARTIFACTS
<exe-dir>\coli_xdna.dll
<exe-dir>\xdna\<the four registry-named artifacts>
resolved absolutely from the executable directory, using the same anchor the
helper already uses -- no PATH, no working directory, no search. Every byte is
verified against the SHA256 values already compiled into the registry before
use, and the whole set is checked at activation rather than the first row, so a
package missing only the larger bucket is a diagnostic before the model loads
instead of a failure on the first long request.
Presence of a complete valid package enables nothing.
DIAGNOSTICS
The reduced-precision warning is emitted only after the package resolves and
every artifact verifies. Build-unsupported, package-missing, package-incomplete,
integrity-failed and helper-unavailable each get their own message naming the
resolved path or the offending file, because they are five different problems
with five different fixes. Integrity failure is fail-closed: bytes that are not
the qualified bytes are not the qualified lane, and asking for acceleration is
not evidence about what they do.
Everything goes to stderr. SCORE writes machine-readable results to stdout and
its parser consumes any line beginning with a digit or a minus sign.
Also adds tools/build_xdna_package.py, which assembles the optional package from
already-built inputs and refuses unless every expected file is present and every
hash matches. It parses the expected names and hashes out of backend_xdna.c
rather than restating them, so the packager cannot drift from the runtime
verifier. It builds no artifacts and downloads nothing.
Default behaviour, default numerics and the XRT-free default host are unchanged.
No automatic selection, no economics, no quality threshold in product code.
The optional XDNA lane had no product build target. `make colibri` never compiled backend_xdna.c, and defining COLI_XDNA through EXTRA_CFLAGS only produced five undefined references at link time, so the Windows release binary could not reach the lane at all -- a release-integration gap, not a source defect. Add an XDNA=1 switch in the same shape CUDA, Metal and Vulkan already use: a switch, a define, and an object contributed to the single colibri rule. There is deliberately no LDFLAGS entry, because the host side links nothing -- backend_xdna.c resolves coli_xdna.dll through LoadLibraryExA at runtime, and only when the user passes `coli --xdna`. An XDNA=1 host therefore has the same import table as a default one and starts normally on a machine with no NPU, no XRT and no helper. Select it from the Windows release row via make_args. The runner needs nothing installed. The helper and the kernel artifacts are NOT built there -- they need XRT and the AIE toolchain -- and continue to ship as a separate optional asset. Document the optional package's runtime prerequisites in docs/xdna.md. The helper imports the MSVC C++ redistributable as well as XRT, which was not previously recorded; without it the helper fails to load and `--xdna` falls back to the normal path with a diagnostic that gave no hint of the cause. The core host requires neither, with or without the capability compiled in. No change to XDNA execution semantics, the artifact registry, the helper ABI, the artifact bytes, or the product CLI. The default build is unaffected: a default host built from this tree is identical to one built from the previous source apart from the PE build timestamp.
The fmt4 -> BF16 prepared-weight converter ran as a serial loop over K*N elements while the rest of the engine used the full OpenMP team. On the GLM shared expert (K=6144, N=2048) that is 12.6M elements per tensor and 150 tensors per process, and it dominated the cost of enabling the lane: measured 84.2 ms per tensor, 12.6 s of prepared-state construction charged to the first forward. Parallelise the loop over output columns. Iteration o writes only dst[i*O + o] for i in [0,I), so distinct o never touch the same destination element and completion order cannot affect the result. No reduction, no shared counter and no allocation happens inside the loop, and the prepared object is still published only after the whole conversion succeeds. The per-element arithmetic is unchanged: the loop body was extracted verbatim into coli_xdna_convert_column, so nibble order, (nib-8)*scale, the group index i/gs, the destination index i*O+o and the f2b rounding are what they were. Prepared images are byte-identical -- verified on four fixtures including the real 25,165,824-byte F3 image -- so the device cannot observe this change and the qualified BF16 correctness carries forward. The failure-injection path stays serial. Its early return crosses a loop boundary, which an OpenMP structured block may not, and the degraded-state behaviour qualified against it depends on that exact ordering: rows before the failure point genuinely converted, the rest left as they were. That path only runs when a test has armed the seam, where speed is irrelevant. This adds no dependency. backend_xdna.c is already compiled with -fopenmp and the engine links libgomp statically, so the host import table is unchanged. Measured: 84.2 ms -> 20.3 ms per tensor.
The optional XDNA download had a producer but no release owner. Nothing in the repository said how the sidecar comes to exist, how its bytes are checked, what it should be called, or how it reaches a release -- and the release workflow cannot build it, because the helper needs the XRT SDK and the artifacts need the AIE toolchain that CI does not have. Give tools/build_xdna_package.py the release role it was already half doing. --release produces the archive, a mechanically generated manifest and a checksum in sha256sum format, all named from version.py so no second version string exists to drift. --verify-release re-checks a built archive against its manifest and against the registry parsed out of backend_xdna.c, and refuses a mismatched archive, a manifest naming a different release, or a digest the engine would not accept at runtime. Verification runs before any upload command is printed. The naming keeps the core archive's stem and adds -xdna, so the sidecar sorts beside colibri-<tag>-windows-x86_64.zip, is matched by the release job's own `sha256sum colibri-*`, and cannot be mistaken for the core download. Attachment uses `gh release upload ... --clobber`, the same command the release job uses; this adds no second publication mechanism and no XRT or AIE anywhere near ordinary CI. Document the procedure for maintainers next to the user instructions that already promised a separate optional download, and add an owner covering the asset set, the naming, the manifest and every rejection path -- with synthetic fixtures, so it gates on any machine rather than only on one with an NPU. Also close file handles in the packaging tool's registry and version reads; the new owner imports the module, which turned them into warnings on every run. No change to XDNA math, the helper ABI, eligibility, precision, fallback semantics or the artifact registry.
The physical qualification owner had its M values compiled in as 1, 32, 64.
That covered the M64 bucket and nothing else, so proving the M256 bucket
physically dispatches meant building a throwaway copy of the probe with the
numbers edited -- evidence that cannot be reproduced from the repository.
Take an optional M-list argument instead. The default is unchanged, so every
figure recorded against the old invocation still reproduces from the same
command line, and the M256 bucket is now qualified by the committed owner:
tests/xdna_physical_probe <artifact-root> <helper> 65,130,256
Nothing about the lane itself moves. The probe still drives the production
registry, integrity check, loader, weight preparation, hard-eligibility gates
and candidate function; only the shapes it asks for are now chosen by the
caller. Passing an M outside the qualified range is a useful control in its
own right -- it reports M_OUT_OF_RANGE, DECLINED and zero dispatches.
Also fix the summary line that claimed one artifact serves every M, which
stops being true the moment a caller mixes buckets in one list.
Two release-engineering gaps, found while freezing the Windows XDNA candidate. clean.py listed the engine binaries by hand and had never gained qwen36. So `make clean` left an old qwen36 in place, `make qwen36` then saw an up-to-date target and did nothing, and the release Package step copies whatever sits at c/qwen36 without asking where it came from. A maintainer staging a release locally could ship an engine that was never rebuilt; CI could not, because CI starts from an empty runner. Derive the engines from family_registry instead -- the same registry the release job already uses to decide what an archive must contain -- so the two sets are equal by construction rather than by vigilance. The physical qualification probe had the same problem for a different reason: tests/xdna_physical_probe matched none of the test_/bench_/fuzz_ globs and survived every clean. A stale one does not merely waste space, it reports PASS for code that is no longer in the tree. Second gap: the optional sidecar archive was content-stable but not byte-stable. ZipFile.write stores each member's mtime and derives create_system and the external attributes from the host, so the same five files staged from two directories produced two different archives -- which is exactly what happened between the last two qualification passes, for identical content. Write the members through explicit ZipInfo with a fixed epoch, fixed mode, fixed host and a pinned deflate level, in sorted order. Same content now means the same sha256, wherever it was staged from. It still means only that. A changed member byte changes the hash, and a changed qualified artifact is refused before an archive exists. The one assumption left is that zlib emits the same stream for the same input at the same level, which is why the level is pinned explicitly rather than left at the default. clean.py's action moves behind __main__ so that reading its list cannot delete anything -- the new test imports it, and during `make check` the working directory is the tree the rest of the suite is using.
GPU_BACKENDS.md grew one section per implementation step, and each step described the state at that step. Read end to end today it says there is no NPU compute path, that the engine does not link the backend, that device discovery, the artifact registry, dispatch and matmul interception are not implemented, that there is deliberately no --xdna flag and no COLI_XDNA variable, that the lane is reachable only from an internal test control, that logical M above 64 declines, and that the artifacts are not shipped because where they should live has not been decided. Every one of those was true when it was written. None of them is true now, and the string "M256" does not appear anywhere. Fixing thirteen sentences would leave a document that still reads as a chronology. Rewrite the section from the code instead: the flag and the environment variable the engine actually reads, their precedence, the two M buckets, the gate order as the engine evaluates it, the failure verdicts, and the optional package. The technical material worth keeping is carried across -- integrity as byte identity, what STATIC_ARTIFACT_QUALIFIED does not mean, the pair-versus-planar layout refusal, the padding argument, the helper build provenance and the /Zc:__cplusplus trap. 492 lines become 291. The CUDA and HIP half of the document is untouched. docs/xdna.md was already accurate but never mentioned COLI_XDNA, so someone driving the engine directly -- which is how a SCORE harness works -- had no documented way in, and "without --xdna nothing changes" was not quite true for anyone who had exported it. Four lines close that. No runtime, build or packaging behaviour changes.
tests/xdna_fake_helper.c exports with __declspec(dllexport), which gcc and clang reject outright. It only ever needed to be a DLL because the two owners that use it -- test_xdna_execution and test_xdna_failure -- bind it through the XDNA loader, and that loader is LoadLibraryExA. Off Windows the loader answers ABSENT by design, so those two have nothing to bind to even if the helper did compile. They were still portable gates, so `make test-c` tried to build the DLL on Linux and macOS and failed the whole C suite before any test ran. Exclude them and add them back under IS_WIN, the same shape test_uring already uses for the opposite platform. The other three XDNA owners -- registry, prepared state and QT state -- touch no loader and stay portable gates, so the registry, integrity and lifetime contracts are still checked on every platform. Windows is unchanged: all five run, and the aggregate is the same 749.
test_xdna_prepared_state has three cases that ask for a buffer the system
cannot satisfy -- 0xFFFFFFFF x 0x80000000, 0xFFFFFFF squared, and a
prepare_from_fmt4 at 0x40000000 x 0x40000000. Each proves the same thing: a
representable-but-unsatisfiable request must be reported as an allocation
failure, not misreported as an arithmetic one. No size ceiling is imposed in
the product, deliberately, so these need a real allocator refusal to test.
AddressSanitizer replaces the allocator with one that treats an oversized
request as fatal rather than refusing it, so the process aborts inside
posix_memalign before prepare_begin can return:
ERROR: AddressSanitizer: requested allocation size 0xffffffff00000000 ...
exceeds maximum supported size of 0x10000000000
JustVugg#1 coli_xdna_prepare_begin backend_xdna.c:709
JustVugg#2 test_size_safety tests/test_xdna_prepared_state.c:98
The cases cannot be expressed under ASan at all, so skip them there and print
that they were skipped rather than passing silently. The arithmetic-overflow
cases never reach the allocator and still run.
Also correct one label while touching the line: 0xFFFFFFFF x 0x80000000 x 2 is
0xFFFFFFFE00000000, which does NOT exceed SIZE_MAX -- it is a second
unsatisfiable-allocation case, not an overflow one, and it only ever passed
because the allocator refused it.
The product is unchanged.
Contributor
Author
|
The remaining red check ( The relevant Qwen sources are unchanged by this PR. What did change is whether that job's sanitizer step rebuilds. So the step named "Same run under ASan + UBSan" has been re-running the Forcing a genuinely fresh sanitized build at 692af87 reproduces the defect, It is intermittent — roughly 3–5 reports per 20 runs, on the base and on this One caveat for whoever picks it up: no source frame is available. The ASan I have not touched |
One conflict, in c/.gitignore, where both sides appended to the end of the file: dev added glm53, qwen36 and the three generated .spv shaders; this branch added the XDNA test scaffolding the *.o and *.dll globs do not cover. They are independent, so both blocks are kept. Everything else merged cleanly. Checked afterwards that the lane's seams survived: the backend_xdna.h include and the two guarded dispatch lines in moe(), the prepared-state release hook, --xdna and its COLI_XDNA line in coli, the XDNA=1 switch and the Windows-only test gating in the Makefile, and XDNA=1 in the Windows release matrix alongside dev's new coli.cmd copy. dev's new glm53 engine needed no change here: clean.py derives the engine list from family_registry, so glm53 is owned by make clean automatically and PACKAGED_BUT_NOT_CLEANED is still 0, now across seven engines. make check: 758 passed, 72 skipped.
Under ASan the prepared-state owner died with a wild-address SEGV inside the memcpy in test_fmt4_immutability -- on some hosts. CI passed it at one head and failed at the next with the file and backend_xdna.c byte-identical, and it reproduces every time on a Zen 5 box at the job's own -O1 -march=native. It is not a defect in the code under test. It disappears at -O0, it disappears with detect_stack_use_after_return=0, and it disappears when the four fixture arrays leave the stack -- backend_xdna.c untouched in every case. That points at a codegen/sanitizer interaction rather than at anything this test asserts, so make the fixtures static and leave the assertions exactly as they were. They are read-only stand-ins for the authoritative tensor, the test is single-threaded and runs once, so .bss costs it nothing and takes 1 KB off an already large inlined main frame. No sanitizer option is disabled and no product code changes. The three portable XDNA owners were rebuilt and run under ASan/UBSan on Linux: registry, prepared state and QT state all clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional, explicitly requested compute lane that runs two GLM
operations — the shared expert's gate and up projections — on an AMD XDNA2
(Ryzen AI) NPU on Windows. Nothing else, on no other platform.
It is off by default, it is never selected by discovering hardware, and the
qualified operations compute in BF16, so output can differ from the normal path.
--xdnaprints that warning before any work starts.What changes
c/backend_xdna.{c,h}— the host side. Links no XRT.c/backend_xdna_helper.cpp— the only file that links XRT, built separately, shipped separatelyc/colibri.c+139/−0: the include, a side pointer onQT, its release hook, and two guarded lines inmoe()c/coli+12/−0: one--xdnaflag, one line settingCOLI_XDNA=1XDNA=1(default0) compiles the lane in; the Windows release job uses itThe two dispatch lines use the same idiom as
vk_matmul_qt— return 0 means"not handled", and the current path runs:
Default behaviour
Unchanged, and measured against this PR's base rather than asserted:
692af87XDNA=1-22.105853-22.105853KERNEL32)[XDNA]diagnosticsThe optional sidecar was installed and valid during that run. The default path
ignores it. An
XDNA=1host has the same import table as a default one andstarts normally on a machine with no NPU.
Turning it on
--xdnasetsCOLI_XDNA=1for the engine, and the engine enables the lane whenthat variable parses as a non-zero integer. Exporting it yourself has the same
effect, which is how a harness driving the engine directly gets in; the flag wins
when both are present. All
[XDNA]output is stderr, never stdout, somachine-readable score output is unaffected.
Qualified scope
Everything else — the down projection, routed experts, attention, every other
model — uses the normal path. The semantic family is passed explicitly by the
call site and never inferred from a shape, so an unrelated 6144×2048 operation
cannot inherit this one's qualification.
Failure and fallback
Every failure is fail-closed to the current path, with a diagnostic that names
the actual problem:
The package integrity gate runs before the helper is loaded, so unknown or
corrupt bytes never reach execution. No partial NPU result is ever exposed.
Restoring the correct bytes restores the lane with no residue.
Packaging and dependencies
The helper and the artifacts need the XRT SDK and the AIE toolchain, which the
release runners do not have, so CI cannot build them. They ship as an optional
sidecar produced on a qualified machine:
The name keeps the core archive's stem, so the release job's existing
sha256sum colibri-*andgh release upload artifacts/*already cover it — noworkflow change is needed to publish it.
--verify-releaserefuses if thearchive disagrees with its manifest, if the manifest names a different version,
or if any artifact digest disagrees with the SHA256 compiled into the engine. The
archive is byte-reproducible: same inputs, same sha256, regardless of staging
path or file timestamps.
XRT and the MSVC redistributable are never bundled. They are the user's to
install, and a test asserts they cannot appear in the archive.
Correctness and quality
Device execution is qualified against a BF16 oracle. That is not a claim of
equivalence with
matmul_qt, and the two legitimately differ: the current pathaccumulates f32 activations against dequantised int4, the lane is BF16
throughout.
What was measured on the real model, both halves:
The first half is not evidence of equivalence and is not offered as such — at
n=24 the confidence interval is roughly 31 percentage points. Both halves are in
docs/xdna.mdso a user can make the call themselves.Performance
No speed claim is made. The lane is not chosen on speed or cost and there is
no automatic selection. For context on one commit title: an earlier large cold
integration penalty was not reproduced after the weight-conversion refinement in
perf(xdna): accelerate prepared weight conversion(84.20 ms → 20.30 ms pertensor, byte-identical output). That is a non-reproduction, not a speedup.
Testing
The +44 are all new and XDNA-owned: 19 packaging/integrity/determinism tests, 18
loader and ABI-rejection tests, 7 clean-ownership tests. No skip change, nothing
disabled.
Five C owners (registry, prepared state, QT state, execution, failure — 248
assertions in the failure owner alone) run in ordinary CI against synthetic
helper DLLs the tests build themselves. No NPU or XRT is needed for any of
this.
On real hardware,
tests/xdna_physical_probedrives the production registry,integrity check, loader, weight preparation, eligibility gates and candidate
function against the device. Both buckets: 6 dispatches, 6 completions, 1 device
open. Above the range: declined, 0 dispatches. Removing the helper, removing an
artifact and flipping one byte in an artifact each fall back to the exact default
result.
Known limitations
M > 256uses the normal path.diverge.
on a qualified machine.
Why this shape
The lane sits behind an explicit flag rather than hardware discovery because the
consequence is reduced precision, and that is a decision a user should make
rather than inherit from owning a laptop. It is one operation family rather than
a backend because that is what has been qualified end to end — the artifacts,
the integrity chain, the fallback and the packaging — and a wider claim would
outrun the evidence.