Skip to content

Add wasm32-unknown-wasip1-threads variant to swift.wasm_sdk - #1860

Open
AttilaTheFun wants to merge 3 commits into
bazelbuild:mainfrom
AttilaTheFun:lshire-wasm-threads-sdk
Open

Add wasm32-unknown-wasip1-threads variant to swift.wasm_sdk#1860
AttilaTheFun wants to merge 3 commits into
bazelbuild:mainfrom
AttilaTheFun:lshire-wasm-threads-sdk

Conversation

@AttilaTheFun

@AttilaTheFun AttilaTheFun commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

What

Adds a threads option to the swift.wasm_sdk module extension so a consumer
can target wasm32-unknown-wasip1-threads (WebAssembly with shared memory,
atomics, and wasi-threads) in addition to the single-threaded
wasm32-unknown-wasip1. This unblocks building threaded Swift-wasm reactors
that run with real multicore parallelism (e.g. under WAMR with wasi-threads) via
bazel build.

To support the second bundle flavor without a second set of hardcoded flags and
paths, the wasm repository rule now derives the target triple, SDK paths, and
compiler/linker flags from the metadata the bundle itself ships
(info.json / swift-sdk.json / toolset.json) — details below.

swift.wasm_sdk(
    toolchain_name = "swift",
    threads = True,
    url = "https://github.com/swiftwasm/swift/releases/download/swift-wasm-6.3-RELEASE/swift-wasm-6.3-RELEASE-wasm32-unknown-wasip1-threads.artifactbundle.zip",
    sha256 = "…",
)

Why an explicit url + sha256

swift.org does not publish a threads bundle — the computed
https://download.swift.org/.../swift-<v>-RELEASE_wasm.artifactbundle.tar.gz
URL 404s for the threads variant. The threads SDK comes from
swiftwasm instead, e.g.
swift-wasm-6.3-RELEASE-wasm32-unknown-wasip1-threads.artifactbundle.zip. So the
extension gains url and sha256 attrs that override the computed swift.org URL
(and are required when threads = True).

The url/sha256 override is also usable on the non-threads path if you want to
pin a mirror.

The repository rule is driven by the bundle's own metadata

Rather than hardcoding per-variant flags and directory layouts, the repository
rule now parses the metadata every Swift SDK artifact bundle ships (both the
swift.org and swiftwasm bundles follow the same scheme):

  1. info.json names the bundle's artifacts; the default (non-embedded)
    artifact's swift-sdk.json path replaces the previous
    bundle-layout computation/scan. (Embedded Swift variants —
    embedded-swift-sdk.json — are ignored.)
  2. swift-sdk.json provides the target triple (validated against the
    threads attribute, so pointing threads = True at a single-threaded
    bundle or vice versa fails with a clear message), plus sdkRootPath and
    swiftStaticResourcesPath, which replace the previously hardcoded
    WASI.sdk / swift.xctoolchain/usr/lib/swift_static paths.
  3. toolset.json (via toolsetPaths, merged in order) provides the
    per-tool extraCLIOptions:
    • cCompiler/cxxCompiler options → the generated cc_args (compile);
      for the threads bundle this is -matomics -mbulk-memory -mthread-model posix -pthread -ftls-model=local-exec.
    • swiftCompiler options → swift_toolchain copts (for the threads bundle:
      -static-stdlib plus the clang flags via -Xcc; for the swift.org
      bundle: -static-stdlib).
    • linker options are raw wasm-ld flags (SwiftPM invokes the linker
      directly), so they are wrapped in -Wl, for the clang driver the
      generated toolchains link through (threads bundle: --import-memory --export-memory --shared-memory --max-memory=1073741824).
    • rootPath and per-tool executable overrides are intentionally ignored:
      the generated toolchains always drive the paired standalone toolchain's
      own swiftc/clang.

The parsing helpers (_relative_metadata_path, _swift_sdk_json_path,
_swift_sdk_target_settings, merged_toolset_options,
linker_options_to_clang_args) are SDK-kind-agnostic, so the Android
repository rule and the Static Linux SDK proposed in #1813 can share them —
_relative_metadata_path is taken verbatim from #1813 so whichever lands
second rebases cleanly onto the other's copy.

Backwards compatibility

The single-threaded path now reads the same metadata. For the swift.org
wasm32-unknown-wasip1 bundle the resolved paths are identical and the only
flag delta is that the bundle toolset's -static-stdlib is now passed to
swiftc — inert for compile actions: rebuilding the wasm example produced
bit-identical objects (all downstream actions cache-hit) and
//examples/cross_compilation/wasm:reactor_test passes.

Verification

  • Single-threaded (swift.org bundle through the metadata path):
    //examples/cross_compilation/wasm:reactor_test passes; the recompiled
    objects are bit-identical to before, so everything downstream cache-hits.
  • Threads, end-to-end on macOS/arm64: a withTaskGroup program compiles
    against the wasm32-unknown-wasip1-threads swiftmodules, links a
    shared-memory module, and runs under
    wasmtime -W threads=y,shared-memory=y -S threads=y. A peak-concurrency
    probe confirms tasks actually run in parallel (8 workers, peak 4 concurrent
    tasks). After the metadata rework the generated toolchains are byte-identical
    to the previously hand-mirrored flags. Full transcript in the PR comments.
  • Unit tests for merged_toolset_options (accumulation across multiple
    toolsets, both bundle shapes) and linker_options_to_clang_args in
    test/utils_tests.bzl.

Files changed

  • swift/extensions.bzlthreads (bool), url (string) attrs on the
    wasm_sdk tag class; _setup_wasm_sdk validation + URL/sha256 selection;
    threading threads/url through to the repository rule.
  • swift/internal/extensions/swift_sdks.bzl — metadata parsing
    (info.json/swift-sdk.json/toolset.json) shared helpers; triple
    validation against threads; toolset-driven template substitutions.
  • swift/internal/extensions/wasmsdk.BUILD — resource-dir/sysroot and
    compile/link flag placeholders filled from the SDK metadata.
  • test/utils_tests.bzl — unit tests for the toolset helpers.

@AttilaTheFun

Copy link
Copy Markdown
Contributor Author

End-to-end verification ✅

Built and ran a Swift Concurrency program through the threads = True SDK produced by this PR (with the 206a15e1 triple fix), on macOS/arm64:

# MODULE.bazel
swift.toolchain(name = "swift_toolchain", swift_version = "6.3")   # match the SDK's swiftmodule format
swift.wasm_sdk(
    toolchain_name = "swift_toolchain",
    threads = True,
    url = "…/swift-wasm-6.3-RELEASE-…-wasip1-threads.artifactbundle.tar.gz",
    sha256 = "…",
)
// Hello.swift
@main struct Hello {
  static func main() async {
    await withTaskGroup(of: Int.self) { group in
      for i in 0..<4 { group.addTask { i * i } }
      var total = 0
      for await v in group { total += v }
      print("sum of squares:", total)
    }
  }
}
$ bazel build //:hello --platforms=//platforms:wasm32
Target //:hello up-to-date: bazel-bin/hello.wasm      # links against …/_Concurrency.swiftmodule/wasm32-unknown-wasip1-threads.swiftmodule

$ wasmtime run -W threads=y,shared-memory=y -S threads=y bazel-bin/hello.wasm
sum of squares: 14

The compile resolves the wasm32-unknown-wasip1-threads swiftmodules and the linked module is the threads variant (shared env::memory), running under wasmtime's threads proposal.

Note on the triple fix (206a15e1): before it, the generated toolchain compiled with --target=wasm32-unknown-wasip1 while the SDK only ships …-threads modules, so swiftc failed module resolution (found: wasm32-unknown-wasip1-threads). Substituting the actual {target_triple} into both the clang --target and CC_TARGET_TRIPLE make-variable fixes it.

One gotcha worth flagging for users: the host swift_version must share a swiftmodule format with the SDK — pairing a 6.3 SDK with a 6.3.2 host fails with module compiled with Swift 6.3 cannot be imported by Swift 6.3.2 compiler. Matching them (6.3 ↔ 6.3-RELEASE SDK) resolves it. Not a defect in this PR, but easy to trip over.

@AttilaTheFun
AttilaTheFun marked this pull request as ready for review July 16, 2026 04:35
@AttilaTheFun

Copy link
Copy Markdown
Contributor Author

Pushed 03a6f45, which replaces the hand-mirrored flag lists and the bundle-layout scan with parsing of the bundle's own metadata (info.jsonswift-sdk.jsontoolset.json) — the PR description is updated to match. Both bundles ship the same metadata scheme, so the single-threaded path goes through it too; the only behavior delta there is the swift.org toolset's -static-stdlib now reaching swiftc (inert for compiles — objects came out bit-identical, and reactor_test passes). Re-ran the threads end-to-end after the rework: generated toolchains byte-identical to the previous hand-mirrored flags, and the wasmtime peak-concurrency probe still parallelizes.

The parsing helpers are deliberately SDK-kind-agnostic (and _relative_metadata_path is lifted verbatim from #1813), so whichever of #1813 / this lands first, the other can rebase onto shared parsing.

@keith keith added this to the 4.0.0 milestone Jul 24, 2026
Add a `threads` option to the `swift.wasm_sdk` module extension so a
consumer can target `wasm32-unknown-wasip1-threads` (WebAssembly with
shared memory, atomics, and wasi-threads) in addition to the
single-threaded `wasm32-unknown-wasip1`.

swift.org does not publish a threads SDK bundle, so the extension also
gains `url` + `sha256` attrs that override the computed swift.org URL;
these are required for the threads variant (point it at a swiftwasm
release). The swiftwasm bundle nests its target directory under a
differently-named inner directory than the swift.org bundle, so the
threads path discovers it by scanning for the target triple.

When threads is enabled, the generated Swift and cc toolchains gain the
atomics/bulk-memory/pthread compile flags and the shared-memory linker
flags. The single-threaded path is unchanged (default threads=False
renders byte-for-byte identical toolchains).
The threads variant injected the atomics/shared-memory build flags but the
generated cc + Swift toolchains still compiled with
`--target=wasm32-unknown-wasip1` (the single-threaded triple). The swiftwasm
threads SDK ships its swiftmodules under `wasm32-unknown-wasip1-threads`, so
swiftc could not resolve `Swift`/`_Concurrency` for the single-threaded triple
("could not find module Swift for target wasm32-unknown-wasip1; found:
wasm32-unknown-wasip1-threads").

Thread the already-computed `triple` through a `{target_triple}` substitution so
both the clang `--target` and the `CC_TARGET_TRIPLE` make variable (which the
Swift toolchain parses into swiftc `-target`) use the correct triple. Byte-for-
byte identical for the default single-threaded path.

Caught by locally building a threaded wasm binary against a swiftwasm threads
SDK bundle; the existing CI matrix does not exercise the threads path.
Instead of hardcoding the threads variant's compiler/linker flags and
scanning the archive for the target-triple directory, parse the metadata
that every Swift SDK artifact bundle ships:

* `info.json` names the default (non-embedded) artifact's
  `swift-sdk.json`, which replaces the layout scan.
* `swift-sdk.json` provides the target triple (validated against the
  `threads` attribute), `sdkRootPath`, and `swiftStaticResourcesPath`,
  which replace the hardcoded `WASI.sdk`/`swift_static` paths.
* `toolset.json` provides the per-tool `extraCLIOptions`, which replace
  the hardcoded threads flag lists. Compiler options are forwarded
  verbatim; linker options are raw wasm-ld flags (SwiftPM invokes the
  linker directly), so they are wrapped in `-Wl,` for the clang driver
  the generated toolchains link through.

For the single-threaded swift.org bundle the only behavior change is
that the toolset's `-static-stdlib` is now passed to swiftc (inert for
compile actions); for the threads bundle the generated toolchains are
identical to before, now sourced from the bundle instead of mirrored
constants.

The parsing helpers (`_relative_metadata_path`, `merged_toolset_options`,
`linker_options_to_clang_args`, `_swift_sdk_json_path`,
`_swift_sdk_target_settings`) are SDK-kind-agnostic so the Android and
Static Linux (bazelbuild#1813) repository rules can share them.
@AttilaTheFun
AttilaTheFun force-pushed the lshire-wasm-threads-sdk branch from 03a6f45 to 308114d Compare July 30, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants