Add swift.use_swiftinterface_for_caching feature - #1880
Conversation
a1f8937 to
a86f57a
Compare
|
Ready for review. External contributor, so the Test workflow appears to be gated on maintainer approval — could someone kick it off when you get a chance? Rationale and design context in #1881. |
Introduces an opt-in feature `swift.use_swiftinterface_for_caching` that makes
Swift dependency `.swiftinterface` files (rather than `.swiftmodule` files)
participate in the Swift compile action's cache key when
`library_evolution` is enabled for those dependencies. Swiftmodule files are
still provided to the compiler via Bazel's `unused_inputs_list` mechanism —
they remain in the sandbox but do not contribute to the action key.
Motivation
----------
With `library_evolution`, a Swift module's public ABI is captured in its
`.swiftinterface`. Internal implementation changes (private functions, changes
to non-`public` symbols) only rewrite the `.swiftmodule`, not the
`.swiftinterface`. Today the swiftmodule of every transitive dependency is
part of every downstream compile action's key, so any internal change in an
upstream library invalidates the cache for its entire reverse dependency
closure — even though nothing observable to those downstream compilations
actually changed.
This is especially painful for CI/remote-cache workflows in large mixed
Swift/Objective-C codebases: an upstream refactor forces recompilation of
thousands of downstream Swift modules that would have produced bit-identical
output.
How it works
------------
- Opt in via `--features=swift.use_swiftinterface_for_caching`. Requires
`--features=swift.enable_library_evolution` and
`--features=swift.emit_swiftinterface` on the dependencies (this repo's
existing features).
- Only affects the compile action cache key. The compiler still receives the
same set of files as before (swiftmodule + swiftinterface); we just tell
Bazel via `unused_inputs_list` that the swiftmodule file's *content* does
not need to trip a cache miss.
- If any transitive dependency lacks a `.swiftinterface` (for example, it is
not built with `library_evolution`), that dependency's swiftmodule falls
back to being a normal cache-key input. Correctness is preserved: mixing
library-evolution and non-library-evolution dependencies is fine, only the
library-evolution ones benefit.
- `private_swiftinterface` is preferred over `swiftinterface` where present,
matching `transitive_swift_dependency_inputs`.
Implementation
--------------
1. `ConfigResultInfo` gains an `unused_inputs: List[File]` field.
`_apply_action_configs` accumulates it across configurators, and
`run_toolchain_action` wires it into `actions.run(unused_inputs_list=...)`
by writing a per-action listing file (named after
`<module_name>_<action>_unused_inputs.txt` for uniqueness, with fallback to
the output path).
2. `compile()` and `compile_module_interface()` collect two parallel lists
from `transitive_modules` — `transitive_swiftinterfaces` and
`transitive_swiftmodules_only` — using the same
`private_swiftinterface`-preferred rule as `transitive_swift_dependency_inputs`.
Both are placed on `prerequisites` alongside the existing
`transitive_swift_dependency_inputs`, together with a
`use_swiftinterface_for_caching` boolean.
3. `_explicit_swift_module_map_info` also reads the feature and returns
`struct(file, inputs, unused_inputs)`. Both call sites propagate
`explicit_swift_module_map_unused_inputs` into `prerequisites`.
4. Three configurators consume the split when the feature is enabled and
swiftinterfaces are present:
- `_dependencies_swiftmodules_and_swiftdocs_configurator`
- `_dependencies_swiftmodules_configurator`
- `_explicit_swift_module_map_configurator`
Each falls back to the original behavior (`transitive_swift_dependency_inputs`
going into `inputs`) when the feature is off or there are no
swiftinterfaces to key off of.
Verification
------------
Manually verified against a real large iOS codebase:
- With the feature enabled, a compile action produces
`<target>_SwiftCompile_unused_inputs.txt` listing its transitive
dependencies' swiftmodule files. Their swiftinterface files remain in the
action inputs (cache key), while the swiftmodules are marked as unused.
- Modifying a `private` function in an upstream dependency (which only changes
the swiftmodule, not the swiftinterface) triggers only that upstream module
to recompile; downstream Swift compile actions hit the action cache.
- Disabling the feature with `--features=-swift.use_swiftinterface_for_caching`
under the same change causes downstream compile actions to re-run, confirming
the feature is what gates the cache reuse.
- Mixed dependency graphs (some deps without library_evolution) build
correctly; those deps' swiftmodules stay in the cache-key inputs.
Tests
-----
- `//test:swiftinterface_for_caching` — a new analysis-test suite covering
the feature. Fixture in `test/fixtures/swiftinterface_for_caching/` sets up
a `library_evolution` upstream, a non-`library_evolution` upstream, and two
downstream clients (one purely library-evolution, one mixed).
- Two `action_inputs_test`s (feature on) assert that the downstream compile
action still declares upstream `.swiftinterface` and `.swiftmodule` files
in its `action.inputs`. `unused_inputs_list` files remain in the sandbox
as regular inputs; Bazel just excludes them from the action cache key.
Since Starlark's action introspection does not expose the
`unused_inputs_list` list itself, these assertions cover "we didn't drop
anything the compiler still needs".
- A mixed-graph test confirms the fallback: `UpstreamLibNoEvolution` has no
`.swiftinterface`, and its `.swiftmodule` continues to appear in inputs so
correctness is preserved.
- A baseline test with `-swift.use_swiftinterface_for_caching` documents
that the input file set is identical when the feature is disabled — the
difference is only in cache-key membership, which we cannot assert on
directly.
- A `build_test` sanity check builds the full fixture end-to-end with the
feature enabled.
a86f57a to
7c9bf4a
Compare
|
I haven't familiarized myself with this change yet. However, I am wondering why wouldn't we have this by default without feature flag? Any downsides? @xiemotongye |
|
Great question — I actually thought about this quite a bit before settling on opt-in. Short answer: I think making it the default is the right long-term direction, but there are a few reasons to gate it behind a flag first. No correctness regression for the common case. The feature only alters the compile action's cache key; the compiler receives exactly the same file set (swiftmodule + swiftinterface), same command line, same output. And when a transitive dep lacks a The reasons I still went with opt-in:
Concretely, my proposal:
Happy to write up the follow-up "flip default" change once this lands, if that's a path you'd support. |
Thank you for well written response. Your proposal makes sense to me. I am pinging @keith to give his oppinion too. |
Introduces an opt-in feature
swift.use_swiftinterface_for_cachingthat makes Swift dependency.swiftinterfacefiles (rather than.swiftmodulefiles) participate in the Swift compile action's cache key whenlibrary_evolutionis enabled for those dependencies. Swiftmodule files are still provided to the compiler via Bazel'sunused_inputs_listmechanism — they remain in the sandbox but do not contribute to the action key.Motivation
With
library_evolution, a Swift module's public ABI is captured in its.swiftinterface. Internal implementation changes (private functions, changes to non-publicsymbols) only rewrite the.swiftmodule, not the.swiftinterface. Today the swiftmodule of every transitive dependency is part of every downstream compile action's key, so any internal change in an upstream library invalidates the cache for its entire reverse dependency closure — even though nothing observable to those downstream compilations actually changed.This is especially painful for CI/remote-cache workflows in large mixed Swift/Objective-C codebases: an upstream refactor forces recompilation of thousands of downstream Swift modules that would have produced bit-identical output.
How it works
--features=swift.use_swiftinterface_for_caching. Requires--features=swift.enable_library_evolutionand--features=swift.emit_swiftinterfaceon the dependencies (this repo's existing features).unused_inputs_listthat the swiftmodule file's content does not need to trip a cache miss..swiftinterface(for example, it is not built withlibrary_evolution), that dependency's swiftmodule falls back to being a normal cache-key input. Correctness is preserved: mixing library-evolution and non-library-evolution dependencies is fine, only the library-evolution ones benefit.private_swiftinterfaceis preferred overswiftinterfacewhere present, matchingtransitive_swift_dependency_inputs.Implementation
ConfigResultInfogains anunused_inputs: List[File]field._apply_action_configsaccumulates it across configurators, andrun_toolchain_actionwires it intoactions.run(unused_inputs_list=...)by writing a per-action listing file (named after<module_name>_<action>_unused_inputs.txtfor uniqueness, with fallback to the output path).compile()andcompile_module_interface()collect two parallel lists fromtransitive_modules—transitive_swiftinterfacesandtransitive_swiftmodules_only— using the sameprivate_swiftinterface-preferred rule astransitive_swift_dependency_inputs. Both are placed onprerequisitesalongside the existingtransitive_swift_dependency_inputs, together with ause_swiftinterface_for_cachingboolean._explicit_swift_module_map_infoalso reads the feature and returnsstruct(file, inputs, unused_inputs). Both call sites propagateexplicit_swift_module_map_unused_inputsintoprerequisites.Three configurators consume the split when the feature is enabled and swiftinterfaces are present:
_dependencies_swiftmodules_and_swiftdocs_configurator_dependencies_swiftmodules_configurator_explicit_swift_module_map_configuratorEach falls back to the original behavior (
transitive_swift_dependency_inputsgoing intoinputs) when the feature is off or there are no swiftinterfaces to key off of.Verification
Manually verified against a real large iOS codebase:
With the feature enabled, a compile action produces
<target>_SwiftCompile_unused_inputs.txtlisting its transitive dependencies' swiftmodule files. Their swiftinterface files remain in the action inputs (cache key), while the swiftmodules are marked as unused.Modifying a
privatefunction in an upstream dependency (which only changes the swiftmodule, not the swiftinterface) triggers only that upstream module to recompile; downstream Swift compile actions hit the action cache.Disabling the feature with
--features=-swift.use_swiftinterface_for_cachingunder the same change causes downstream compile actions to re-run, confirming the feature is what gates the cache reuse.Mixed dependency graphs (some deps without library_evolution) build correctly; those deps' swiftmodules stay in the cache-key inputs.