fix: preserve hidden visibility when merging symbols and localize dynsym exports#1700
Conversation
davidlattimore
left a comment
There was a problem hiding this comment.
Thanks for working on this. I've benchmarked the change and unfortunately it incurs an 8% slowdown on linking librustc-driver and a 4% slowdown linking the Zed editor.
|
|
||
| symbol_db.buckets = buckets; | ||
|
|
||
| for name in &symbol_db.hidden_undefined_names { |
There was a problem hiding this comment.
Why does this need to be done from the main thread? Perhaps it could be done in parallel from each object or from each hash bucket? Better yet, could we avoid storing a collection of hidden undefined names at all and update the flags during the resolution phase? - i.e. when we resolve undefined symbols
There was a problem hiding this comment.
yeah, Didn't think about it, so the hidden visibility merge cab be taken care during alternative resolution phase itself, reducing the main thread iteration
There was a problem hiding this comment.
I think it could probably be done during resolution itself, which is after alternative symbol selection. Resolution is where undefined symbols are looked up. For performance reasons, we want to do just a single hash lookup of each symbol. For defined symbols this is done when building the symbol DB. For undefined symbols, it's during resolution. I suspect you can handle both definitions and undefined symbols in resolution though, since for defined symbols, we know what symbol ID they resolve to without having to do hashmap lookup.
| { | ||
| let info = self.get_symbol_name_and_version(symbol, local_index)?; | ||
| outputs | ||
| .add_hidden_undefined_name(UnversionedSymbolName::prehashed(info.name())); |
There was a problem hiding this comment.
Why are we storing symbol names? Would SymbolIds work?
There was a problem hiding this comment.
we can't store it in symbolid because if the symbol is hidden and undefined then there won't be any entry in symbol_definitions vec and can't find the definition ID so the same behaviour might repeat
|
BTW, to what extent did you use LLMs/AI when writing the PR description? I don't know your usual writing style, but it has an unusual feel to it that reminds me of how LLMs write |
|
@davidlattimore So, I initially I write the PR description myself explaining the issue and the fix, and then use an llm to help polish the wording and formatting |
|
OK, thanks. Going forward, would you be able to just post what you write? Don't worry if it's not polished - if an LLM can understand what you wrote, then I'm sure we can too |
|
@davidlattimore Sure, noted from next time I will take care of it, I just had a doubt while working on this the hidden visibility, handling was being skipped in the test files, so it was clear why wild's behaviour was wrong for the hidden merge. But the protected visibility seems to be skipped both in the tests and in
|
|
It's good to keep PRs small and focused on one issue where possible. So I'd say fixing hidden symbols in this PR and protected in a separate PR would be best. But certainly good to think about how protected visibility might be handled and take that into consideration when deciding how to solve hidden. The exception, where I'd say it might be reasonable to fix both in the same PR would be if that actually made the PR simpler, but otherwise, probably keep them separate. |
a29c11c to
2f0f8e5
Compare
|
@davidlattimore any review on this? |
| if !local_symbol_attributes.default_visibility { | ||
| let visibility = resources.symbol_db.input_symbol_visibility(local_symbol_id); | ||
| if visibility != Visibility::Default { | ||
| resources |
There was a problem hiding this comment.
I think in order to get acceptable performance, we need to not store into any SegQueues or Vecs. Would it work to get access to the AtomicPerSymbolFlags here and directly update it?
There was a problem hiding this comment.
Yeah, we can do it I thought doing it sequentially would affect the efficiency but doing it in parallel can save a lot of time. Thanks for telling me about the atomic symbols, they do exactly what we're trying to do
096aacd to
caa5ae7
Compare
|
#1754 moved all test sources into separate subdirectories. The test files were mostly unchanged, so hopefully rebasing should only require moving your changes to the file in the new location. |
caa5ae7 to
9b77a43
Compare
|
@davidlattimore I rebased the branch, git handled the rebase itself, so everything is in the right place |
9b77a43 to
21b6246
Compare
davidlattimore
left a comment
There was a problem hiding this comment.
Nice! I can confirm that the performance loss is gone, so this is very close to being able to be merged.
| if visibility != Visibility::Default | ||
| && let Some(def_id) = symbol_db.get_unversioned(&pre_hashed) | ||
| { | ||
| symbol_db::apply_visibility_to_definition( |
There was a problem hiding this comment.
If I comment out both calls to apply_visibility_to_definition, all the tests still pass. Do you think you can come up with an integration test that is fixed by this part of the change?
There was a problem hiding this comment.
understood, so you want me to create a integration test, which tests part of the code is working or not, I will do that
There was a problem hiding this comment.
Thanks. It looks like the new test successfully fails if this call is commented out, but commenting out the other call below still results in the tests passing. Do you think the test can be extended to cover that as well?
BTW, the failing test:
/usr/bin/ld.bfd: /__w/wild/wild/wild/tests/build/elf/x86_64/hidden-weak-archive-visibility/default/hidden-weak-archive-visibility.c.o: relocation R_X86_64_32 against hidden symbol `get_value' can not be used when making a shared object
/usr/bin/ld.bfd: failed to set dynamic section sizes: bad value
I think might be fixed by adding CompArgs:-fPIC, although I haven't tried.
There was a problem hiding this comment.
Thank you @davidlattimore , i did try extending the test for the second call but couldn’t make it fail when the second call is commented out i added that call just to be safe thinking the same symbol might show up twice in undefined_symbols but the first call already covers it, i will remove the 2nd call
There was a problem hiding this comment.
What about having two objects with references to an undefined symbol, both weak references but only one hidden, then a configuration where the object containing the hidden symbol is first and a configuration where it's second?
There was a problem hiding this comment.
damn, I didn't think in that way, Now i have created one more ref of get_value which is weak and protected which goes through the 2nd function call
06fc429 to
59a2ac9
Compare
| //#Object:hidden-weak-archive-visibility-2.c | ||
| //#NoDynSym:get_value | ||
| //#ExpectDynSym:public_func | ||
| //#Config:vacant:default |
There was a problem hiding this comment.
Could you put a blank line before each of the configs? That's how all our other tests are formatted since it makes it easier to see where they start
59a2ac9 to
8312c01
Compare
davidlattimore
left a comment
There was a problem hiding this comment.
Thanks for doing this :)

When a symbol has mixed visibility definitions across object files, the most restrictive visibility should win (
hidden > protected > default). Wild was computing the merged visibility correctly but silently discarding it, so symbols that should be local were still leaking intodynsym.Fixes #1693
Root cause
Two issues in
handle_non_default_visibility:NON_INTERPOSABLEwas set, neverDOWNGRADE_TO_LOCAL— the flag that actually prevents export into dynsymA third issue:
data5is declared hidden in one object but defined (with default visibility) in another. Since one is undefined and the other defined, they never appear in the same alternative-definition group, so the merged visibility path never ran.This is fixed by tracking hidden undefined references during symbol loading and downgrading their definitions after resolution.