From 6250bbe30a0bfb4eff356514f70d12dc4a2f5089 Mon Sep 17 00:00:00 2001 From: rofl0r Date: Thu, 7 Oct 2021 14:45:25 +0000 Subject: [PATCH 01/12] FAQ: add dlclose() essay from IRC --- faq.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/faq.md b/faq.md index 2eb3853..f53f80c 100644 --- a/faq.md +++ b/faq.md @@ -65,6 +65,26 @@ note that if you're building a whole system with musl you never need any of them. you don't need them for building musl, and you don't need them for building apps against musl (but they won't hurt, generally, either). +# Q: why is dlcose() a NO-OP ? + +C does not have any model for functions or pseudo-static data whose lifetimes are not the lifetime of the whole process. +Any attempt to add them is pretty much ad hoc and underspecified because doing it right would require a detailed model compatible with the rest of the language. +It's possible to write individual libraries that are meant to be used with such a model, but it requires explicit care to do it right: +Documenting that you can't keep pointers to their functions or data after unload, etc. +But the problem is that dlopen is recursive, and loading such a module usually involves loading of libraries it depends on. +Libraries which were written to standard C, not this underspecified "C with unloading modules". +They might have registered atexit handlers or otherwise stored pointers to their code or data in places it persists beyond the caller's knowledge of it. +Glibc 'takes care of' the atexit (and dtor) case by doing something wild: executing global ctors at a time other than process termination: at unload time. +This has all sorts of weirdness and violates principle that all dtors are executed in reverse order of ctors and if the library was written assuming dtors run at process exit time, it may be doing something that's not appropriate to happen while the process is continuing to run. +Now, you could try to say "ok let's just unload modules meant to be loaded as modules, but refuse to unload recursive deps and anything with dtors" +Well, bad news: gcc/binutils make *all libraries* appear to have dtors as far as the dynamic linker can tell; the dtors might just be a no-op - +and to know that you'd have to interpret code. +The ELF headers actually have a way to signal "never unload this library". +The problem is that the default is backwards; only libraries explicitly created with that flag set have it, whereas, semantically, the default is "unsafe to unload" +Further, managing thread-local-storage lifetime is a lot more problematic when slots can be freed and reused. (actually a lot of ppl think this is why musl doesn't unload libraries, which isn't the case. it's possible to do right, just more work). +On glibc this actually makes for some weird race conditions where a library that 'could be' unloaded doesn't get unloaded, and unloading it either never happens or gets deferred to the next dlclose operation - which breaks anything assuming that dlclose will actually unload. +So programs making this assumption are already broken on glibc too, just with random rare failures rather than failure every time. + # Q: Why am I getting "error: redefinition of struct ethhdr/tcphdr/etc"? The application you're trying to compile mixes userspace and kernel headers From 25cb95041f65d49246acc41cb6622b56ccab360d Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:00:24 +0000 Subject: [PATCH 02/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/faq.md b/faq.md index f53f80c..07c1d0b 100644 --- a/faq.md +++ b/faq.md @@ -65,7 +65,7 @@ note that if you're building a whole system with musl you never need any of them. you don't need them for building musl, and you don't need them for building apps against musl (but they won't hurt, generally, either). -# Q: why is dlcose() a NO-OP ? +# Q: Why is dlclose() a no-op? C does not have any model for functions or pseudo-static data whose lifetimes are not the lifetime of the whole process. Any attempt to add them is pretty much ad hoc and underspecified because doing it right would require a detailed model compatible with the rest of the language. From 59dfdcec19f3a0c4b7a1439489991737bd60c4ce Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:01:05 +0000 Subject: [PATCH 03/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/faq.md b/faq.md index 07c1d0b..00894eb 100644 --- a/faq.md +++ b/faq.md @@ -68,7 +68,7 @@ building apps against musl (but they won't hurt, generally, either). # Q: Why is dlclose() a no-op? C does not have any model for functions or pseudo-static data whose lifetimes are not the lifetime of the whole process. -Any attempt to add them is pretty much ad hoc and underspecified because doing it right would require a detailed model compatible with the rest of the language. +, and any attempt to add them is ad hoc and underspecified because doing it right would require a detailed model compatible with the rest of the language. It's possible to write individual libraries that are meant to be used with such a model, but it requires explicit care to do it right: Documenting that you can't keep pointers to their functions or data after unload, etc. But the problem is that dlopen is recursive, and loading such a module usually involves loading of libraries it depends on. From 186b60edf8af6084524d2262905860fcda3e633f Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:01:44 +0000 Subject: [PATCH 04/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/faq.md b/faq.md index 00894eb..1d8c452 100644 --- a/faq.md +++ b/faq.md @@ -69,8 +69,7 @@ building apps against musl (but they won't hurt, generally, either). C does not have any model for functions or pseudo-static data whose lifetimes are not the lifetime of the whole process. , and any attempt to add them is ad hoc and underspecified because doing it right would require a detailed model compatible with the rest of the language. -It's possible to write individual libraries that are meant to be used with such a model, but it requires explicit care to do it right: -Documenting that you can't keep pointers to their functions or data after unload, etc. +It's possible to write individual libraries that are meant to be used with such a model, but it requires explicit care to do it right, including documenting that you can't keep pointers to their functions or data after unload, among others. But the problem is that dlopen is recursive, and loading such a module usually involves loading of libraries it depends on. Libraries which were written to standard C, not this underspecified "C with unloading modules". They might have registered atexit handlers or otherwise stored pointers to their code or data in places it persists beyond the caller's knowledge of it. From 72521f314a04c81b588acfb51d0bea28e087804e Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:02:26 +0000 Subject: [PATCH 05/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/faq.md b/faq.md index 1d8c452..9e3a18d 100644 --- a/faq.md +++ b/faq.md @@ -70,8 +70,7 @@ building apps against musl (but they won't hurt, generally, either). C does not have any model for functions or pseudo-static data whose lifetimes are not the lifetime of the whole process. , and any attempt to add them is ad hoc and underspecified because doing it right would require a detailed model compatible with the rest of the language. It's possible to write individual libraries that are meant to be used with such a model, but it requires explicit care to do it right, including documenting that you can't keep pointers to their functions or data after unload, among others. -But the problem is that dlopen is recursive, and loading such a module usually involves loading of libraries it depends on. -Libraries which were written to standard C, not this underspecified "C with unloading modules". +But the problem is that dlopen is recursive, and loading such a module usually involves loading the libraries it depends on, which were written to standard C, not this underspecified "C with unloading modules". They might have registered atexit handlers or otherwise stored pointers to their code or data in places it persists beyond the caller's knowledge of it. Glibc 'takes care of' the atexit (and dtor) case by doing something wild: executing global ctors at a time other than process termination: at unload time. This has all sorts of weirdness and violates principle that all dtors are executed in reverse order of ctors and if the library was written assuming dtors run at process exit time, it may be doing something that's not appropriate to happen while the process is continuing to run. From e87060c52dbe1538abb3ebdc9a98d9e7df0c8f82 Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:03:05 +0000 Subject: [PATCH 06/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/faq.md b/faq.md index 9e3a18d..34db983 100644 --- a/faq.md +++ b/faq.md @@ -73,7 +73,7 @@ It's possible to write individual libraries that are meant to be used with such But the problem is that dlopen is recursive, and loading such a module usually involves loading the libraries it depends on, which were written to standard C, not this underspecified "C with unloading modules". They might have registered atexit handlers or otherwise stored pointers to their code or data in places it persists beyond the caller's knowledge of it. Glibc 'takes care of' the atexit (and dtor) case by doing something wild: executing global ctors at a time other than process termination: at unload time. -This has all sorts of weirdness and violates principle that all dtors are executed in reverse order of ctors and if the library was written assuming dtors run at process exit time, it may be doing something that's not appropriate to happen while the process is continuing to run. +This has all sorts of weirdness and violates the principle that all dtors (destructors) are executed in reverse order of ctors (constructors). Furthermore, if the library was written assuming dtors run at process exit time, it may be doing something that's not appropriate to happen while the process is continuing to run. Now, you could try to say "ok let's just unload modules meant to be loaded as modules, but refuse to unload recursive deps and anything with dtors" Well, bad news: gcc/binutils make *all libraries* appear to have dtors as far as the dynamic linker can tell; the dtors might just be a no-op - and to know that you'd have to interpret code. From d2a6704ac0106fd8fda707d5fd051a7ca2f950d1 Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:03:53 +0000 Subject: [PATCH 07/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/faq.md b/faq.md index 34db983..b459fc3 100644 --- a/faq.md +++ b/faq.md @@ -74,9 +74,8 @@ But the problem is that dlopen is recursive, and loading such a module usually i They might have registered atexit handlers or otherwise stored pointers to their code or data in places it persists beyond the caller's knowledge of it. Glibc 'takes care of' the atexit (and dtor) case by doing something wild: executing global ctors at a time other than process termination: at unload time. This has all sorts of weirdness and violates the principle that all dtors (destructors) are executed in reverse order of ctors (constructors). Furthermore, if the library was written assuming dtors run at process exit time, it may be doing something that's not appropriate to happen while the process is continuing to run. -Now, you could try to say "ok let's just unload modules meant to be loaded as modules, but refuse to unload recursive deps and anything with dtors" -Well, bad news: gcc/binutils make *all libraries* appear to have dtors as far as the dynamic linker can tell; the dtors might just be a no-op - -and to know that you'd have to interpret code. +One possible implementation could be 'unload modules meant to be loaded as modules, but refuse to unload recursive dependencies and anything with dtors'. +Unfortunately, GCC and GNU Binutils make all libraries appear to have dtors as far as the dynamic linker can tell; the dtors might just be a no-op, and to know that you'd have to interpret code. The ELF headers actually have a way to signal "never unload this library". The problem is that the default is backwards; only libraries explicitly created with that flag set have it, whereas, semantically, the default is "unsafe to unload" Further, managing thread-local-storage lifetime is a lot more problematic when slots can be freed and reused. (actually a lot of ppl think this is why musl doesn't unload libraries, which isn't the case. it's possible to do right, just more work). From cad7079572e0c0969072a7f06ff266d51ada6c1d Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:04:12 +0000 Subject: [PATCH 08/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/faq.md b/faq.md index b459fc3..d23d574 100644 --- a/faq.md +++ b/faq.md @@ -76,8 +76,8 @@ Glibc 'takes care of' the atexit (and dtor) case by doing something wild: execut This has all sorts of weirdness and violates the principle that all dtors (destructors) are executed in reverse order of ctors (constructors). Furthermore, if the library was written assuming dtors run at process exit time, it may be doing something that's not appropriate to happen while the process is continuing to run. One possible implementation could be 'unload modules meant to be loaded as modules, but refuse to unload recursive dependencies and anything with dtors'. Unfortunately, GCC and GNU Binutils make all libraries appear to have dtors as far as the dynamic linker can tell; the dtors might just be a no-op, and to know that you'd have to interpret code. -The ELF headers actually have a way to signal "never unload this library". -The problem is that the default is backwards; only libraries explicitly created with that flag set have it, whereas, semantically, the default is "unsafe to unload" +ELF headers actually have a way to signal "never unload this library". +The problem is that the default is backwards: only libraries explicitly created with that flag set have it, whereas, semantically, the default is "unsafe to unload". Further, managing thread-local-storage lifetime is a lot more problematic when slots can be freed and reused. (actually a lot of ppl think this is why musl doesn't unload libraries, which isn't the case. it's possible to do right, just more work). On glibc this actually makes for some weird race conditions where a library that 'could be' unloaded doesn't get unloaded, and unloading it either never happens or gets deferred to the next dlclose operation - which breaks anything assuming that dlclose will actually unload. So programs making this assumption are already broken on glibc too, just with random rare failures rather than failure every time. From 346f19a4e0e127088dab2183fc21d2360703c934 Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:04:29 +0000 Subject: [PATCH 09/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/faq.md b/faq.md index d23d574..92ed8ed 100644 --- a/faq.md +++ b/faq.md @@ -78,7 +78,8 @@ One possible implementation could be 'unload modules meant to be loaded as modul Unfortunately, GCC and GNU Binutils make all libraries appear to have dtors as far as the dynamic linker can tell; the dtors might just be a no-op, and to know that you'd have to interpret code. ELF headers actually have a way to signal "never unload this library". The problem is that the default is backwards: only libraries explicitly created with that flag set have it, whereas, semantically, the default is "unsafe to unload". -Further, managing thread-local-storage lifetime is a lot more problematic when slots can be freed and reused. (actually a lot of ppl think this is why musl doesn't unload libraries, which isn't the case. it's possible to do right, just more work). +Furthermore, managing thread-local-storage lifetime is a lot more problematic when slots can be freed and reused. +It seems to be a common misconception that this is why musl doesn't unload libraries, which isn't the case. It is possible to do right, just more work. On glibc this actually makes for some weird race conditions where a library that 'could be' unloaded doesn't get unloaded, and unloading it either never happens or gets deferred to the next dlclose operation - which breaks anything assuming that dlclose will actually unload. So programs making this assumption are already broken on glibc too, just with random rare failures rather than failure every time. From 499e4fed13b9a03891cdda692e23185742dec9ab Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:04:51 +0000 Subject: [PATCH 10/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/faq.md b/faq.md index 92ed8ed..729efe5 100644 --- a/faq.md +++ b/faq.md @@ -80,8 +80,8 @@ ELF headers actually have a way to signal "never unload this library". The problem is that the default is backwards: only libraries explicitly created with that flag set have it, whereas, semantically, the default is "unsafe to unload". Furthermore, managing thread-local-storage lifetime is a lot more problematic when slots can be freed and reused. It seems to be a common misconception that this is why musl doesn't unload libraries, which isn't the case. It is possible to do right, just more work. -On glibc this actually makes for some weird race conditions where a library that 'could be' unloaded doesn't get unloaded, and unloading it either never happens or gets deferred to the next dlclose operation - which breaks anything assuming that dlclose will actually unload. -So programs making this assumption are already broken on glibc too, just with random rare failures rather than failure every time. +On glibc this actually makes for some weird race conditions where a library that 'could be' unloaded doesn't get unloaded, and unloading it either never happens or gets deferred to the next `dlclose` operation, which breaks anything assuming that `dlclose` will actually unload. +Therefore, programs making this assumption are already broken on glibc, just with random rare failures rather than failure every time. # Q: Why am I getting "error: redefinition of struct ethhdr/tcphdr/etc"? From 7d7fffded36a0c6a1786d74c7c667c18594d9fd3 Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:05:10 +0000 Subject: [PATCH 11/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/faq.md b/faq.md index 729efe5..e7f7d86 100644 --- a/faq.md +++ b/faq.md @@ -71,7 +71,7 @@ C does not have any model for functions or pseudo-static data whose lifetimes ar , and any attempt to add them is ad hoc and underspecified because doing it right would require a detailed model compatible with the rest of the language. It's possible to write individual libraries that are meant to be used with such a model, but it requires explicit care to do it right, including documenting that you can't keep pointers to their functions or data after unload, among others. But the problem is that dlopen is recursive, and loading such a module usually involves loading the libraries it depends on, which were written to standard C, not this underspecified "C with unloading modules". -They might have registered atexit handlers or otherwise stored pointers to their code or data in places it persists beyond the caller's knowledge of it. +They might have registered `atexit` handlers or otherwise stored pointers to their code or data in places it persists beyond the caller's knowledge of it. Glibc 'takes care of' the atexit (and dtor) case by doing something wild: executing global ctors at a time other than process termination: at unload time. This has all sorts of weirdness and violates the principle that all dtors (destructors) are executed in reverse order of ctors (constructors). Furthermore, if the library was written assuming dtors run at process exit time, it may be doing something that's not appropriate to happen while the process is continuing to run. One possible implementation could be 'unload modules meant to be loaded as modules, but refuse to unload recursive dependencies and anything with dtors'. From 706f50df3bc765e51937b17751755f0bd33870f2 Mon Sep 17 00:00:00 2001 From: rofl0r Date: Sat, 9 Oct 2021 23:06:21 +0000 Subject: [PATCH 12/12] Update faq.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Érico Nogueira Rolim <34201958+ericonr@users.noreply.github.com> --- faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/faq.md b/faq.md index e7f7d86..2fd07c9 100644 --- a/faq.md +++ b/faq.md @@ -72,7 +72,7 @@ C does not have any model for functions or pseudo-static data whose lifetimes ar It's possible to write individual libraries that are meant to be used with such a model, but it requires explicit care to do it right, including documenting that you can't keep pointers to their functions or data after unload, among others. But the problem is that dlopen is recursive, and loading such a module usually involves loading the libraries it depends on, which were written to standard C, not this underspecified "C with unloading modules". They might have registered `atexit` handlers or otherwise stored pointers to their code or data in places it persists beyond the caller's knowledge of it. -Glibc 'takes care of' the atexit (and dtor) case by doing something wild: executing global ctors at a time other than process termination: at unload time. +Glibc 'takes care' of the `atexit` (and destructor) case by diverging from the expected behavior, by executing global destructors at a time other than process termination: at unload time. This has all sorts of weirdness and violates the principle that all dtors (destructors) are executed in reverse order of ctors (constructors). Furthermore, if the library was written assuming dtors run at process exit time, it may be doing something that's not appropriate to happen while the process is continuing to run. One possible implementation could be 'unload modules meant to be loaded as modules, but refuse to unload recursive dependencies and anything with dtors'. Unfortunately, GCC and GNU Binutils make all libraries appear to have dtors as far as the dynamic linker can tell; the dtors might just be a no-op, and to know that you'd have to interpret code.