tryResolveLibrary in pkg/nvlib/info/root.go checks each candidate path
with filepath.EvalSymlinks (via resolveLink) and returns the first one
that resolves. EvalSymlinks succeeds for anything that exists, directories
included. So a stray directory named after the library in an early search
path shadows the real library in a later one.
/usr/lib64 is searched before /usr/lib/x86_64-linux-gnu. On Ubuntu the
driver installs the library in the latter, so an empty directory
/usr/lib64/libnvidia-ml.so.1 (left behind by a buggy script or package) is
enough to make tryResolveLibrary return the directory path instead of the
real library.
Reproduction
On an Ubuntu host with the NVIDIA driver installed:
sudo mkdir /usr/lib64/libnvidia-ml.so.1
Any consumer of tryResolveLibrary (for example
Interface.ResolveNVMLLibraryPath()) now resolves to the directory, and the
dlopen that follows fails.
Real-world impact
The kubelet plugins in kubernetes-sigs/dra-driver-nvidia-gpu carry a copy of
this discovery code and crash-loop with ERROR_LIBRARY_NOT_FOUND when this
happens. That was reported as kubernetes-sigs/dra-driver-nvidia-gpu#1272 and
fixed on their side in kubernetes-sigs/dra-driver-nvidia-gpu#1275. The copy
here is still affected, and go-nvlib is consumed by the device plugin, the
container toolkit, and the DRA driver, so a fix here reaches all of them.
Expected behavior
Resolution should skip candidates that are not regular files and keep
searching the remaining paths. If nothing matches, fall back to the
documented behavior of returning the library name as is.
Proposed fix
After resolveLink succeeds, os.Stat the resolved path and skip the
candidate unless info.Mode().IsRegular(). I have the fix and a unit test
ready and will open a PR for this issue.
tryResolveLibraryinpkg/nvlib/info/root.gochecks each candidate pathwith
filepath.EvalSymlinks(viaresolveLink) and returns the first onethat resolves.
EvalSymlinkssucceeds for anything that exists, directoriesincluded. So a stray directory named after the library in an early search
path shadows the real library in a later one.
/usr/lib64is searched before/usr/lib/x86_64-linux-gnu. On Ubuntu thedriver installs the library in the latter, so an empty directory
/usr/lib64/libnvidia-ml.so.1(left behind by a buggy script or package) isenough to make
tryResolveLibraryreturn the directory path instead of thereal library.
Reproduction
On an Ubuntu host with the NVIDIA driver installed:
Any consumer of
tryResolveLibrary(for exampleInterface.ResolveNVMLLibraryPath()) now resolves to the directory, and thedlopen that follows fails.
Real-world impact
The kubelet plugins in kubernetes-sigs/dra-driver-nvidia-gpu carry a copy of
this discovery code and crash-loop with
ERROR_LIBRARY_NOT_FOUNDwhen thishappens. That was reported as kubernetes-sigs/dra-driver-nvidia-gpu#1272 and
fixed on their side in kubernetes-sigs/dra-driver-nvidia-gpu#1275. The copy
here is still affected, and go-nvlib is consumed by the device plugin, the
container toolkit, and the DRA driver, so a fix here reaches all of them.
Expected behavior
Resolution should skip candidates that are not regular files and keep
searching the remaining paths. If nothing matches, fall back to the
documented behavior of returning the library name as is.
Proposed fix
After
resolveLinksucceeds,os.Statthe resolved path and skip thecandidate unless
info.Mode().IsRegular(). I have the fix and a unit testready and will open a PR for this issue.