CHEF-37610: Fix error 126 by registering DLL search directory for Windows PowerShell (.NET 481) path

[johnmccrae]

Summary

Fixes a critical DLL-loading regression (Windows error 126) affecting the Windows PowerShell (.NET Framework 4.8.1 / Desktop) execution path when run against real Chef-18 installations. Also adds regression tests that would have caught this bug, and closes the gap that let it slip through the existing test suite.

JIRA Issue

CHEF-37610

Root Cause

Windows' default DLL search order used by FFI (LOAD_LIBRARY_SEARCH_DEFAULT_DIRS) does not include the directory containing the DLL being loaded, nor PATH, nor the current working directory. It only searches the hosting EXE's directory, System32, and any directories explicitly registered via AddDllDirectory/SetDllDirectory.

Chef.PowerShell.Wrapper.dll depends on native VC++ runtime DLLs (vcruntime140.dll, msvcp140.dll, etc.) that live alongside it in the gem's bin_dir. The Pwsh (.NET 10 / PowerShell Core) execution path already registered this directory correctly in Pwsh#exec. However, the base PowerShell class's #initialize (.NET Framework 4.8.1 / Windows PowerShell Desktop path) never registered bin_dir as a search directory at all.

On a target machine without the VC++ redistributable already present in System32 (such as the Chef-18 test environment where this was discovered), the wrapper DLL loaded fine via its absolute path, but its dependent native DLLs could not be resolved — producing error 126.

Why Existing Tests Didn't Catch This

• The integration test helper for the Desktop path (raw_powershell) used ChefPowerShell::PowerShell.allocate plus manual instance variable injection to bypass gem path resolution — which also bypassed #initialize entirely, the exact method containing the bug.
• AddDllDirectory/SetDefaultDllDirectories mutate global, process-wide Windows search-path state that is never unregistered. Because the whole rspec suite runs in a single Ruby process, earlier Pwsh-path tests had already registered the same physical directory used by the Desktop-path DLLs, silently masking the missing registration.
• Dev/CI machines often already have the VC++ redistributable installed under System32 (verified on this workstation), which satisfies the dependency regardless of whether the directory was registered — so even a naive "does it run" test can pass while the underlying registration bug is still present.

Changes Made

• Relocated the shared Kernel32 FFI module ( SetDllDirectoryA, SetDefaultDllDirectories, AddDllDirectory, register_search_directory) from being duplicated/nested under Pwsh to a single shared definition directly under ChefPowerShell in lib/chef-powershell/powershell.rb.
• Added DLL search directory registration to PowerShell#initialize (mirroring the pattern already used in Pwsh#exec): calls SetDefaultDllDirectories, then register_search_directory(bin_dir) (raising LoadError on failure), then SetDllDirectoryA(bin_dir) as defense-in-depth for older Windows versions — before invoking exec.
• Removed the now-duplicate Kernel32 module from lib/chef-powershell/pwsh.rb. All bare Kernel32.* references throughout the codebase (pwsh.rb, smoke_test_dlls.rb, verify_hab_build.rb) continue to resolve correctly via Ruby lexical scoping with no further changes required.
• Updated fully-qualified references in spec/integration/dll_integration_spec.rb from ChefPowerShell::Pwsh::Kernel32 to ChefPowerShell::Kernel32.

New Regression Tests

Added two tests to spec/integration/dll_integration_spec.rb under "PowerShell#initialize DLL directory registration (process isolation)":

• Subprocess isolation test: spawns a fresh, single-purpose Ruby process (script written to a temp file, executed via Open3.capture3) that calls the real ChefPowerShell::PowerShell.new constructor — reproducing the exact single-purpose-process condition production code runs under, with no possibility of state bleed from other examples in the suite.
• Deterministic spy-based test: stubs ChefPowerShell::Kernel32 with and_call_original and asserts SetDefaultDllDirectories, register_search_directory(bin_dir), and SetDllDirectoryA(bin_dir) are actually invoked during PowerShell.new. Unlike the subprocess test, this does not depend on whether the VC++ redistributable happens to already be present on the machine running the suite — verified by temporarily reverting the fix and confirming this test fails deterministically, then restoring the fix and confirming it passes.

Testing

• Full rspec suite: 68 examples, 0 failures (66 pre-existing + 2 new).
• Cookstyle/chefstyle lint: no offenses on modified files.
• Verified the new spy-based test fails deterministically when the fix is reverted, and passes with the fix restored, confirming it provides real regression coverage.

AI Assistance

This work was completed with AI assistance following Progress AI policies.