Homebrew does not run relocation on files when HOMEBREW_BOTTLE_DOMAIN points to a mirror site

[iBug]

brew doctor output

Your system is ready to brew.

Verification

• I ran brew update twice and am still able to reproduce my issue.
• My "brew doctor output" above says Your system is ready to brew or a definitely unrelated Tier message.
• This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

brew config output

HOMEBREW_VERSION: 6.0.11
ORIGIN: https://mirrors.ustc.edu.cn/brew.git
HEAD: 6bd951d96e7ebc54787799dba77bfb26ec956c4c
Last commit: 5 days ago
Branch: stable
Core tap: N/A
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_API_DOMAIN: https://mirrors.ustc.edu.cn/homebrew-bottles/api
HOMEBREW_BOTTLE_DOMAIN: https://mirrors.ustc.edu.cn/homebrew-bottles
HOMEBREW_BREW_GIT_REMOTE: https://mirrors.ustc.edu.cn/brew.git
HOMEBREW_CASK_OPTS: []
HOMEBREW_CORE_GIT_REMOTE: https://mirrors.ustc.edu.cn/homebrew-core.git
HOMEBREW_DOWNLOAD_CONCURRENCY: 64
HOMEBREW_EDITOR: vim
HOMEBREW_FORBID_PACKAGES_FROM_PATHS: set
HOMEBREW_MAKE_JOBS: 32
HOMEBREW_REQUIRE_TAP_TRUST: set
Homebrew Ruby: 4.0.5 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/4.0.5_1/bin/ruby

Toolchain information is truncated as they're not considered relevant.

What were you trying to do (and why)?

Install formulae with dynamically-linked binaries from mirrors. Tested examples include git-delta and zstd.

Reason for using mirrors is simple: Mainland China's network connectivity to GitHub is poor and mirrors are readily available (and fast).

This has been reported multiple times to mirror sites: ustclug/discussions#382, tuna/issues#1749, tuna/issues#2555 (this was the initial finding in the right direction).

What happened (include all command output)?

The command zstd doesn't execute with the following output:

-bash: /home/linuxbrew/.linuxbrew/bin/zstd: cannot execute: required file not found

ldd and readelf shows placeholders for paths:

$ readelf -l /home/linuxbrew/.linuxbrew/bin/zstd
...
     [Requesting program interpreter: @@HOMEBREW_PREFIX@@/lib/ld.so]
...
$ readelf -d /home/linuxbrew/.linuxbrew/bin/zstd
...
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../lib:@@HOMEBREW_PREFIX@@/Cellar/zstd/1.5.7_1/lib:@@HOMEBREW_PREFIX@@/opt/gcc/lib/gcc/current:@@HOMEBREW_PREFIX@@/opt/lz4/lib:@@HOMEBREW_PREFIX@@/opt/xz/lib:@@HOMEBREW_PREFIX@@/opt/zlib-ng-compat/lib:@@HOMEBREW_PREFIX@@/lib]
...

What did you expect to happen?

The zstd binary is valid and ready to use.

Step-by-step reproduction instructions (running brew commands)

export HOMEBREW_BREW_GIT_REMOTE=https://mirrors.ustc.edu.cn/brew.git
export HOMEBREW_BOTTLE_DOMAIN=https://mirrors.ustc.edu.cn/homebrew-bottles
export HOMEBREW_API_DOMAIN=https://mirrors.ustc.edu.cn/homebrew-bottles/api
brew install zstd

Notes

It is worth noting that simply removing the environment variables (reverting to the official remote) fixes this:

unset HOMEBREW_BREW_GIT_REMOTE
unset HOMEBREW_BOTTLE_DOMAIN
unset HOMEBREW_API_DOMAIN
brew reinstall zstd

Until #12759 has been properly fixed, here is a verified valid workaround:

diff --git a/Library/Homebrew/formula_installer.rb b/Library/Homebrew/formula_installer.rb
index 1dba7fe059..2bd30fbde7 100644
--- a/Library/Homebrew/formula_installer.rb
+++ b/Library/Homebrew/formula_installer.rb
@@ -1628,6 +1628,9 @@ on_request: installed_on_request?, options:)

     keg = Keg.new(formula.prefix)
     skip_linkage = formula.bottle_specification.skip_relocation?(tab:)
+    # Homebrew doesn't download bottle manifests from 3rd-party domains.
+    # Until Homebrew/brew#12759 has been fixed, we can work around by forcing relocation upon manifests missing.
+    skip_linkage = false if tab.changed_files.blank?
     keg.replace_placeholders_with_locations(tab.changed_files, skip_linkage:)

     cellar = formula.bottle_specification.tag_to_cellar(Utils::Bottles.tag)

Another fix proposed in ustclug/discussions#382 (comment) seems more "canonical" to me, but is no longer working for Homebrew 6.0.11:

diff --git a/Library/Homebrew/bottle.rb b/Library/Homebrew/bottle.rb
index ccaedbcf65..3451794159 100644
--- a/Library/Homebrew/bottle.rb
+++ b/Library/Homebrew/bottle.rb
@@ -300,7 +300,8 @@ class Bottle

   sig { returns(T.nilable(Resource::BottleManifest)) }
   def github_packages_manifest_resource
-    return if @resource.download_strategy != CurlGitHubPackagesDownloadStrategy
+    return if @resource.download_strategy != CurlGitHubPackagesDownloadStrategy \
+      && Homebrew::EnvConfig.bottle_domain == HOMEBREW_BOTTLE_DEFAULT_DOMAIN

     @github_packages_manifest_resource ||= T.let(
       begin

[MikeMcQuaid]

It is worth noting that simply removing the environment variables (reverting to the official remote) fixes this:

unset HOMEBREW_BREW_GIT_REMOTE
unset HOMEBREW_BOTTLE_DOMAIN
unset HOMEBREW_API_DOMAIN
brew reinstall zstd
Until #12759 has been properly fixed, here is a verified valid workaround

We'll review a PR for this, we don't support third-party mirror issues.

[iBug]

This is not a 3rd-party mirror issue. Mirrors are providing files from ghcr.io verbatim. It's Homebrew's code that skips fetching manifests when it shouldn't.

[MikeMcQuaid]

This is not a 3rd-party mirror issue. Mirrors are providing files from ghcr.io verbatim.

This definitely is a 3rd-party issue. You said yourself, if you don't use the 3rd-party mirror: it goes away.

The bottle archive may be mirrored but that's not enough. Homebrew stores critical information in bottle manifests e.g. in the OCI manifest’s sh.brew.tab annotation, not in the tarball blob.

I will open a PR to improve behaviour and output warnings so that mirrors know that they aren't working correctly here. I also expect to deprecate HOMEBREW_BOTTLE_DOMAIN in future in favour of HOMEBREW_ARTIFACT_DOMAIN..

A legacy flat bottle mirror does not provide Homebrew with the corresponding OCI manifest. The correct solution for mirror operators is to proxy both GHCR blobs and manifests and have users configure that proxy through HOMEBREW_ARTIFACT_DOMAIN.

[iBug]

The problem is brew skips any attempt to download the manifests when using HOMEBREW_BOTTLE_DOMAIN. This can be fixed by trying to fetch the manifest from the mirror site before falling back to ghcr.io.