TLS trust anchor not recognized for OAuth2 communication

[yoe]

As of docker 29, doing a "docker pull" to a registry that requires OAuth2 authentication with private TLS certificates no longer works if TLS verification is enabled.

Steps to reproduce:

1. Run a private registry that requires OAuth2 for authentication with both the registry and the OAuth2 host having TLS certificates under a private CA (e.g., by using the GitLab registry)
2. Install the private CA for the registry in /etc/docker/certs.d/<registry hostname>/ca.crt
3. Also install the private CA for the OAuth2 host in /etc/docker/certs.d/<OAuth2 hostname>/ca.crt
4. docker network create dind
5. docker volume create dind-certs
6. docker run -ti --rm --privileged -e DOCKER_TLS_VERIFY=1 -e DOCKER_TLS_CERTDIR=/certs -e DOCKER_CERT_PATH=/certs/client --name docker --network dind -v /etc/docker/certs.d:/etc/docker/certs.d:ro -v dind-certs:/certs docker:dind
7. docker run -ti --rm -e DOCKER_TLS_VERIFY=1 -e DOCKER_TLS_CERTDIR=/certs -e DOCKER_CERT_PATH=/certs/client --network dind -v /etc/docker/certs.d:/etc/docker/certs.d:ro -v dind-certs:/certs docker:latest docker pull <private registry URL>

Expected result: the image is pulled with no error.

Actual result: Error response from daemon: failed to resolve reference "registry.gitlab.<censored>/<censored>:latest": failed to authorize: failed to fetch anonymous token: Get "https://gitlab.<censored>/jwt/auth?scope=repository%3A<censored>%3Apull&service=container_registry": tls: failed to verify certificate: x509: certificate signed by unknown authority, with a similar message in the container log.

When falling back to docker:28-dind as the image in step 6, the image pull works with no error.

(not sure if this is the right place to report this issue; if not, feel free to redirect)

[martonivan]

My experience may strong relates: I was installing Docker Engine on Ubuntu 24.04.03, following the installation guide located at https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository

My local docker registry is using mTLS authentication using the key and certificate found under /etc/docker/certs.d/:/client.key and /etc/docker/certs.d/:/client.cert respectively.

According to the strace output, these files are being read by dockerd when attempting to pull the images but are not in use, how it can be seen on server side.

Surprisingly, installing docker-ce_28.5.2-1ubuntu.24.04noble_amd64.deb then reinstalling docker-ce 29.1.3 (https://download.docker.com/linux/ubuntu/dists/noble/pool/stable/amd64/docker-ce_29.1.3-1~ubuntu.24.04~noble_amd64.deb) resolved the issue.

On a fresh reproduction, the same worked when I did not just simply downgrade and upgrade again, but instead used the v29.1.3 Debian deb package for the same version.

Originally the issue was experienced on Ubuntu 24.04 with 29.1.3, but the same applies to 5:29.1.4-1ubuntu.24.04noble and for Ubuntu 22.04 (jammy) too so this must be related to the packaging of the recent versions.

[emanueleaina]

I have a private GitLab test instance with self-signed TLS certificates and when using insecure-registries everything worked up to Docker 28, but with Docker 29 I now get:

Failed to pull image with policy "always": Error response from daemon: failed to resolve reference "registry.gitlab-test.local/test/imagestest-100286": failed to authorize: failed to fetch oauth token: Post "https://gitlab-test.local/jwt/auth": tls: failed to verify certificate: x509: certificate signed by unknown authority (manager.go:238:0s)

Note that the TLS validation is correctly skipped when accessing registry.gitlab-test.local as it gets sent to gitlab-test.local for authentication, but then Docker 29 tries to validate the TLS certificate fo the authentication endpoint and fails.