[PD] Health API can panic with nil pointer dereference during cluster startup/restart

[FangCundi]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Start a small TiDB cluster and poll the PD health API while the cluster is still in a startup/restart window.

The topology used when the panic was observed was:

PD:   1 node
TiKV: 3 nodes
TiDB: 2 nodes

A simplified way to trigger the same class of condition is:

# Terminal 1: start a TiDB cluster with one PD, three TiKV nodes, and two TiDB nodes.
# The exact deployment method should not matter; the important condition is that
# PD has started serving HTTP requests while other cluster/member state may still
# be initializing or reconnecting.

# Terminal 2: poll the PD health API continuously during startup/restart.
while true; do
  curl -sS "http://127.0.0.1:2379/pd/api/v1/health" || true
  sleep 0.2
done

Then restart or bounce cluster components while the health API is still being polled:

# Restart PD/TiKV/TiDB components, or restart the whole small cluster.
# Keep the health polling loop running while components are rejoining.

The panic happened after PD had already reached the point where it was serving requests. The log just before the panic showed the service-discovery/startup phase finishing:

[pd] waiting for DNS: pd
[pd] waiting for DNS: tikv1
[pd] waiting for DNS: tikv2
[pd] waiting for DNS: tikv3
[pd] waiting for DNS: tidb1
[pd] waiting for DNS: tidb2
[pd] DNS ready.
[pd] using binary=/path/to/pd-server

The health API request then went through the normal PD HTTP middleware path and entered the PD health handler:

github.com/tikv/pd/server/api.(*healthHandler).GetHealthStatus(...)

2. What did you expect to see? (Required)

The PD health API should not panic.

If PD is not fully ready, or if member/etcd/cluster state is temporarily unavailable during startup or restart, the health API should return a normal unhealthy response, for example:

HTTP 503 Service Unavailable

or another explicit error response.

A health endpoint is normally called by deployment scripts, orchestrators, load balancers, and monitoring systems. It should be safe to call repeatedly, including during startup, restart, and rejoin windows.

3. What did you see instead (Required)

PD logged a nil pointer panic while handling the health API request:

[negroni] PANIC: runtime error: invalid memory address or nil pointer dereference

goroutine 517 [running]:
github.com/urfave/negroni/v3.(*Recovery).ServeHTTP.func1()
panic({ ... })

go.etcd.io/etcd/client/v3.(*Client).Ctx(...)
go.etcd.io/etcd/client/v3/client.go:159

github.com/tikv/pd/server/cluster.GetMembers(...)
github.com/tikv/pd/server/cluster/cluster.go:2729

github.com/tikv/pd/server/api.(*healthHandler).GetHealthStatus(...)
github.com/tikv/pd/server/api/health.go:53

github.com/tikv/pd/server/api.(*serviceMiddlewareBuilder).createHandler.WrapFunc.func1(...)
github.com/urfave/negroni/v3.HandlerFunc.ServeHTTP(...)
github.com/tikv/pd/server/api.(*rateLimitMiddleware).ServeHTTP(...)
github.com/tikv/pd/server/api.(*auditMiddleware).ServeHTTP(...)
github.com/tikv/pd/server/api.(*requestInfoMiddleware).ServeHTTP(...)
github.com/gorilla/mux.(*Router).ServeHTTP(...)

The relevant panic path is:

healthHandler.GetHealthStatus
  -> cluster.GetMembers
  -> etcd client Ctx()
  -> nil pointer dereference

This looks like a missing nil/state check around the PD health handler's member retrieval path. Even if PD is in a transient state, a health API request should not reach a process-level panic.

4. What is your TiDB version? (Required)

SELECT tidb_version();

8.0.11-TiDB-v8.5.6

The PD stack trace points to the corresponding v8.5.6 PD source path around:

pd/server/api/health.go:53
pd/server/cluster/cluster.go:2729

[ti-chi-bot]

Welcome @FangCundi! It looks like this is your first issue to tikv/pd 🎉