The hang happens when resolving hostname. The exact moment depends on the race scenario. For instance in this example Rally hangs while downloading ES artifact needed to start local cluster:
% esrally race --track=geonames --challenge=append-no-conflicts-index-only --distribution-version=9.2.4 \
--track-params="ingest_percentage:10"
____ ____
/ __ \____ _/ / /_ __
/ /_/ / __ `/ / / / / /
/ _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
/____/
[INFO] Race id is [6c2ec092-d937-4b8c-b408-1eceef35ac5a]
[INFO] Preparing for race ...
Rally logs are not particularly useful, but at least we see the process ID - 26858.
[..]
2026-04-14 18:23:13,703 ActorAddr-(T|:63007)/PID:26858 esrally.actor INFO Actor initialized: NodeMechanicActor (pid=26858)
2026-04-14 18:23:13,674 ActorAddr-(T|:62984)/PID:26821 esrally.mechanic.mechanic INFO Cluster consisting of [{'host': '127.0.0.1', 'port': 39200}] will be provisioned by Rally.
2026-04-14 18:23:13,707 ActorAddr-(T|:63007)/PID:26858 esrally.mechanic.mechanic INFO Starting node(s) [0] on [127.0.0.1].
Thespian debug logs contain a clue. Process 26858 was supposed to start Elasticsearch 9.2.4 artifact download.
(This log never propagates to a log file because process gets stuck, and Thespian never gets an opportunity to process the queued message).
2026-04-14 20:23:14.309499 p26858 dbg actualTransmit of TransportIntent(ActorAddr-(T|:62957)-pending-ExpiresIn_0:04:59.999983-<class 'logging.LogRecord'>-<LogRecord: esrally.mechanic.supplier, 20, /Users/grzegorz/src/rally/esrally/mechanic/supplier.py, 599, "Resolved download URL [https://artifacts.elas...-quit_0:04:59.999980)
2026-04-14 20:23:14.309917 p26858 dbg actualTransmit of TransportIntent(ActorAddr-(T|:62957)-pending-ExpiresIn_0:04:59.999986-<class 'logging.LogRecord'>-<LogRecord: esrally.mechanic.supplier, 20, /Users/grzegorz/src/rally/esrally/mechanic/supplier.py, 602, "Starting download of Elasticsearch [9.2.4]">-quit_0:04:59.999983)
py-spy shows process is stuck waiting for getaddrinfo:
% sudo py-spy dump --pid 26858
Password:
Process 26858: /Users/grzegorz/src/rally/.venv/bin/python3 /Users/grzegorz/src/rally/.venv/bin/esrally race --track=geonames --challenge=append-no-conflicts-index-only --distribution-version=9.2.4 --track-params=ingest_percentage:10
Python v3.13.0 (/Users/grzegorz/.local/share/uv/python/cpython-3.13.7-macos-aarch64-none/bin/python3.13)
Thread 0x1F91F98C0 (active): "MainThread"
getaddrinfo (socket.py:977) <--- HERE
create_connection (urllib3/util/connection.py:60)
_new_conn (urllib3/connection.py:204)
connect (urllib3/connection.py:759)
_validate_conn (urllib3/connectionpool.py:1093)
_make_request (urllib3/connectionpool.py:464)
urlopen (urllib3/connectionpool.py:787)
urlopen (urllib3/poolmanager.py:457)
request_encode_url (urllib3/_request_methods.py:182)
request (urllib3/_request_methods.py:135)
_request (esrally/utils/net.py:298)
_download_http (esrally/utils/net.py:193)
download_http (esrally/utils/net.py:228)
download (esrally/utils/net.py:271)
[..]
The C-level call stack shows a characteristic sequence:
getaddrinfo
si_addrinfo
_gai_nat64_second_pass ← NAT64 check triggered because host is dual-stack
nw_nat64_v4_address_requires_synthesis
nw_path_evaluator_evaluate ← evaluates network path for NAT64 synthesis
...
os_log_type_enabled
_os_log_preferences_refresh ← DEADLOCK HERE
_sigtramp (Python signal handler fires ~3000x during the 5s sample)
Details
% sudo sample 26858 5
Sampling process 26858 for 5 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Sample analysis of process 26858 written to file /tmp/python3_2026-04-14_203357_MXDv.sample.txt
Analysis of sampling python3 (pid 26858) every 1 millisecond
Process: python3.13 [26858]
Path: /Users/USER/*/python3.13
Load Address: 0x102214000
Identifier: python3.13
Version: ???
Code Type: ARM64
Platform: macOS
Parent Process: python3.13 [26857]
Target Type: live task
Date/Time: 2026-04-14 20:33:57.837 +0200
Launch Time: 2026-04-14 20:23:13.699 +0200
OS Version: macOS 26.4.1 (25E253)
Report Version: 7
Analysis Tool: /usr/bin/sample
Physical footprint: 24.1M
Physical footprint (peak): 24.1M
Idle exit: untracked
----
Call graph:
3511 Thread_1741820 DispatchQueue_1: com.apple.main-thread (serial)
3511 start (in dyld) + 6992 [0x18d033da4]
3511 Py_BytesMain (in libpython3.13.dylib) + 36 [0x103708258]
3511 pymain_main (in libpython3.13.dylib) + 468 [0x103708438]
3511 Py_RunMain (in libpython3.13.dylib) + 1068 [0x103711d58]
3511 pymain_run_file (in libpython3.13.dylib) + 76 [0x1037abaa0]
3511 pymain_run_file_obj (in libpython3.13.dylib) + 164 [0x1037abc60]
3511 _PyRun_AnyFileObject (in libpython3.13.dylib) + 80 [0x1037abd80]
3511 _PyRun_SimpleFileObject (in libpython3.13.dylib) + 256 [0x1037abfc0]
3511 pyrun_file (in libpython3.13.dylib) + 164 [0x1037ac4cc]
3511 run_mod.llvm.7635899141245430400 (in libpython3.13.dylib) + 148 [0x10354dd48]
3511 run_eval_code_obj (in libpython3.13.dylib) + 144 [0x10354e7d8]
3511 PyEval_EvalCode (in libpython3.13.dylib) + 132 [0x10354e930]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 63560 [0x1035a12b4]
3511 type_call (in libpython3.13.dylib) + 148 [0x103834c9c]
3511 slot_tp_init (in libpython3.13.dylib) + 512 [0x103836f1c]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 63560 [0x1035a12b4]
3511 type_call (in libpython3.13.dylib) + 148 [0x103834c9c]
3511 slot_tp_init (in libpython3.13.dylib) + 512 [0x103836f1c]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 41576 [0x10359bcd4]
3511 type_call (in libpython3.13.dylib) + 148 [0x103834c9c]
3511 slot_tp_init (in libpython3.13.dylib) + 296 [0x103836e44]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 41576 [0x10359bcd4]
3511 slot_tp_call (in libpython3.13.dylib) + 216 [0x10368fa7c]
3511 _PyObject_Call_Prepend (in libpython3.13.dylib) + 152 [0x10368ff98]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 55392 [0x10359f2cc]
3511 method_vectorcall.llvm.7737475008596783471 (in libpython3.13.dylib) + 300 [0x103812358]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 55392 [0x10359f2cc]
3511 method_vectorcall.llvm.7737475008596783471 (in libpython3.13.dylib) + 300 [0x103812358]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 55392 [0x10359f2cc]
3511 method_vectorcall.llvm.7737475008596783471 (in libpython3.13.dylib) + 300 [0x103812358]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 55392 [0x10359f2cc]
3511 method_vectorcall.llvm.7737475008596783471 (in libpython3.13.dylib) + 300 [0x103812358]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 55392 [0x10359f2cc]
3511 method_vectorcall.llvm.7737475008596783471 (in libpython3.13.dylib) + 300 [0x103812358]
3511 _PyEval_EvalFrameDefault (in libpython3.13.dylib) + 74764 [0x1035a3e78]
3511 cfunction_call (in libpython3.13.dylib) + 76 [0x1037d33c0]
3511 socket_getaddrinfo (in libpython3.13.dylib) + 480 [0x103e0e0f0]
3511 getaddrinfo (in libsystem_info.dylib) + 172 [0x18d404254]
3511 si_addrinfo (in libsystem_info.dylib) + 1388 [0x18d404868]
3511 _gai_nat64_second_pass (in libsystem_info.dylib) + 276 [0x18d418304]
3511 nw_nat64_v4_address_requires_synthesis (in Network) + 240 [0x196660ad4]
3511 nw_path_create_evaluator_for_endpoint (in Network) + 72 [0x19686bc90]
3511 nw_path_evaluator_evaluate(NWConcrete_nw_path_evaluator*, int*) (in Network) + 2160 [0x19686cde0]
3511 nw_path_snapshot_path(NWConcrete_nw_path*) (in Network) + 68 [0x19686da28]
3511 _nw_path_update_is_viableTm (in Network) + 84 [0x195b1d71c]
3511 ??? (in Network) load address 0x19575a000 + 0xdd1d4 [0x1958371d4]
3511 ??? (in Network) load address 0x19575a000 + 0xdda54 [0x195837a54]
3511 nw_path_access_agent_cache (in Network) + 188 [0x19689c030]
3511 ??? (in Network) load address 0x19575a000 + 0xe101c [0x19583b01c]
3511 ??? (in Network) load address 0x19575a000 + 0xe70fc [0x1958410fc]
3511 ??? (in Network) load address 0x19575a000 + 0xe71c4 [0x1958411c4]
3511 ??? (in Network) load address 0x19575a000 + 0xe395c [0x19583d95c]
3511 ??? (in Network) load address 0x19575a000 + 0x23099c [0x19598a99c]
3511 os_log_type_enabled (in libsystem_trace.dylib) + 772 [0x18d117ad4]
3002 _os_log_preferences_refresh (in libsystem_trace.dylib) + 36 [0x18d11715c]
+ 3001 _sigtramp (in libsystem_platform.dylib) + 0 [0x18d3f976c]
+ 1 _sigtramp (in libsystem_platform.dylib) + 56 [0x18d3f97a4]
+ 1 signal_handler (in libpython3.13.dylib) + 32 [0x103e48e00]
+ 1 trip_signal (in libpython3.13.dylib) + 56 [0x103e4827c]
509 _os_log_preferences_refresh (in libsystem_trace.dylib) + 56 [0x18d117170]
Similar problem was observed in https://bugs.ruby-lang.org/issues/21790 where MacOS releases older than Tahoe were confirmed non-affected.
It seems that the problem is triggered when IPv6 is active on a network interface and loopback interface, but there's no global IPv6 connectivity. That's why NAT64 code path is activated.
% ifconfig en6
en6: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=6464<VLAN_MTU,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
ether 30:23:03:8d:73:19
inet 192.168.1.61 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::1417:1121:3fbe:1be5%en6 prefixlen 64 secured scopeid 0xd <--- HERE (link-local address only)
nd6 options=201<PERFORMNUD,DAD>
media: autoselect (1000baseT <full-duplex>)
status: active
% ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128 <--- HERE (required to trigger problematic code path)
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=201<PERFORMNUD,DAD>
As a workaround IPv6 can be disabled as follows, but ::1 address removal might not persist the reboot.
# list network interfaces
networksetup -listallnetworkservices
# disable IPv6
sudo networksetup -setv6off "<interface-name>"
# remove ::1 address from loopback
sudo ifconfig lo0 inet6 ::1 delete
The alternative would be to enforce AF_INET address family in Rally code.
The hang happens when resolving hostname. The exact moment depends on the race scenario. For instance in this example Rally hangs while downloading ES artifact needed to start local cluster:
Rally logs are not particularly useful, but at least we see the process ID - 26858.
Thespian debug logs contain a clue. Process 26858 was supposed to start Elasticsearch 9.2.4 artifact download.
(This log never propagates to a log file because process gets stuck, and Thespian never gets an opportunity to process the queued message).
py-spyshows process is stuck waiting forgetaddrinfo:The C-level call stack shows a characteristic sequence:
Details
Similar problem was observed in https://bugs.ruby-lang.org/issues/21790 where MacOS releases older than Tahoe were confirmed non-affected.
It seems that the problem is triggered when IPv6 is active on a network interface and loopback interface, but there's no global IPv6 connectivity. That's why NAT64 code path is activated.
As a workaround IPv6 can be disabled as follows, but
::1address removal might not persist the reboot.The alternative would be to enforce
AF_INETaddress family in Rally code.