[BUG] ring and jaccl both fail to connect (error 60/65) on 4-node M3 Ultra cluster — IP network verified routable, full mesh confirmed

[rogerstom-lgtm]

Description
On a 4-node Mac Studio M3 Ultra cluster with a verified full Thunderbolt 5 mesh, mx.distributed.init() fails to establish connections across all three transports — jaccl and jaccl-ring over Thunderbolt, and plain ring over ethernet — with the same class of error (connection timeout / EHOSTUNREACH), even though the underlying IP network is confirmed fully routable between all node pairs. This reproduces on a trivial 10-element all_sum, so it is not memory-, model-, or load-related.

Environment

4× Mac Studio M3 Ultra (Mac15,14), 256 GB each
macOS 26.4.1 (build 25E253) — uniform across all nodes
MLX 0.31.2 — uniform across all nodes
Python 3.12 (uv-managed venv, per-node)
Full Thunderbolt 5 mesh: all 6 cables present, verified
RDMA enabled via Recovery (rdma_ctl enable); ibv_devices lists all devices
No bridge0 / Thunderbolt Bridge on any node
macOS application firewall disabled on all nodes (socketfilterfw --getglobalstate = disabled on all 4)
EXO present but its io.exo.networksetup daemon disabled; no virtual Thunderbolt interfaces present (confirmed)

What's verified working

mlx.distributed_config --over thunderbolt --dot detects a complete full mesh (all 6 pairs):

a--d [en3/en5] a--c [en4/en5] a--b [en5/en5]
b--d [en3/en4] b--c [en4/en4] c--d [en3/en3]

mlx.distributed_config --auto-setup (both jaccl and ring) completes without error and writes hostfiles
ibv_devinfo shows 3× PORT_ACTIVE per node (full mesh)
IP reachability verified on every relevant path: c1↔c4 and c3↔c4 ping with 0% loss over both the Thunderbolt point-to-point links (192.168.0.x) and the 10GbE management network (10.0.0.x); route get confirms expected interfaces

Failure 1 — JACCL (RDMA over Thunderbolt)
mx.distributed.init(backend="jaccl") fails at init:
[jaccl] Connection attempt 0 waiting 1000 ms
... (backoff 1s/2s/4s/8s)
RuntimeError: [jaccl] Couldn't connect (error: 60)
Persists across: hostfile regeneration via --auto-setup, full reboot of all nodes (to clear any PD exhaustion per mlx-lm#955), re-confirming all ports PORT_ACTIVE, and a clean-boot 10-element all_sum. Note: one TB link was found PORT_DOWN after a reboot and recovered by reseating the cable (all ports PORT_ACTIVE afterward), but JACCL still failed identically after that.
Failure 2 — ring over ethernet (the decisive case)
Switching to plain TCP ring over the flat, routable 10GbE network (--over ethernet; hostfile lists 10.0.0.x addresses that all mutually ping) fails identically:
[ring] Rank 0 accepting
[ring] Rank 0 connecting to 1
[ring] Rank 1 accepting
[ring] Rank 1 connecting to 2
[ring] Rank 2 accepting
[ring] Rank 2 connecting to 3
[ring] Attempt 0 waiting 1000 ms (error: 65 )
[ring] Rank 3 connecting to 0
[ring] Rank 3 accepting
[ring] Attempt 1 waiting 2000 ms (error: 65 )
... (continues backoff)
RuntimeError: [ring] Couldn't connect (error: 60)
The break is consistently at rank 2 → rank 3 and the closing rank 3 → rank 0, with error: 65 (EHOSTUNREACH) — despite c3→c4 and c4→c1 both pinging cleanly at the IP level. Ranks 0→1 and 1→2 connect fine; failures cluster on the later hops including the ring-closing connection.
Reproduction (minimal)
pythonimport mlx.core as mx
world = mx.distributed.init(backend="ring")
x = mx.distributed.all_sum(mx.ones(10))
mx.eval(x)
print(f"rank {world.rank()}/{world.size()} -> {x[0].item()}")
mlx.launch --verbose --backend ring --hostfile ring-eth.json --python /path/to/python ring_test.py
ring-eth.json: 4 hosts, each with its single 10.0.0.x address, generated via mlx.distributed_config --over ethernet --backend ring --auto-setup.

[rogerstom-lgtm]

Flagging the distinction from #2944, since the symptoms can look similar: that issue resolved as a model-support problem (Nemotron supports neither tensor parallelism nor pipelining, which surfaced as a misleading connection-style error). That is not what's happening here.

This failure reproduces on a bare mx.distributed.init(backend="ring") followed by a 10-element all_sum — no model is loaded at any point, so model tensor-parallelism support can't be a factor. The connection never establishes; execution never reaches a model.
Minimal repro:
pythonimport mlx.core as mx
world = mx.distributed.init(backend="ring")
x = mx.distributed.all_sum(mx.ones(10))
mx.eval(x)
print(f"rank {world.rank()}/{world.size()} -> {x[0].item()}")
mlx.launch --verbose --backend ring --hostfile ring-eth.json --python /path/to/python ring_test.py

This is plain ring over the 10GbE network (--over ethernet), with each node's single 10.0.0.x address in the hostfile — all of which mutually ping with 0% loss. It fails at init() with error: 65 (EHOSTUNREACH) on the rank 2→3 and ring-closing rank 3→0 hops, despite those hosts being directly reachable at the IP level. The same failure occurs on jaccl over Thunderbolt.
So whatever the cause, it's in connection establishment before any model is involved — not model architecture support. Happy to provide full --verbose logs and per-node ifconfig / route get output.

Question:

Given that the IP network is verified routable between all pairs, the firewall is off, the full mesh is detected, and the failure reproduces on plain ring over ethernet as well as jaccl: what determines which address/interface each rank binds its listening socket to and advertises to peers? The error: 65 (EHOSTUNREACH) on hops where the hosts demonstrably ping suggests a rank may be binding or advertising an interface its peer cannot reach. Is there an env var or hostfile field to pin the bind interface? Is ring expected to close a 4-node cycle on a flat network, and is rank→rank connection order significant?
Happy to provide full --verbose logs, all hostfiles, and ibv_devinfo / ifconfig / route get output from all nodes.

[rogerstom-lgtm]

Update — narrowed to a single node: connect() reaches SYN_SENT but never emits a SYN, while nc to the same endpoint succeeds

After extensive elimination (table below), this is now isolated to one specific host (c3, rank 2) and one specific behavior that appears to sit below the rendezvous logic.

Signature

mx.distributed.init(backend="jaccl") fails on the c3 rank with [jaccl] Couldn't connect (error: 60) after the 4 standard retry waits. The coordinator (rank 0, c1) is confirmed listening on the management IP and stays up for the whole retry window:

# on rank 0 (c1), captured during a launch:
tcp4  0  0  10.0.0.143.32323  *.*  LISTEN   # stable across the full retry window

From the failing host c3, plain TCP to that exact endpoint succeeds:

c3 % nc -zv 10.0.0.143 32323
Connection to 10.0.0.143 port 32323 [tcp/*] succeeded!

But c3's jaccl process never completes the connection. netstat shows its socket stuck in SYN_SENT, to the correct address:

c3 % netstat -an | grep SYN_SENT
tcp4  0  0  10.0.0.135.56927  10.0.0.143.32323  SYN_SENT

Routing for that destination is correct (en0 = the 10GbE management interface):

c3 % route -n get 10.0.0.143
   route to: 10.0.0.143
  interface: en0

The key observation: a packet capture on c3 across all interfaces, running for the entire retry window, captured zero SYNs to the coordinator port:

c3 % sudo tcpdump -i any -n port 32323
0 packets captured
3833 packets received by filter   # tcpdump was live; nothing matched 32323

So on this host the socket enters SYN_SENT but no SYN is ever placed on any interface — despite a correct route and with pf disabled — while nc to the identical IP:port from the same host connects fine. This points at how jaccl creates/binds the coordinator client socket on this machine, rather than the rendezvous protocol or the network path.

It follows the machine, not the rank

Reordering the hostfile so c3 became rank 1 (instead of rank 2): the failure moved with c3 (rank 1 then failed), not with rank 2. So it is specific to host c3, not a rank-ordering / accept-backlog issue.

Leading (un-confirmed) hypothesis

c3 has several active interfaces carrying 169.254.x.x link-local addresses (en14, en17, en19) in addition to en0 (10.0.0.135) and the three TB /30s. If jaccl is binding/sourcing the coordinator connection from one of those link-local interfaces rather than en0, a SYN toward 10.0.0.143 would have no valid egress and could be dropped silently — which matches "SYN_SENT but nothing on the wire." I could not confirm this because I couldn't determine which local address jaccl binds the coordinator client socket to. Is there a way to make jaccl log the local address/interface it binds for the coordinator connection? That single piece of information would confirm or kill this hypothesis.

Note: I did not fully verify whether the other (working) nodes also have these stray link-local interfaces, so "only c3 has them" is not established — flagging it as the most likely remaining cause, not a proven one.

Environment

• 4× Mac Studio M3 Ultra (Mac15,14), 256 GB, uniform
• macOS 26.4.1 (25E253), uniform across all nodes
• MLX 0.31.2, Python 3.12, uniform
• Full TB5 mesh, all 6 cables; RDMA enabled via Recovery (rdma_ctl enable); ibv_devices populated
• jaccl coordinator rendezvous runs over the 10GbE management net (10.0.0.x / en0); RDMA/TB is the data plane
• The same RDMA fabric previously carried tensor-parallel inference successfully under another launcher, so the fabric itself is functional

Minimal repro

mesh_test.py:

import mlx.core as mx
world = mx.distributed.init(backend="jaccl")
x = mx.distributed.all_sum(mx.ones(10))
mx.eval(x)
print(f"rank {world.rank()}/{world.size()} -> {x[0].item()}")

mlx.launch --backend jaccl --hostfile jaccl-mesh.json --python <py> mesh_test.py

Fails at init() on the c3 rank, before reaching the all_sum.

Eliminated (each with evidence)

Candidate cause	Status	Evidence
TB/RDMA mesh incomplete	ruled out	--dot shows 6/6 edges; verified after warm and cold reboot
Wedged TB link	found + fixed	one link (c3↔c4) was physically wedged (would not train); reseated; mesh whole since
Hostfile rdma matrix asymmetric	ruled out	matrix symmetric, matches --dot topology
Hostfile stale / mismatched across nodes	ruled out	identical md5 on all 4 nodes
Path / home-dir mismatch	ruled out	shims at identical abs paths; hostfile present per node
EXO interference	ruled out	EXO killed; port 52415 dead on all 4
GPU wired limit	ruled out	iogpu.wired_limit_mb=229376 on all 4
Wrong barrier size	ruled out	uses mx.ones(10) (per #3207)
PD exhaustion (#955)	ruled out	fails on a freshly rebooted c3
Transient network-stack state	ruled out	fails after a genuine cold boot (power off → power on)
Firewall (pf)	ruled out	pfctl -s info → Status: Disabled, no rules
Coordinator unreachable	ruled out	nc from c3 → 10.0.0.143:32323 succeeds

Caveat on the capture

tcpdump showed 0 packets; the capture ran on -i any for the full ~15s retry window (so it should have overlapped the c3 process), and it is corroborated by the netstat SYN_SENT observation — two independent observations of "socket in SYN_SENT, no SYN on the wire." A maintainer may still want to re-confirm with a capture explicitly time-bracketed around the c3 process PID.

Summary

Socket reaches SYN_SENT but no SYN is emitted, on a host with a correct route and no firewall, while nc to the same endpoint works — reproducible, isolated to one machine, surviving warm and cold reboot, with every other identified cause eliminated. The most useful next step on the MLX side is a way to log the local address jaccl binds for the coordinator connection.