Disjoint per-rank ZE_AFFINITY_MASK deadlocks ZE IPC exchange (drmfd fallback); CCL_ZE_IPC_EXCHANGE=none not honored #217

Summary

When two ranks on one node are each restricted to a single, different GPU via ZE_AFFINITY_MASK (rank0 -> 0, rank1 -> 1), oneCCL initialization hangs indefinitely in the ZE IPC/fd exchange when the pidfd path is unavailable. Under Docker's default seccomp profile the pidfd path reports unsupported (pidfd is not supported, fallbacks to drmfd exchange mode) and the drmfd fallback never completes. CCL_ZE_IPC_EXCHANGE=none is acknowledged by config (value of CCL_ZE_IPC_EXCHANGE changed to be none (default:pidfd)) but the drmfd fallback warning still appears and the hang persists — none appears not to be honored on this path.

With --cap-add SYS_PTRACE (or --security-opt seccomp=unconfined), the pidfd path works and everything completes normally — so the exchange itself is functional; the problem is the silent drmfd fallback.

Additionally, without a /dev/dri bind mount into the container (--device /dev/dri only, so /dev/dri/by-path is absent), the exchange fails fast instead with:

oneCCL: ze_fd_manager.cpp:144 init_device_fds: EXCEPTION: opendir failed: could not open device directory

Minimal reproducer (~60 s)

# m.py — two processes on one node:
#   MASTER_ADDR=127.0.0.1 MASTER_PORT=29500 RANK=0 ZE_AFFINITY_MASK=0 python m.py &
#   MASTER_ADDR=127.0.0.1 MASTER_PORT=29500 RANK=1 ZE_AFFINITY_MASK=1 python m.py &
import os, datetime, torch, torch.distributed as dist
rank = int(os.environ["RANK"])
torch.xpu.set_device(0)
dist.init_process_group("xccl", init_method="env://", world_size=2, rank=rank,
                        timeout=datetime.timedelta(seconds=60))
t = torch.zeros(8, device="xpu:0") + rank + 1
dist.all_reduce(t)   # hangs without SYS_PTRACE; completes with it
dist.destroy_process_group()

Container flags that reproduce: --device /dev/dri -v /dev/dri:/dev/dri:ro (with bind mount: silent hang), --device /dev/dri alone (fast opendir exception).

Observations (oneCCL as shipped in vllm/vllm-openai-xpu, level-zero 1.15.39122)

• Default container caps: CCL_WARN ... pidfd is not supported, fallbacks to drmfd exchange mode, then silence; all_reduce never completes (verified >15 min).
• --cap-add SYS_PTRACE: all_reduce completes correctly on both ranks.
• CCL_ZE_IPC_EXCHANGE=none: the "changed to be none" warning is printed, yet the drmfd fallback warning still appears and the hang persists.
• Container without /dev/dri bind mount: immediate ze_fd_manager.cpp:144 init_device_fds: opendir failed exception instead of a hang.

Related warnings observed right before the hang (may help locate the path):

CCL_WARN| comm_dev_uuids is not sub-vector of node_dev_uuids, comm_dev_uuids size 2, node_dev_uuids size 1, this may happen due to narrow device affinity mask
CCL_WARN| number of result device uuids does not match number of ranks per host, result size 1, host_rank_info_vec size 2, this may happen due to narrow device affinity mask

Requests

1. Honor CCL_ZE_IPC_EXCHANGE=none (skip the exchange entirely — hosts without P2P lose nothing), or
2. fail fast with a clear error when the drmfd exchange cannot complete, instead of hanging silently, and
3. document the SYS_PTRACE/seccomp requirement for the pidfd path in containers, and the /dev/dri (by-path) bind-mount requirement for the drmfd path.

Environment: 2 x Intel Arc Pro B70 (8086:e223), xe KMD, kernel 7.0.0, level-zero 1.15.39122, oneCCL bundled with the pinned vllm/vllm-openai-xpu image, docker containers.