Summary
Hello uxlfoundation and Intel team,
ccl::alltoall triggers a Level Zero device loss when four ranks in one
process use ccl::create_communicatorExt with four independent one-device
Level Zero contexts. The failure occurs on the default oneCCL GPU all-to-all
path when peer access is available but peer atomics are not.
The attached reproducer has no XLA, PJRT, LLMD, MPI launcher, payload model,
or command-line options. It creates the four contexts, submits 256 fixed-size
BF16 all-to-alls per rank, waits once, and verifies the received data. On the
affected official package, every rank reports UR_RESULT_ERROR_DEVICE_LOST.
Version and environment
- oneCCL: official
intel-oneapi-ccl-2022.0 package version 2022.0.0-49301,
installed at /opt/intel/oneapi/ccl/2022.0.
- Corresponding legacy source revision used by the XLA integration:
4ceafd15c03ce46f11eeaf91781a92afebd3cecf.
- MPI: Intel MPI Library 2021.18.1, build 20260626
(intel-oneapi-mpi-2021.18 package version 2021.18.1-9). The reproducer
has four ranks in one process; it does not use an MPI launcher or network.
- Compiler: Intel oneAPI DPC++/C++ Compiler 2026.1.0
(2026.1.0.20260617; package intel-oneapi-compiler-dpcpp-cpp-2026.1
version 2026.1.0-235).
- OS: Ubuntu 24.04.4 LTS, kernel
7.0.0 and custom test kernel 7.1.5-xe-range with some xe driver patch related to the issue in compute-runtime.
- GPU driver/runtime: in-kernel Xe driver from that kernel. The Level Zero
Unified Runtime reports Arc Pro B70 driver/runtime 20.2.0 [1.15.38646+6].
No Agama package is installed.
- Hardware: AMD Ryzen Threadripper PRO 9955WX 16-Core CPU and four discrete
Intel Arc Pro B70 (Battlemage G31, PCI ID 8086:e223) GPUs.
- GPU interconnect: PCIe P2P. oneCCL reports
p2p_access_enabled=1 and
p2p_atomics_enabled=0 for this four-GPU node.
- Network: not applicable; this is a single-process, single-node test.
There is no known-good upstream revision. The local candidate workaround below
is not an upstream release or a regression baseline.
Affected projects
This blocks the four-GPU XLA oneAPI/PJRT integration used by LLMD inference.
XLA creates separate one-device contexts, which reaches the same create_communicatorExt configuration.
The attached standalone reproducer removes XLA, PJRT, LLMD, and model allocations so the problem can be diagnosed
within oneCCL and the Level Zero stack.
Reproducer
collective_alltoall.cpp — 102 lines; no command-line options.Makefile
The fixed workload is four ranks, 256 all-to-alls, and 262,144 BF16 elements per peer.
CCL_SYCL_COPY_ENGINE=0 explicitly retains the default GPU all-to-all path.
With CCL_SYCL_COPY_ENGINE=1 it hangs forever.
With the Intel oneAPI APT repository configured, install the exact packages
used for this report:
sudo apt install \
intel-oneapi-ccl-2022.0=2022.0.0-49301 \
intel-oneapi-ccl-devel-2022.0=2022.0.0-49301 \
intel-oneapi-mpi-2021.18=2021.18.1-9 \
intel-oneapi-mpi-devel-2021.18=2021.18.1-9 \
intel-oneapi-compiler-dpcpp-cpp-2026.1=2026.1.0-235
source /opt/intel/oneapi/setvars.sh
make clean
make ONECCL_ROOT=/opt/intel/oneapi/ccl/2022.0
LD_LIBRARY_PATH=/opt/intel/oneapi/ccl/2022.0/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} \
ONEAPI_DEVICE_SELECTOR=level_zero:0,1,2,3 \
CCL_LOG_LEVEL=debug \
CCL_SYCL_COPY_ENGINE=0 \
timeout -k 20s 120s ./collective_alltoallThe expected failing exit status is 1, with all four ranks reporting
UR_RESULT_ERROR_DEVICE_LOST. The process should be run on an isolated test
machine because the failure can reset all participating GPU engines.
Logs
Relevant excerpt:
CCL_WARN| value of CCL_LOG_LEVEL changed to be trace (default:warn)
MT topo_manager: p2p matrix built, p2p_access_enabled=1
p2p_atomics_enabled=0, number of node devices: 4
rank 0: exception: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
rank 1: exception: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
rank 2: exception: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
rank 3: exception: level_zero backend failed with error: 20 (UR_RESULT_ERROR_DEVICE_LOST)
Expected behavior
All four ranks should complete the 256 all-to-alls, verify their received data, and exit with status 0.
A oneCCL collective must not reset a GPU or cause a
later queue operation to return UR_RESULT_ERROR_DEVICE_LOST merely because
its ranks have separate one-device Level Zero contexts.
Observed behavior
oneCCL 2022.0 package exits with status 1.
After all-to-all work has been submitted, the queue wait reports UR_RESULT_ERROR_DEVICE_LOST for every rank.
In the trace I can see that oneCCL detects peer access but no peer atomics.
Existing workarounds
A candidate oneCCL-only host-barrier fallback has been tested locally:
For the same fixed reproducer, that build logs the in-process caller-thread
barrier selection and all four ranks print verified.
It is a correctness-first fallback for the no-P2P-atomics multi-threaded node communicator.
It adds host synchronization and is offered for maintainer review, not as a claim of a final IPC-mapping design.
cd /home/kevin/documents/issue-collective-alltoall
source /opt/intel/oneapi/setvars.sh
make clean
make ONECCL_ROOT=/home/kevin/oneCCL/build-kevin-host-barrier/_install
LD_LIBRARY_PATH=/home/kevin/oneCCL/build-kevin-host-barrier/_install/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH} \
ONEAPI_DEVICE_SELECTOR=level_zero:0,1,2,3 \
CCL_LOG_LEVEL=debug \
CCL_SYCL_COPY_ENGINE=0 \
./collective_alltoall > oneCCL-host-barrier.log 2>&1rank 0: verified
rank 1: verified
rank 2: verified
rank 3: verified
CCL_SYCL_COPY_ENGINE=0 and CCL_SYCL_COPY_ENGINE=1 both complete successfully with the host-barrier fallback.
Are these changes in my forked oneCCL branch correct enough to ask for a pull request to the upstream oneCCL repo?
Hello @alexr-hab i tag you because i updated the host-barrier in the invoke_barrier() you created.
Logs:
Reproducer:
Summary
Hello uxlfoundation and Intel team,
ccl::alltoalltriggers a Level Zero device loss when four ranks in oneprocess use
ccl::create_communicatorExtwith four independent one-deviceLevel Zero contexts. The failure occurs on the default oneCCL GPU all-to-all
path when peer access is available but peer atomics are not.
The attached reproducer has no XLA, PJRT, LLMD, MPI launcher, payload model,
or command-line options. It creates the four contexts, submits 256 fixed-size
BF16 all-to-alls per rank, waits once, and verifies the received data. On the
affected official package, every rank reports
UR_RESULT_ERROR_DEVICE_LOST.Version and environment
intel-oneapi-ccl-2022.0package version2022.0.0-49301,installed at
/opt/intel/oneapi/ccl/2022.0.4ceafd15c03ce46f11eeaf91781a92afebd3cecf.(
intel-oneapi-mpi-2021.18package version2021.18.1-9). The reproducerhas four ranks in one process; it does not use an MPI launcher or network.
(
2026.1.0.20260617; packageintel-oneapi-compiler-dpcpp-cpp-2026.1version
2026.1.0-235).7.0.0and custom test kernel7.1.5-xe-rangewith some xe driver patch related to the issue in compute-runtime.Unified Runtime reports Arc Pro B70 driver/runtime
20.2.0 [1.15.38646+6].No Agama package is installed.
Intel Arc Pro B70 (Battlemage G31, PCI ID
8086:e223) GPUs.p2p_access_enabled=1andp2p_atomics_enabled=0for this four-GPU node.There is no known-good upstream revision. The local candidate workaround below
is not an upstream release or a regression baseline.
Affected projects
This blocks the four-GPU XLA oneAPI/PJRT integration used by LLMD inference.
XLA creates separate one-device contexts, which reaches the same
create_communicatorExtconfiguration.The attached standalone reproducer removes XLA, PJRT, LLMD, and model allocations so the problem can be diagnosed
within oneCCL and the Level Zero stack.
Reproducer
collective_alltoall.cpp— 102 lines; no command-line options.MakefileThe fixed workload is four ranks, 256 all-to-alls, and 262,144 BF16 elements per peer.
CCL_SYCL_COPY_ENGINE=0explicitly retains the default GPU all-to-all path.With
CCL_SYCL_COPY_ENGINE=1it hangs forever.With the Intel oneAPI APT repository configured, install the exact packages
used for this report:
The expected failing exit status is
1, with all four ranks reportingUR_RESULT_ERROR_DEVICE_LOST. The process should be run on an isolated testmachine because the failure can reset all participating GPU engines.
Logs
Relevant excerpt:
Expected behavior
All four ranks should complete the 256 all-to-alls, verify their received data, and exit with status 0.
A oneCCL collective must not reset a GPU or cause a
later queue operation to return
UR_RESULT_ERROR_DEVICE_LOSTmerely becauseits ranks have separate one-device Level Zero contexts.
Observed behavior
oneCCL 2022.0 package exits with status 1.
After all-to-all work has been submitted, the queue wait reports
UR_RESULT_ERROR_DEVICE_LOSTfor every rank.In the trace I can see that oneCCL detects peer access but no peer atomics.
Existing workarounds
A candidate oneCCL-only host-barrier fallback has been tested locally:
neudinger/oneCCL:kevin/host-barrier-multi-threadedd63aa8ad25dacb5f443a6632fdec2802a329e2fbFor the same fixed reproducer, that build logs the in-process caller-thread
barrier selection and all four ranks print
verified.It is a correctness-first fallback for the no-P2P-atomics multi-threaded node communicator.
It adds host synchronization and is offered for maintainer review, not as a claim of a final IPC-mapping design.
CCL_SYCL_COPY_ENGINE=0andCCL_SYCL_COPY_ENGINE=1both complete successfully with the host-barrier fallback.Are these changes in my forked oneCCL branch correct enough to ask for a pull request to the upstream oneCCL repo?
Hello @alexr-hab i tag you because i updated the host-barrier in the
invoke_barrier()you created.Logs:
Reproducer: