RDMA in gVisor: A Deep Dive into GPU Networking

July 2026

It is helpful to think of the Operating System (OS) on a machine as having three distinct spaces:

1. The kernel space, or the OS's core. It manages hardware resources like RAM and CPU.
2. The user-space, which is the home of most user processes. These applications communicate with the OS via system calls (syscalls) to the kernel.
3. The device driver space, which contains interfaces for programs that run on hardware like GPUs and Network Interface Cards (NICs).

DMA (Direct Memory Access) allows devices to directly access RAM without copying data through the CPU. Reads and writes to DMA-able memory¹ bypass the kernel, reducing the latency of data-transfers between RAM and GPUs, NICs, etc.

RDMA (Remote Direct Memory Access) expands DMA to different machines on a network. One peer can read and write into another peer's memory without involving either peer's kernel. The bandwidth required for RDMA payloads can be much larger than the standard 10Gbps Ethernet, especially during large-scale model training².

Special networking hardware for RDMA exists that can send up to 800 Gbps of data over the wire. The popular transport types are InfiniBand, which is dominated by NVIDIA-acquired Mellanox, and RoCE (RDMA over Converged Ethernet), which implements InfiniBand semantics on upgraded ethernet hardware. To allow the user-space direct InfiniBand/RoCE access, RDMA consumers make requests to "uverbs" devices, which call into a software library called libibverbs.

RDMA architecture in gVisor — Figure 1: Applications communicate by creating work requests to the low-level hardware. Messages get sent over the network and acknowledged by the receiver host. Successful completion events are propagated back to the application. Pictured are terms like Work Queue Elements (WQE), Completion Queue Elements (CQE), and Queue Pairs (QP). Here is my Anki deck of RDMA jargon for the curious.

The gVisor Container Runtime

gVisor is an application kernel for containerized applications (see "What is a container?"). Using our three-space model, the container is a fourth space, and gVisor acts as an intermediary between the container space and the host space. For example, a containerized application calls memcpy() on a CUDA device. gVisor:

1. Intercepts the syscall via a seccomp filter.
2. Checks what CUDA device it's targeting based on the file descriptor (FD) of the call.
3. Issues the syscall if the container mounted the device and therefore owns it.

gVisor spaces — Figure 2: The "Sentry" runs as a separate process in the gVisor sandbox and traps syscalls from the containerized application to the host kernel.

The Sentry in gVisor is given the same virtual address space, filesystem access, network namespace, and mounted devices that the sandbox gets; only, the Sentry controls what the container can access according to gVisor's security model³.

Supporting GPUDirect RDMA in gVisor

NVProxy and TPUProxy are two popular features in gVisor that allow pass-through access to GPU and TPU devices respectively. Supporting RDMA with InfiniBand is a highly-requested feature, and the target implementation adds a "RDMAProxy" library interface similar to NVProxy and TPUProxy.

My goal is to match the bandwidth performance of RDMA to within <10% of runc, a popular container runtime lacking the security isolation of gVisor. In default operation, CUDA applications use GPUDirect RDMA, which allows for true P2P data transfer⁴ between GPUs.

Memory Registration in RDMA

After a gVisor container receives all information about the host's RDMA hardware, the application registers memory for the data transfer, whether from host or GPU RAM. An RDMA-enabled NIC receives a scatter-gather list containing the physically contiguous regions of memory. The "scatter" comes from how the OS page table works: virtual regions of memory may be scattered across many different physical pages. "Gather" refers to a NIC grouping these physical pages into one logical range for transmission or reception of data.

Scatter-gather list — Figure 3: The RDMA NICs map 1:1 to every GPU on the host. During memory registration, uverbs programs the NIC with a scatter-gather list of memory regions to be used in RDMA. The scatter-gather list contains virtual addresses and the corresponding length of physically contiguous memory starting at that address.

RDMA is designed to bypass the kernel in the data-path (hot-path) of a RDMA transfer. However, an application cannot simply map some memory, program it into the NIC, and start the RDMA transfer. What happens if the kernel realizes that these pages are no longer being referenced or written to and decides to evict the memory pages from RAM? To keep the kernel completely out of the hot path, most RDMA applications "pin" the virtual regions of memory to prevent the kernel from moving the underlying physical backing of memory critical in the RDMA hot-path.

For gVisor, the RDMA application process picks the virtual address regions it wants to pin, user-space libibverbs receives a ibv_reg_mr call, and gVisor forwards any ioctls to the kernel-mode RDMA driver. On GPU machines, the pinned memory pages are allocated for each GPU device by the CUDA driver. One problem for multi-gpu systems is that each GPU device has its own virtual address range, which may be the same across different devices. To delineate between devices, CUDA creates a separate application process for each GPU, and the NVIDIA kernel module resolves the right device based on the caller's process identifier (PID).

Yet, the sentry runs as one process, forwarding host-bound syscalls from many different processes inside a single sandbox. All calls to ibv_reg_mr, which eventually makes a NVIDIA kernel module ioctl, appear to come from the same process. When the NIC driver calls nvidia_p2p_get_page, the kernel module sees two CUDA contexts pinning the same range!

Address resolution in gVisor — Figure 4: The sentry translates application virtual addresses to an internal virtual address backed by host memory. An application issues a call to ibv_reg_mr, but the virtual address can overlap between separate CUDA processes. The sentry now maps process-specific addresses to a single host address, causing a fatal collision when ibv_reg_mr gets forwarded to the NVIDIA kernel module.

Using DMABUF to Bypass Memory Collisions

During development of gVisor RDMA support, I ran into the above problem where NCCL hangs due to virtual address collisions:

NCCL Couldn't register memory region with regattr. RC: -14, ERROR: Bad address

Figure 5: NCCL is the NVIDIA Collective Communications Library, commonly used in multi-node training.

Preventing address collisions would require transforming the Sentry from a single process architecture to a multi-process architecture, which is too significant of a rewrite. NVIDIA's main RDMA documentation mentioned only a single mechanism, the nvidia-peermem kernel module, for performing GPUDirect RDMA. However, I did a little digging into the NCCL environment variables for RDMA and noticed a second, hidden flavor of GPUDirect RDMA: DMABUF.

DMABUF leverages the existing dma-buf Linux subsystem for sharing DMA memory between processes. An "exporter" creates the DMABUF object containing a scatter-gather list, and an "importer" receives the file descriptor pointing to this object. We can create a DMABUF object via the CUDA user-space libcuda.so, receive the object's file handle upon a successful syscall to the NVIDIA kernel module nvidia.ko, and share the file handle with uverbs, the NIC's RDMA user-interface library. Most importantly, the DMABUF exporter in NVIDIA's kernel module does not depend on the PID of the calling process, and we can use sentry without rewriting any architecture.

DMABUF Implementation in gVisor

CUDA creates DMABUF objects via the NV_ESC_EXPORT_TO_DMABUF_FD ioctl. This syscall asks the dma_buf subsystem to create a new DMABUF object and exports the file descriptor for that object back to the caller. Whenever an application returns a file descriptor, gVisor creates a "guest" file descriptor number that maps to the host file descriptor via an internal table.

The internals of the NV_ESC_EXPORT_TO_DMABUF_FD syscall take two paths. On creation events, the application takes -1 for the FD field, which signals the function to create a new DMABUF object. However, the application may call the function multiple times afterwards and specify handles to physically contiguous memory it wants to attach to the DMABUF object. Thus, handling this ioctl requires rewriting the ioctl params to the real host file descriptor when FD >= 0 or keeping the value as is when FD = -1.

/* If fd >= 0, dma-buf already exists with this fd, so get dma-buf from fd. */
/* If fd == -1, dma-buf is not created yet, so create it and then store */
/* additional handles. */

if (params->fd == -1)
{
	status = nv_dma_buf_create(nv, params);
}
else if (params->fd >= 0)
{
	status = nv_dma_buf_reuse(nv, params);
}
else
{
	status = NV_ERR_INVALID_ARGUMENT;
}

Figure 7: This handler to NV_ESC_EXPORT_TO_DMABUF_FD invokes the ioctl on the raw FD. When the FD matches a table entry in gVisor, the FD is translated to the underlying host FD. A -1 value passes through unrecognized, which is correct behavior for creating a new DMABUF object. Inspecting the source code for this ioctl helped design its interface in gVisor.

The handler for this ioctl in gVisor checks if the FD field is -1 and passes through the ioctl to the kernel. Upon a successful response, the returned FD is wrapped in a guest FD specific to DMABUF objects. If the FD field is >= 0, the handler will unwrap the underlying host FD before passing through the ioctl.

Networking Challenges

When do two hosts connect to set up RDMA? What data gets exchanged during the bootstrap (handshake) step? How are network devices identified within a cluster of nodes?

To address other devices, libibverbs relies on a GID (Global Identifier), which is a 128-bit number like an IPv6 address. The rdma_cm (RDMA Connection Manager) uses these GIDs to perform the bootstrap step. This step exchanges queue pair (QP) numbers, which tell the NICs where to post RDMA work requests. Without rdma_cm, all bootstrap happens via TCP over standard ethernet devices. NVIDIA applications do not use the rdma_cm and choose the TCP path instead.

Between RoCE and regular InfiniBand, the GIDs are derived differently. On InfiniBand, the vendor burns a GUID (Globally Unique Identifier) into the NIC firmware, but the RoCE GIDs are derived from the MAC address of the ethernet device. GIDs for InfiniBand look like <subnet-prefix>:<mac-address> while the GIDs for RoCE are either the assigned IPv6 address or an IPv6 formatted IPv4 address like <80-zeros:ffff:<ipv4-address>. Per NIC on both transports, a GID table contains the mapping of ports on the NIC to GIDs.

The first networking problem for gVisor arises in the ibv_resolve_eth_dmac step after the bootstrap has completed and device IPs are exchanged. First, Host A checks its ARP cache to see if there's a corresponding MAC address for the NIC on Host B. If there is none, an ARP broadcast containing the IP address of the target NIC is sent out to all devices in the cluster. Host B responds with the MAC address of the NIC addressed by that IP address. However, a non-target NIC is the first to receive the ARP request and acknowledges it with its own MAC address. The RDMA data-path hangs because a message contains an IP address absent from that NIC's GID table and gets dropped silently.

ARP Flux in RDMA Bootstrap — Figure 8: The ARP Flux problem is when non-target NICs respond for ARP broadcasts destined for a target NIC. This error occurs during ibv_resolve_eth_dmac after the bootstrap step when the two hosts have exchanged QP numbers and IP addresses via TCP.

Why would a NIC respond to an ARP request if it's not destined for that NIC's IP? A strange network setting called arp_ignore can be set to false, and any interface will answer the ARP request if the target IP is configured somewhere on the host. This problem is bad for gVisor for two reasons:

1. Under gVisor's --network=host mode, arp_ignore defaults to 0 and uses the weak-host model described above.
2. Typically, containers are created in a new network namespace, and the RDMA network interfaces are later moved into that namespace. Since the GID table is namespace dependent, any cached entries from a prior ARP broadcast on the host with a stronger network model disappear.

Another problem arises due to moving RDMA interfaces between network namespaces. RoCE relies on the IPv4 addresses of these interfaces to populate the device's GID. A background service, such as the machine vendor's startup process, will assign static IPs to the interfaces upon boot. When the interfaces are moved out of the host namespace, the kernel clears the IP addresses. Nothing in the container assigns IP addresses to the interfaces, causing GIDs to be unpopulated and ibv_modify_qp to fail.

My solutions include strengthening gVisor's network model for --network=host mode under RDMA and manually assigning the IP addresses to the new interfaces in the container. The latter problem was particularly challenging to debug. As IP addresses disappear, moving them back into the host namespace fails to restore them, and your nodes are cooked!

Questions

1. How do RDMA and NCCL differ between GPU instance types (e.g. AWS EFA vs Mellanox, Blackwell vs Hopper, IPoIB vs RoCE)?

EFA resembles InfiniBand uverbs devices but uses Amazon's custom RDMA provider implementation. The whitepaper discusses preventing head-of-line delays by allowing messages to arrive out-of-order, which reduces congestion for large AWS datacenters.

Since the EFA interface is quite similar to InfiniBand, replicating the syscall shim was easy except for a few issues with the aws-ofi-nccl plugin. AWS only started allowing DMABUF two months before this post, and my image's NCCL version pre-dated the release that removed the feature gate. Next, the kernel version gVisor presents was gated by the plugin, but hard-coding a version >= 5.12 fixed the issue.

Regarding GPU architecture, Blackwell chips are shipped with 2x higher bandwidth networking cards than Hopper chips. NCCL support is quite fragile for B200s, but setting NVIDIA_CUMEM_ENABLE=0 seems to pop the bubbles.

Most cloud providers are using RoCE with their NVIDIA chips, and InfiniBand consumers set IPoIB inactive for their virtual machines.

2. What happens to containers that fail or get preempted? Would retries work with NCCL?

For gVisor, the sentry holds a handle to the uverbs devices, and the kernel will destroy any RDMA resources upon process death. However, if one sandbox dies while some still exist, gVisor will free that sandbox's memory and reallocate it to a different sandbox. The virtual pages are committed to gVisor's host memfd, physical pages were pinned by the earlier sandbox, and the NIC can continue to write directly into a new sandbox's memory.

CUDA will automatically deregister the NIC's memory region upon process death, but the same guarantee may not hold for host RAM <-> RAM RDMA.

3. What is the length of a RDMA transfer? How often does memory registration occur?

For a single all-reduce operation, the RDMA setup happens once upfront and NCCL can reuse this setup for multiple collective operations. Below, I am running a multi-node benchmark on an all-reduce operation transferring 4.0 GB of data over 50 trials.

CUDA trace — Figure 9: Perfetto trace of an all-reduce benchmark profiling gVisor syscalls, uverbs ioctls, and CUDA calls. RM_ALLOC is called in the first group of uverbs ioctls but not in the second group, hinting that all 50 trials executed during the ~250ms gap in uverbs/CUDA ioctls.

The span of setup + transfer + teardown takes about 2s in total. RDMA setup reduces end-to-end latency for large, steady workloads but not for small, bursty workloads.

Results

Below are benchmark results for the RDMA implementation on instance types from various cloud providers. All metrics match runc performance 1:1 within a couple of Gbps of variance.

Instance Type	Busbw	Algbw
Crusoe B200s⁵	`873.1 Gbps`	`465.7 Gbps`
Oracle B200s	`5536.8 Gbps`	`2953.0 Gbps`
GCP B200s	`4931.7 Gbps`	`2630.3 Gbps`
Whitefiber H200s	`3836.4 Gbps`	`2046.1 Gbps`

Here are my team's PRs for those interested:

1. Mounting RDMA files from sysfs in the container
2. Supporting DMABUF in nvproxy
3. Adding RDMAProxy interface
4. Creating ConnectX Mellanox plugin for RDMAProxy

I predict that RDMA network cards will continue to increase in bandwidth, and my hope is that hardware production expands across multiple manufacturers. Google is partnering with Intel to manufacture custom NICs for their TPUs. gVisor is already working on supporting Falcon hardware, which is very exciting!

I want to thank the team at Modal who collaborated with me on this project: Ayush Ranjan, the gVisor maintainer, and Peyton Walters, my internship mentor. Thank you Erik Dunteman, Ayush, and Marmik Chaudhari for providing helpful feedback on this post.