User Guides on DRANET

Ray on GKE using DRANET

Mon, 14 Jul 2025 10:10:40 +0000

To get started, follow the instructions to create a GKE cluster with DRA support and using DRANET, it is important to follow the instructions, since there are multiple dependencies on the Kubernetes API version, the RDMA NCCL installer and the DRANET component.

The worker nodes in this configuration are a4-highgpu-8g instances, each equipped with eight NVIDIA B200 GPUs and eight RDMA-capable RoCE NICs.

Deploy RayCluster

Install Ray CRDs and the KubeRay operator:

GKE with NVIDIA DRA and DRANET

Fri, 20 Jun 2025 10:10:40 +0000

To get started, create a GKE cluster with DRA support and the corresponding VPC and subnets

It should look like

PROJECT="gke-dranet"
CLUSTER="dranet-dranet"
REGION="us-west8"
ZONE="us-west8-c"
GVNIC_NETWORK_PREFIX="dranet-gvnic"
RDMA_NETWORK_PREFIX="dranet-rdma"
VERSION="1.34"

gcloud container clusters create "${CLUSTER}" \
 --cluster-version="${VERSION}" \
 --enable-multi-networking \
 --enable-dataplane-v2 \
 --no-enable-autorepair \
 --no-enable-autoupgrade \
 --zone="${ZONE}" \
 --project="${PROJECT}"

# Create a VPC for the additional Google Titanium CPU NIC
gcloud compute --project=${PROJECT?} \
 networks create \
 ${GVNIC_NETWORK_PREFIX?}-net \
 --subnet-mode=custom

gcloud compute --project=${PROJECT?} \
 networks subnets create \
 ${GVNIC_NETWORK_PREFIX?}-sub \
 --network=${GVNIC_NETWORK_PREFIX?}-net \
 --region=${REGION?} \
 --range=192.168.0.0/24

gcloud compute --project=${PROJECT?} \
 firewall-rules create \
 ${GVNIC_NETWORK_PREFIX?}-internal \
 --network=${GVNIC_NETWORK_PREFIX?}-net \
 --action=ALLOW \
 --rules=tcp:0-65535,udp:0-65535,icmp \
 --source-ranges=192.168.0.0/16

# Create HPC VPC for the RDMA NICs with 8 subnets.
gcloudcompute --project=${PROJECT?} \
 networks create ${RDMA_NETWORK_PREFIX?}-net \
 --network-profile=${ZONE?}-vpc-roce \
 --subnet-mode=custom

# Create subnets for the HPC VPC.
for N in $(seq 0 7); do
 gcloud compute --project=${PROJECT?} \
 networks subnets create \
 ${RDMA_NETWORK_PREFIX?}-sub-$N \
 --network=${RDMA_NETWORK_PREFIX?}-net \
 --region=${REGION?} \
 --range=192.168.$((N+1)).0/24 & # offset to avoid overlap with gvnics
done

gcloud container node-pools create dranet-a4 \
 --cluster ${CLUSTER} \
 --project ${PROJECT} \
 --zone ${ZONE} \
 --node-locations ${ZONE} \
 --machine-type a4-highgpu-8g\
 --accelerator "type=nvidia-b200,count=8,gpu-driver-version=default" --num-nodes "2" \
 --additional-node-network network=${GVNIC_NETWORK_PREFIX}-net,subnetwork=${GVNIC_NETWORK_PREFIX}-sub \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-0 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-1 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-2 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-3 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-4 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-5 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-6 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-7

Apply the following DaemonSet to install the RDMA binaries and the NCCL library on the node. The RDMA binaries are stored in /home/kubernetes/bin/gib directory and the NCCL library is stored in /home/kubernetes/bin/nvidia/lib64 directory on the VM:

GKE and Cloud TPU v6e (Trillium)

Tue, 27 May 2025 11:30:40 +0000

If you use TPU Trillium and you want to improve the network performance of your Pods you can balance your network traffic over the VM NICs.

The ct6e-standard-4t machine type is backed by two physical NICs, since the main interface of the VM is used for all the applications and Pods on the host, you can create two additional vNICs on the VM that will be attached to each of the physical NICs, and pass them to the Pod directly, so you can multiplex your traffic to consume the total capacity of the physical NICs.

GKE and GPUDirect RDMA with DRA

Tue, 27 May 2025 11:30:40 +0000

On Google Cloud A3 Ultra and A4 machine types, you can utilize GPUDirect RDMA to run distributed AI workloads that require high performance networking support. To get started, create a GKE cluster with DRA support and the corresponding VPC and subnets for the RDMA network for the A3Ultra or A4 Node Pools, the gcloud commands should be something like:

PROJECT="gke-dranet"
CLUSTER="dranet-dranet"
REGION="us-west8"
ZONE="us-west8-c"
GVNIC_NETWORK_PREFIX="dranet-gvnic"
RDMA_NETWORK_PREFIX="dranet-rdma"
VERSION="1.34"

gcloud container clusters create "${CLUSTER}" \
 --cluster-version="${VERSION}" \
 --enable-multi-networking \
 --enable-dataplane-v2 \
 --no-enable-autorepair \
 --no-enable-autoupgrade \
 --zone="${ZONE}" \
 --project="${PROJECT}"

# Create a VPC for the additional Google Titanium CPU NIC
gcloud compute --project=${PROJECT?} \
 networks create \
 ${GVNIC_NETWORK_PREFIX?}-net \
 --subnet-mode=custom

gcloud compute --project=${PROJECT?} \
 networks subnets create \
 ${GVNIC_NETWORK_PREFIX?}-sub \
 --network=${GVNIC_NETWORK_PREFIX?}-net \
 --region=${REGION?} \
 --range=192.168.0.0/24

gcloud compute --project=${PROJECT?} \
 firewall-rules create \
 ${GVNIC_NETWORK_PREFIX?}-internal \
 --network=${GVNIC_NETWORK_PREFIX?}-net \
 --action=ALLOW \
 --rules=tcp:0-65535,udp:0-65535,icmp \
 --source-ranges=192.168.0.0/16

# Create HPC VPC for the RDMA NICs with 8 subnets.
gcloudcompute --project=${PROJECT?} \
 networks create ${RDMA_NETWORK_PREFIX?}-net \
 --network-profile=${ZONE?}-vpc-roce \
 --subnet-mode=custom

# Create subnets for the HPC VPC.
for N in $(seq 0 7); do
 gcloud compute --project=${PROJECT?} \
 networks subnets create \
 ${RDMA_NETWORK_PREFIX?}-sub-$N \
 --network=${RDMA_NETWORK_PREFIX?}-net \
 --region=${REGION?} \
 --range=192.168.$((N+1)).0/24 & # offset to avoid overlap with gvnics
done

gcloud container node-pools create dranet-a4 \
 --cluster ${CLUSTER} \
 --project ${PROJECT} \
 --zone ${ZONE} \
 --node-locations ${ZONE} \
 --machine-type a4-highgpu-8g\
 --accelerator "type=nvidia-b200,count=8,gpu-driver-version=default" --num-nodes "2" \
 --additional-node-network network=${GVNIC_NETWORK_PREFIX}-net,subnetwork=${GVNIC_NETWORK_PREFIX}-sub \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-0 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-1 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-2 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-3 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-4 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-5 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-6 \
 --additional-node-network network=${RDMA_NETWORK_PREFIX}-net,subnetwork=${RDMA_NETWORK_PREFIX}-sub-7

MPI Operator on GKE and GPUDirect RDMA

Tue, 27 May 2025 11:30:40 +0000

Running distributed applications, such as those using the Message Passing Interface (MPI) or NVIDIA’s Collective Communications Library (NCCL) for GPU communication, often requires each participating process (or Pod, in Kubernetes terms) to have access to high-speed, low-latency interconnects. Simply sharing a generic network interface among many high-performance jobs can lead to contention, unpredictable performance, and underutilization of expensive hardware.

The goal is resource compartmentalization: ensuring that each part of your distributed job gets dedicated access to the specific resources it needs – for instance, one GPU and one dedicated RDMA-capable NIC per worker.

Interface Configuration

Sun, 25 May 2025 11:30:40 +0000

To configure network interfaces in DRANET, users can provide custom configurations through the parameters field of a ResourceClaim or ResourceClaimTemplate. This configuration adheres to the NetworkConfig structure, which defines the desired state for network interfaces and their associated routes.

Network Configuration Overview

The primary structure for custom network configuration is NetworkConfig. It encompasses settings for the network interface itself and any specific routes and rules to be applied within the Pod’s network namespace.