InfiniBand™ networking for Compute virtual machines with GPUs

You can group your virtual machines with GPUs into a GPU cluster. The cluster accelerates high-performance computing (HPC) tasks such as training and inference. These tasks require a lot of processing power that a single VM cannot provide. The GPU clusters are built with InfiniBand secure high-speed networking. Each GPU in a VM is connected through a network interface card (NIC) that provides 400 Gbps. As a compute VM for GPU clusters consists of 8 GPUs, the total bandwidth for a node is 3.2 Tbps. Nebius AI Cloud uses GPUDirect RDMA, an NVIDIA technology of remote direct memory access (RDMA) that allows data to flow directly between each GPU and its NIC, avoiding CPU, thus boosting the data exchange speed.

InfiniBand fabrics

Each GPU cluster is created in one of physical InfiniBand fabrics. This is where GPUs interconnected over InfiniBand are located. Each fabric has limited GPU capacity. When creating a GPU cluster, select an InfiniBand fabric for it. Take into account the type of GPUs you are going to use. For example, if you select fabric-7, you can only add NVIDIA® H200 NVLink with Intel Sapphire Rapids GPUs to this cluster. Available fabrics and corresponding regions (private regions are marked with *):

Fabric	GPU platform	Region
`fabric-2`	NVIDIA® H100 NVLink with Intel Sapphire Rapids (`gpu-h100-sxm`)	`eu-north1`
`fabric-3`	NVIDIA® H100 NVLink with Intel Sapphire Rapids (`gpu-h100-sxm`)	`eu-north1`
`fabric-4`	NVIDIA® H100 NVLink with Intel Sapphire Rapids (`gpu-h100-sxm`)	`eu-north1`
`fabric-5`	NVIDIA® H200 NVLink with Intel Sapphire Rapids (`gpu-h200-sxm`)	`eu-west1`
`fabric-6`	NVIDIA® H100 NVLink with Intel Sapphire Rapids (`gpu-h100-sxm`)	`eu-north1`
`fabric-7`	NVIDIA® H200 NVLink with Intel Sapphire Rapids (`gpu-h200-sxm`)	`eu-north1`
`eu-north2-a`	NVIDIA® H200 NVLink with Intel Sapphire Rapids (`gpu-h200-sxm`)	`eu-north2`
`me-west1-a`	NVIDIA® B200 NVLink with Intel Emerald Rapids (`gpu-b200-sxm-a`)	`me-west1`
`uk-south1-a`	NVIDIA® B300 NVLink with Intel Granite Rapids (`gpu-b300-sxm`)	`uk-south1`
`us-central1-a`	NVIDIA® H200 NVLink with Intel Sapphire Rapids (`gpu-h200-sxm`)	`us-central1`
`us-central1-b`	NVIDIA® B200 NVLink with Intel Emerald Rapids (`gpu-b200-sxm`)	`us-central1`

In most cases, you do not need to change the preselected fabric. We recommend that you create a GPU cluster in another fabric only if it is better suited for a different platform or if you experience capacity issues with an existing GPU cluster.

How to enable InfiniBand for VMs with GPUs

Web console
CLI

Create a GPU cluster:
1. In the sidebar, go to Compute → GPU clusters.
2. Click Create GPU cluster.
3. On the page that opens, specify the cluster name. It should contain from 3 to 63 characters: lowercase letters, numbers and hyphens.
4. Select the InfiniBand fabric.
5. Click Create GPU cluster.
Add VMs to the cluster. You can do it only when creating the VMs:
All virtual machines added to the GPU cluster, including Managed Service for Kubernetes® nodes, must be in the same project.
1. In the sidebar, go to Compute → Virtual machines.
2. Click Create virtual machine.
3. On the page that opens, specify the VM’s details and select the GPU cluster name in the GPU cluster list.

You can also create a GPU cluster while creating the first VM in it:

In the Computing resources section of the VM creation form:

Select With GPU.

Select a platform and a preset compatible with GPU clusters. The compatible platforms and presets:

Platform	Presets	Regions
NVIDIA® B300 NVLink with Intel Granite Rapids (`gpu-b300-sxm`)	`8gpu-192vcpu-2768gb`	`uk-south1`
NVIDIA® B200 NVLink with Intel Emerald Rapids (`gpu-b200-sxm`)	`8gpu-160vcpu-1792gb`	`us-central1`
NVIDIA® B200 NVLink with Intel Emerald Rapids (`gpu-b200-sxm-a`)	`8gpu-160vcpu-1792gb`	`me-west1`
NVIDIA® H200 NVLink with Intel Sapphire Rapids (`gpu-h200-sxm`)	`8gpu-128vcpu-1600gb`	`eu-north1`, `eu-north2`, `eu-west1`, `us-central1`
NVIDIA® H100 NVLink with Intel Sapphire Rapids (`gpu-h100-sxm`)	`8gpu-128vcpu-1600gb`	`eu-north1`

In the Boot disk section of the VM creation form, select the boot disk for NVIDIA GPUs. For details, see Boot disk images for Compute virtual machines.

Check that your project ID is saved in the Nebius AI Cloud CLI profile configuration:
```
cat ~/.nebius/config.yaml
```
If you have not set your project ID as parent-id, or you want to create resources in a different project, update your CLI profile:
```
nebius profile update --parent-id <project_ID>
```
You can copy the project ID from the Project settings.
Depending on your project’s region, select an InfiniBand fabric for VM interconnection and save it to an environment variable:
```
export INFINIBAND_FABRIC=<fabric>
```

Create a GPU cluster and save its ID:

export GPU_CLUSTER_ID=$(nebius compute gpu-cluster create \
  --name <gpu_cluster_name> \
  --infiniband-fabric $INFINIBAND_FABRIC \
  --format json \
  | jq -r ".metadata.id")

Where:

name: A cluster name that you can use to quickly find the cluster.

Create a boot disk optimized for VMs with NVIDIA GPUs:

export BOOT_DISK_ID=$(nebius compute disk create \
  --name <boot_disk_name> \
  --size-gibibytes 200 \
  --type network_ssd \
  --source-image-family-image-family ubuntu22.04-cuda12 \
  --block-size-bytes 4096 \
  --format json \
  | jq -r ".metadata.id")

For compatible boot disk images (--source-image-family-image-family), see Boot disk images.

Create a virtual machine with GPUs and specify the GPU cluster ID in its parameters.

All virtual machines added to the GPU cluster, including Managed Service for Kubernetes nodes, must be in the same project.

Depending on how you are specifying the parameters in the nebius compute instance create command:

JSON Use .spec.gpu_cluster.id for the GPU cluster ID. Specify a VM platform with GPUs in .spec.resources.platform, and a preset compatible with GPU clusters in .spec.resources.preset. The compatible platforms and presets are:

Platform	Presets	Regions
NVIDIA® B300 NVLink with Intel Granite Rapids (`gpu-b300-sxm`)	`8gpu-192vcpu-2768gb`	`uk-south1`
NVIDIA® B200 NVLink with Intel Emerald Rapids (`gpu-b200-sxm`)	`8gpu-160vcpu-1792gb`	`us-central1`
NVIDIA® B200 NVLink with Intel Emerald Rapids (`gpu-b200-sxm-a`)	`8gpu-160vcpu-1792gb`	`me-west1`
NVIDIA® H200 NVLink with Intel Sapphire Rapids (`gpu-h200-sxm`)	`8gpu-128vcpu-1600gb`	`eu-north1`, `eu-north2`, `eu-west1`, `us-central1`
NVIDIA® H100 NVLink with Intel Sapphire Rapids (`gpu-h100-sxm`)	`8gpu-128vcpu-1600gb`	`eu-north1`

For example:

{
  "spec": {
    "resources": {
      "platform": "gpu-h100-sxm",
      "preset": "8gpu-128vcpu-1600gb"
    },
    "gpu_cluster": {
      "id": "$GPU_CLUSTER_ID"
    },
    "boot_disk": {
      "attach_mode": "READ_WRITE",
      "existing_disk": {
        "id": "$BOOT_DISK_ID"
      }
    },
    ...
  },
  ...
}

CLI parameters Use --gpu-cluster-id for the GPU cluster ID and --boot-disk-existing-disk-id for the boot disk ID. Specify a VM platform with GPUs in --resources-platform, and a preset compatible with GPU clusters in --resources-preset. The compatible platforms and presets are:

Platform	Presets	Regions
NVIDIA® B300 NVLink with Intel Granite Rapids (`gpu-b300-sxm`)	`8gpu-192vcpu-2768gb`	`uk-south1`
NVIDIA® B200 NVLink with Intel Emerald Rapids (`gpu-b200-sxm`)	`8gpu-160vcpu-1792gb`	`us-central1`
NVIDIA® B200 NVLink with Intel Emerald Rapids (`gpu-b200-sxm-a`)	`8gpu-160vcpu-1792gb`	`me-west1`
NVIDIA® H200 NVLink with Intel Sapphire Rapids (`gpu-h200-sxm`)	`8gpu-128vcpu-1600gb`	`eu-north1`, `eu-north2`, `eu-west1`, `us-central1`
NVIDIA® H100 NVLink with Intel Sapphire Rapids (`gpu-h100-sxm`)	`8gpu-128vcpu-1600gb`	`eu-north1`

For example:

nebius compute instance create \
  --resources-platform gpu-h100-sxm \
  --resources-preset 8gpu-128vcpu-1600gb \
  --gpu-cluster-id $GPU_CLUSTER_ID \
  --boot-disk-existing-disk-id $BOOT_DISK_ID \
  ...

How to test the connection with the NCCL tests

To test InfiniBand performance in a Compute cluster, you can run the NVIDIA NCCL test in it. For instructions, see our tutorial on running distributed jobs with MPIrun: it uses the NCCL test as an example.

Virtual machines

Slurm and Soperator in Nebius AI Cloud

Managed Service for Kubernetes®

InfiniBand™ networking for Compute virtual machines with GPUs

InfiniBand fabrics

How to enable InfiniBand for VMs with GPUs

How to test the connection with the NCCL tests

See also

Virtual machines

Slurm and Soperator in Nebius AI Cloud

Managed Service for Kubernetes®

​InfiniBand fabrics

​How to enable InfiniBand for VMs with GPUs

​How to test the connection with the NCCL tests

​See also

InfiniBand fabrics

How to enable InfiniBand for VMs with GPUs

How to test the connection with the NCCL tests

See also