Skip to main content
This guide explains how to use Open MPI and MPIrun to run parallel jobs on Compute virtual machines (VMs) that have GPUs and are added to a GPU cluster. The guide uses the NCCL tests developed by NVIDIA® as an example of a job that you can run with MPIrun. You can also run these tests by using Slurm, or in a Managed Service for Kubernetes® cluster with a node group that has a GPU cluster attached.

​
Costs

Nebius AI Cloud only charges you for running the virtual machines that make up your GPU cluster. For more details, see the Compute pricing.

​
Prerequisites

  1. Create a GPU cluster if you don’t already have one.
  2. Create virtual machines and add them to the cluster.

​
Steps

​
Install Open MPI on each VM in the cluster

For each VM in the GPU cluster:
  1. Get the VM’s private IP address.
  2. Connect to the VM through SSH.
  3. Install the Open MPI library on the VM:

​
Build the tests on one of the VMs

Choose one of the VMs as the main VM – you will run the tests from it. Build the tests on the main VM:
  1. Clone the NVIDIA repository with the tests:
  2. Build the tests with Open MPI:
  3. Copy the built binary file, all_reduce_perf, to the same directory on other VMs.

​
Set up SSH connectivity between the VMs in the cluster

  1. Generate an SSH key pair. When generating the key pair, don’t configure a passphrase.
  2. Copy the generated pair, ~/.ssh/id_ed25519 and ~/.ssh/id_ed25519.pub, to the same directory on each other VM.
  3. On all other VMs, add the public key from the pair to the list of authorized keys:
For more details, see the Open MPI documentation.

​
Run the tests

Run the tests from the main VM with the mpirun command:
Where:
  • IP_address_[1-4]: IP address of the VM where you want to run the test.
  • :8: Amount of GPUs on the VM.
  • -mca pml ucx: Instruction for MPI communications to go through InfiniBand™ by using UCX. To use Ethernet instead, replace the parameter with -mca btl_tcp_if_include eth0. This doesn’t affect InfiniBand data exchanges of the test itself.
  • ~/nccl-tests/build/all_reduce_perf: A path to the binary file that should be available on all VMs.
In the result, check the average bus bandwith. If its value is higher than 300 GB/s, the connection is stable. Example:
The average bus bandwith is not equal to the InfiniBand™ one as some of the NCCL operations it measures use NVLink. Nevertheless, it accurately estimates the connection.

​
How to delete the created resources

The virtual machines that make up your GPU cluster are chargeable. If you don’t need the VMs, delete them, so Nebius AI Cloud doesn’t charge for them:
  1. In the sidebar, go to Compute → Virtual machines.
  2. On the Standalone VMs tab, next to the virtual machine’s name, click → Delete.
  3. Enter the VM name and confirm deletion.
InfiniBand and InfiniBand Trade Association are registered trademarks of the InfiniBand Trade Association.