Moving workload from the existing node group

The workload is moved automatically after you evict it from the existing node group. Before moving your workload, you need to:

Install jq if you don’t have it on your system:
sudo apt-get install jq
brew install jq
Get your cluster ID which is returned in the .metadata.id field of the cluster resource.

The examples in this guide assume that the cluster ID is saved to the K8S_CLUSTER_ID environment variable for the CLI or to the k8sClusterID, k8s_cluster_id or k8sClusterId variable for the Go, Python or JavaScript SDK. To move your workload:

Create a node group.

For example, the following command creates a group of two nodes, each with one NVIDIA H100 GPU, and all drivers and components required for the GPU:

nebius mk8s node-group create \
  --parent-id $K8S_CLUSTER_ID \
  --name mk8s-node-group-test \
  --fixed-node-count 2 \
  --template-resources-platform gpu-h100-sxm \
  --template-resources-preset 1gpu-16vcpu-200gb \
  --template-gpu-settings-drivers-preset cuda12

Save the ID of the node group from which you want to move the workload. For example, you can get it by the node group’s name (if you have set and know it) and the parent cluster’s ID:

CLI
Go SDK
Python SDK
JavaScript SDK

export K8S_NODE_GROUP_ID=$(nebius mk8s node-group get-by-name \
  --parent-id $K8S_CLUSTER_ID \
  --name node-group-name \
  --format jsonpath='{.metadata.id}')

lookupService := sdk.Services().MK8S().V1().NodeGroup()
oldNodeGroup, err := lookupService.GetByName(
    ctx,
    &common.GetByNameRequest{
        ParentId: k8sClusterID,
        Name:     oldNodeGroupName,
    },
)
if err != nil {
    return err
}
oldNodeGroupID := oldNodeGroup.GetMetadata().GetId()
if oldNodeGroupID == "" {
    return errors.New("node group ID is missing")
}

Assign the node group name to the oldNodeGroupName variable before you run the code.

lookup_service = NodeGroupServiceClient(sdk)
old_node_group = await lookup_service.get_by_name(
    GetByNameRequest(
        parent_id=k8s_cluster_id,
        name=old_node_group_name,
    ),
)
old_node_group_id = old_node_group.metadata.id
if not old_node_group_id:
    raise ValueError("node group ID is missing")

Assign the node group name to the old_node_group_name variable before you run the code.

const lookupService = new NodeGroupService(sdk);
const oldNodeGroup = await lookupService.getByName(
  GetByNameRequest.create({
    parentId: k8sClusterId,
    name: oldNodeGroupName,
  }),
);
const oldNodeGroupId = oldNodeGroup.metadata?.id;
if (!oldNodeGroupId) {
  throw new Error("node group ID is missing");
}

Assign the node group name to the oldNodeGroupName variable before you run the code.

Get the list of node names in the node group from which you want to move the workload:

export OLD_NODES=$(kubectl get nodes -o json \
  | jq '.items[].metadata
    | select(.annotations."cluster.x-k8s.io/owner-name" = "$K8S_NODE_GROUP_ID")
    | .name')

Cordon off the old nodes so that no new Pods are scheduled on them:

for node in $OLD_NODES; do
  kubectl cordon $node;
done

Drain the old nodes so that the existing Pods can be evicted from them:
for node in $OLD_NODES; do kubectl drain --force --ignore-daemonsets --delete-emptydir-data $node; done
Kubernetes will automatically move evicted Pods to suitable nodes.

Delete the old node group:

CLI
Go SDK
Python SDK
JavaScript SDK

nebius mk8s node-group delete --id $K8S_NODE_GROUP_ID

deleteService := sdk.Services().MK8S().V1().NodeGroup()
deleteOperation, err := deleteService.Delete(
    ctx,
    &mk8s.DeleteNodeGroupRequest{Id: oldNodeGroupID},
)
if err != nil {
    return err
}
if _, err = deleteOperation.Wait(ctx); err != nil {
    return err
}

delete_service = NodeGroupServiceClient(sdk)
delete_operation = await delete_service.delete(
    DeleteNodeGroupRequest(id=old_node_group_id),
)
await delete_operation.wait()

const deleteService = new NodeGroupService(sdk);
const deleteOperation = await deleteService.delete(
  DeleteNodeGroupRequest.create({
    id: oldNodeGroupId,
  }),
).result;
await deleteOperation.wait();

Virtual machines

Slurm and Soperator in Nebius AI Cloud

Managed Service for Kubernetes®

Moving workload from the existing node group