Maintenance in Managed Service for Kubernetes®

The goal of the Managed Service for Kubernetes maintenance is to terminate a node as gracefully as possible. When a node termination is required, Nebius AI Cloud issues a maintenance event. Maintenance events are triggered when software or hardware fails on the physical machines that host your nodes, or when Nebius AI Cloud runs planned maintenance. Software and hardware failures account for the vast majority of maintenance events. Managed Kubernetes listens to maintenance events that underlying services launch. In particular, as every Kubernetes node represents a Compute virtual machine, Managed Kubernetes tracks Compute maintenance events.

How maintenance occurs

Nebius AI Cloud issues a maintenance event. When the event is issued, Managed Kubernetes assigns the NebiusMaintenanceScheduled Kubernetes condition. You can check the list of conditions to make sure that the service has issued the event.

The Managed Kubernetes service detects an event on a node. The service groups nodes into batches within a given node group. If a lot of maintenance events are expected in a Managed Kubernetes cluster, batches allow you to avoid stopping all nodes at once. The batch size equals either 1 or the value of the maximum unavailable deployment strategy parameter if that value is greater than 1. You can check the node group specifications:

CLI
Go SDK
Python SDK
JavaScript SDK

nebius mk8s node-group get --id <node_group_ID>

nodeGroup, err := sdk.Services().MK8S().V1().
    NodeGroup().Get(
        ctx,
        &mk8s.GetNodeGroupRequest{
            Id: "<node_group_ID>",
        },
    )
if err != nil {
    return err
}
fmt.Println(nodeGroup)

node_group_service = NodeGroupServiceClient(sdk)
node_group = await node_group_service.get(
    GetNodeGroupRequest(id="<node_group_ID>"),
)
print(node_group)

const nodeGroupService = new NodeGroupService(sdk);
const nodeGroup = await nodeGroupService.get(
  GetNodeGroupRequest.create({
    id: "<node_group_ID>",
  }),
);
console.log(nodeGroup);

To stop scheduling new Pods, Managed Kubernetes cordons the node.
The service waits for running GPU Pods and labeled CPU Pods to finish. If any of these Pods are still running one hour before the SLA deadline, the service stops waiting. The SLA deadline is the latest time the maintenance event should take place. You can check the SLA deadline together with Kubernetes conditions.
To remove existing Pods, Managed Kubernetes drains the node. The drain takes up to one hour.
Nebius AI Cloud stops the Compute VM (that is, the node).
Nebius AI Cloud starts the VM.
Managed Kubernetes uncordons the node and enables scheduling new Pods.
Managed Kubernetes removes the NebiusMaintenanceScheduled condition from the node.

After that, the node is considered to be healthy. Workloads can run on this node again.

Manual launch of maintenance

The service runs maintenance automatically. However, you can launch it manually as well if a maintenance event is issued for your node. For more information, see How to launch maintenance manually in Managed Kubernetes.

Postponing node drain

For GPU Pods, Managed Kubernetes waits until the Pods finish or until one hour before the SLA deadline, whichever happens first. Then, the service starts draining the node. For CPU Pods, Managed Kubernetes doesn’t wait for the Pods to finish. If you want Managed Kubernetes to wait for your CPU Pods, assign the nebius.com/defer-maintenance-node-drain: "true" Kubernetes label to the CPU Pods. A continuously running Pod, such as a Pod managed by a DaemonSet, can’t postpone the drain past the SLA deadline. For more information, see How to postpone node drain for CPU Pods.

Virtual machines

Slurm and Soperator in Nebius AI Cloud

Managed Service for Kubernetes®

Maintenance in Managed Service for Kubernetes®

How maintenance occurs

Manual launch of maintenance

Postponing node drain

​How maintenance occurs

​Manual launch of maintenance

​Postponing node drain

How maintenance occurs

Manual launch of maintenance

Postponing node drain