Automatic job restarts

By default, a job runs once and isn’t restarted. You can set up a job to be automatically restarted when its container exits with an error.

Automatic restarts aren’t available for Serverless AI endpoints.

Prerequisites

CLI
Go SDK
Python SDK
JavaScript SDK
REST API

Install and configure the Nebius AI Cloud CLI. Check that your project ID is saved in the Nebius AI Cloud CLI profile configuration:
cat ~/.nebius/config.yaml
Make sure you are in a group that has at least the editor role within your tenant or project; for example, the default editors group. You can check this in the Administration → IAM section of the web console.

Install and initialize the Nebius SDK for Go.
Make sure you are in a group that has at least the editor role within your tenant or project; for example, the default editors group. You can check this in the Administration → IAM section of the web console.

Install and initialize the Nebius SDK for Python.
Make sure you are in a group that has at least the editor role within your tenant or project; for example, the default editors group. You can check this in the Administration → IAM section of the web console.

Install and initialize the Nebius SDK for JavaScript.
Make sure you are in a group that has at least the editor role within your tenant or project; for example, the default editors group. You can check this in the Administration → IAM section of the web console.

Get an access token to authenticate to the REST API.
Make sure you are in a group that has at least the editor role within your tenant or project; for example, the default editors group. You can check this in the Administration → IAM section of the web console.

Configuring automatic restarts

To turn automatic restarts on or off, set the restart policy when you create the job. In the CLI, use the --restart-policy parameter. In the SDKs and the REST API, use the restart attempts parameter: Spec.RestartAttempts in Go, spec.restart_attempts in Python and spec.restartAttempts in JavaScript and the REST API. The values correspond as follows:

CLI value	SDK and REST API value	Behavior
`never`	`0` or omitted	Default. The job runs once. If the container exits with an error, the job is marked as failed and isn’t restarted.
`on-failure`	`-1`	If the container exits with a non-zero exit code, Serverless AI automatically restarts it.

Automatic restarts occur when your container exits with an error while the job is running. They are useful for workloads that can fail transiently and succeed on a retry, for example, a training workload that can resume from a checkpoint after a crash. For example, to create a job whose container restarts on failure:

CLI
Go SDK
Python SDK
JavaScript SDK
REST API

nebius ai job create \
  --name my-job \
  ... \
  --restart-policy on-failure

Instead of ..., specify the other job parameters, as described in How to create a job.

restartJobService := sdk.Services().Ai().V1().Job()
restartJobOperation, err := restartJobService.Create(
    ctx,
    &ai.CreateJobRequest{
        Metadata: &common.ResourceMetadata{
            ParentId: "<project_ID>",
            Name:     "my-job",
        },
        Spec: &ai.JobSpec{
            // ...
            // -1 restarts the container on failure without a limit.
            RestartAttempts: -1,
        },
    },
)
if err != nil {
    return fmt.Errorf("create job with restart policy: %w", err)
}
if _, err = restartJobOperation.Wait(ctx); err != nil {
    return fmt.Errorf("wait for job with restart policy: %w", err)
}

Setting Spec.RestartAttempts to -1 turns automatic restarts on. Instead of // ..., specify the other job parameters, as described in How to create a job.

restart_job_service = JobServiceClient(sdk)
restart_job_operation = await restart_job_service.create(
    CreateJobRequest(
        metadata=ResourceMetadata(
            parent_id="<project_ID>",
            name="my-job",
        ),
        spec=JobSpec(
            # ...
            # -1 restarts the container on failure without a limit.
            restart_attempts=-1,
        ),
    ),
)
await restart_job_operation.wait()

Setting spec.restart_attempts to -1 turns automatic restarts on. Instead of # ..., specify the other job parameters, as described in How to create a job.

dayjs.extend(duration);
const restartJobService = new JobService(sdk);
const restartJobOperation = await restartJobService.create(
  CreateJobRequest.create({
    metadata: {
      parentId: "<project_ID>",
      name: "my-job",
    },
    spec: {
      // ...
      // -1 restarts the container on failure without a limit.
      restartAttempts: -1n,
    },
  }),
).result;
await restartJobOperation.wait();

Setting spec.restartAttempts to -1n turns automatic restarts on. Instead of // ..., specify the other job parameters, as described in How to create a job.

curl --request POST \
  --url 'https://api.nebius.cloud/ai/v1/jobs' \
  --header "Authorization: Bearer <access_token>" \
  --header 'Content-Type: application/json' \
  --data '{
    "metadata": {
      "parentId": "<project_ID>",
      "name": "my-job"
    },
    "spec": {
      "image": "nvidia/cuda:13.1.1-runtime-ubuntu24.04",
      "containerCommand": "bash",
      "args": "-c nvidia-smi",
      "platform": "gpu-l40s-a",
      "preset": "1gpu-8vcpu-32gb",
      "timeout": "3600s",
      "restartAttempts": "-1",
      "subnetId": "<subnet_ID>",
      "shmSizeBytes": "17179869184",
      "disk": {
        "type": "NETWORK_SSD",
        "sizeBytes": "268435456000"
      }
    }
  }'

In the Authorization header, replace <access_token> with the access token that you got in the prerequisites.Setting spec.restartAttempts to "-1" turns automatic restarts on. For the other parameters, see How to create a job.

When jobs aren’t restarted

The restart policy responds only to the container’s own exit code. It doesn’t recover from:

No available capacity at start: if the requested compute can’t be allocated when the job starts.
Preemption: if you run on preemptible compute and Nebius AI Cloud reclaims (preempts) the underlying virtual machine (VM).
The underlying VM stopping unexpectedly.
Startup failures and timeouts.

In these cases, the job moves to the ERROR status and isn’t restarted automatically. To run it again, create a new job or restart it, for example with nebius ai job restart in the CLI.

For resilience against preemption or lost capacity, design your workload to be resumable, for example, write and resume from checkpoints, and resubmit the job rather than relying on automatic restarts.

AI services in Nebius AI Cloud

Serverless AI

Managed Service for MLflow

Applications in Nebius AI Cloud

Tutorials

Third-party integrations

Prerequisites

Configuring automatic restarts

When jobs aren’t restarted

​Prerequisites

​Configuring automatic restarts

​When jobs aren’t restarted

Prerequisites

Configuring automatic restarts

When jobs aren’t restarted