Skip to main content
If a job, endpoint or Devlab created from a custom image fails, gets canceled or returns an error, Serverless AI deletes the underlying virtual machine (VM) and boot disk to avoid costs. In this case, you can’t connect to the deployment and debug it. To keep the VM alive for debugging, run the deployment with a “sleep-on-fail” wrapper.
For Devlabs created from a template (--template), you can’t override the container command or arguments. You cannot add the “sleep-on-fail” wrapper in that mode. Use this approach only for Devlabs created from a custom image (--image).

​
Keep the VM alive on failure

To keep the VM alive for debugging when a job, endpoint or Devlab created from a custom image fails, use bash -lc and append a long sleep if the main command fails. Add the following parameters when creating the resource:
Replace <your_main_command> with the actual command you want to run. When you use this wrapper, the deployment behaves as follows:
  • If the main command succeeds, the resource exits normally.
  • If the main command fails, the resource keeps running for 24 hours. During this time, you can connect to the underlying VM via SSH and debug.
Use sleep 3600 for one hour if you want a shorter debug window.

​
Connect to the container by using SSH

You can connect to the container of a deployment in the following ways:
To connect to the container by using SSH, you should create a Devlab, create a job or create an endpoint with at least one --ssh-key.To run a shell inside the container for debugging, run nebius ai devlab ssh <devlab_ID>, nebius ai job ssh <job_ID> or nebius ai endpoint ssh <endpoint_ID>. A shell starts in the container.You can add the following parameters to these commands:
  • -i or --identity-file: Identity file for SSH authentication. Default: your SSH key.
  • -s or --shell: Shell to run inside the container. Default: sh.
For example, to use bash and a specific key, run:
  • For the Devlab:
  • For the job:
  • For the endpoint:
After you run the command, a shell inside the container is automatically opened.

​
Run tests inside the container

After you connected to the container:
  • Check currently running processes:
    You can identify which processes used the most computing resources, check if the script was running as expected and see when the processes started.
  • Check if GPUs are accessible:
    This test helps confirm whether the GPUs were allocated correctly and whether the Devlab, job or endpoint workloads could use them.