Skip to main content
If a job or an endpoint fails, gets canceled or returns an error, Serverless AI deletes the underlying virtual machine (VM) and boot disk to avoid costs. In this case, you can’t connect to the job or endpoint and debug it. To keep the VM alive for debugging, run the job or endpoint with a “sleep-on-fail” wrapper.

​
Keep the VM alive on failure

To keep the VM alive for debugging when a job or endpoint fails, use bash -lc and append a long sleep if the main command fails. Add the following parameters when creating the job or endpoint: 
nebius ai <job|endpoint> create \
  ... \
  --container-command bash \
  --args "-lc '<your_main_command> || (echo FAILED; sleep 86400)'"
Replace <your_main_command> with the actual command you want to run. When you use this wrapper, the job or endpoint behaves as follows:
  • If the main command succeeds, the job or endpoint exits normally.
  • If the main command fails, the job or endpoint keeps running for 24 hours. During this time, you can connect to the underlying VM via SSH and debug.
Use sleep 3600 for one hour if you want a shorter debug window.

​
Connect to the container by using SSH

You can connect to the container of the job or endpoint in the following ways:
To connect to the container by using SSH, you should create a job or create an endpoint with at least one --ssh-key.To run a shell inside the container for debugging, run nebius ai job ssh <job_ID> or nebius ai endpoint ssh <endpoint_ID>. A shell starts in the container.You can add the following parameters to these commands:
  • -i or --identity-file: Identity file for SSH authentication. Default: your SSH key.
  • -s or --shell: Shell to run inside the container. Default: sh.
For example, to use bash and a specific key, run:
  • For the job: 
    nebius ai job ssh <job_ID> -i ~/.ssh/id_rsa -s bash
  • For the endpoint: 
    nebius ai endpoint ssh <endpoint_ID> -i ~/.ssh/id_rsa -s bash
After you run the command, a shell inside the container is automatically opened.

​
Run tests inside the container

After you connected to the container:
  • Check currently running processes: 
    ps aux
    You can identify which processes used the most computing resources, check if the script was running as expected and see when the processes started.
  • Check if GPUs are accessible: 
    nvidia-smi
    This test helps confirm whether the GPUs were allocated correctly and whether the job or endpoint workloads could use them.