Deploying an LLM on a virtual machine with Nebius Tunnels

You can deploy and run a large language model (LLM) on a GPU virtual machine (VM) and then chat with this model via a secure tunnel by Nebius Tunnels. The VM goes without a public IP address, but by using the tunnel, anyone from the internet can access the LLM and connect to the VM. Meanwhile, the VM stays isolated. To establish the tunnel, the VM configuration includes a tunnel agent. The agent runs on the VM automatically after the VM starts. When you send a message to the model, the tunnel routes the traffic to the agent. Thus, only the IP address of the tunnel is exposed to the internet.

Use one VM per tunnel. Do not add several VMs to a single tunnel.

Costs

Nebius AI Cloud charges you for the following billing items:

Compute virtual machine
Boot disk attached to the VM

Nebius Tunnels is in preview and provided free of charge.

Steps

Create a tunnel

Web console
CLI

In the web console, select the project where you want to create the infrastructure. All resources must be located in the same project.
Go to Networking → Tunnels.
Click Create tunnel.
On the page that opens, set the tunnel name to my-tunnel.
Click Create tunnel.

Install and configure the Nebius AI Cloud CLI. When configuring the CLI profile, set the project where you want to create the infrastructure. All resources must be located in the same project. All nebius commands by default use the project ID from your CLI profile configuration.

Create a tunnel:

nebius tunnel create --title my-tunnel

Prepare a service account

To authorize the VM in the tunnel, prepare a service account with the applicationtunnel.agent role. Later, you attach this service account to the VM.

Create the service account:
- Web console
- CLI
1. In the sidebar, go to Administration → IAM.
2. Click Create resource → Service account.
3. In the window that opens, set the name to tunnel-agent-sa and select the project.
4. Click Create and continue.
nebius iam service-account create --name tunnel-agent-sa
Create a custom group in Identity and Access Management to assign the role to the service account:
- Web console
- CLI
1. In the sidebar, go to Administration → IAM.
2. Click Create resource → Group.
3. In the window that opens, set the group name to tunnel-agents. In the Scope field, select the project where you create all the resources.
4. Click Create.
nebius iam group create --name tunnel-agents
Attach the tunnel and grant the applicationtunnel.agent role to this group. You can only grant this role by using the CLI.
- CLI
1. Get the group ID:
  nebius iam group list
  The ID is specified in the items.metadata.id parameter in the output.
2. Get the tunnel ID:
  nebius tunnel list
3. Create an access permit in the custom group:
  nebius iam access-permit create \ --parent-id <group_ID> \ --resource-id <tunnel_ID> \ --role applicationtunnel.agent
Add the service account to the group:
- Web console
- CLI
1. Go to Administration → IAM.
2. On the Groups tab, find and open the tunnel-agents group.
3. Click Add members.
4. In the window that opens, add the tunnel-agent-sa service account.
5. Once the account is added, close the window.
Check the group page in general. Make sure that:
- The tunnel-agent-sa service account is specified on the Members tab.
- The tunnel and the applicationtunnel.agent role are specified on the Access permits tab.
nebius iam group-membership create \ --parent-id <group_ID> \ --member-id <service_account_ID>
To get the service account ID, run:
nebius iam service-account list

Create a virtual machine

Prepare the VM configuration that includes the Qwen/Qwen3-8B model and the tunnel agent. Then, create the VM — the model and agent are deployed on the VM automatically.

If you need to change the VM configuration later, delete the VM and create a new one. A tunnel should be associated with only one VM, and you can’t change the cloud-init configuration of an existing VM.

Generate an SSH key pair.

Prepare the cloud-init configuration file. You’ll attach it to the VM. In this file, specify two parameters:

users.ssh_authorized_keys: Public SSH key.
write_files.content.tunnel_id: ID of the tunnel.

llm.yaml

users:
  - name: ubuntu
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    shell: /bin/bash
    ssh_authorized_keys:
      - <public_SSH_key>

write_files:
  - path: /etc/nebius-tunnel-agent/config.yaml
    permissions: "0640"
    content: |
      tunnel_id: <tunnel_ID>

      iam:
        metadata:
          enabled: true

      logging:
        level: info
        format: json

      services:
        - name: llm
          type: http
          target: 127.0.0.1:8000
        - name: ssh
          type: tcp
          target: 127.0.0.1:22

  - path: /etc/systemd/system/nebius-tunnel-agent.service
    permissions: "0644"
    content: |
      [Unit]
      Description=Nebius tunnel agent
      Documentation=https://docs.nebius.com/tunnels
      After=network-online.target
      Wants=network-online.target

      [Service]
      Type=simple
      User=tunna
      Group=tunna
      ExecStart=/usr/local/bin/nebius-tunnel-agent --config /etc/nebius-tunnel-agent/config.yaml
      Restart=always
      RestartSec=5
      NoNewPrivileges=true
      ProtectSystem=strict
      ProtectHome=true
      PrivateTmp=true
      ReadOnlyPaths=/etc/nebius-tunnel-agent

      [Install]
      WantedBy=multi-user.target

  - path: /usr/local/sbin/install-nebius-tunnel-agent
    permissions: "0755"
    content: |
      #!/bin/bash
      set -euo pipefail

      VERSION="${AGENT_VERSION:-latest}"
      BASE="https://storage.eu-north1.nebius.cloud/products/releases/nebius-tunnel-agent"
      CONF_DIR=/etc/nebius-tunnel-agent

      case "$(uname -m)" in
        x86_64)        PLATFORM=linux-x86_64 ;;
        aarch64|arm64) PLATFORM=linux-aarch64 ;;
        *) echo "unsupported architecture: $(uname -m)" >&2; exit 1 ;;
      esac
      ASSET="nebius-tunnel-agent-${PLATFORM}.tar.gz"

      TMP="$(mktemp -d)"
      trap 'rm -rf "$TMP"' EXIT

      fetch() {
        curl -fsSL --retry 5 --retry-delay 3 --retry-connrefused -o "$2" "$1"
      }
      fetch "$BASE/$VERSION/$ASSET" "$TMP/$ASSET"
      fetch "$BASE/$VERSION/checksums-sha256.txt" "$TMP/checksums-sha256.txt"

      (cd "$TMP" && sha256sum --ignore-missing -c checksums-sha256.txt)

      tar -xzf "$TMP/$ASSET" -C "$TMP"
      install -m0755 "$TMP/nebius-tunnel-agent" /usr/local/bin/nebius-tunnel-agent

      getent group tunna >/dev/null || groupadd --system tunna
      id -u tunna >/dev/null 2>&1 || useradd --system --gid tunna \
        --no-create-home --shell /usr/sbin/nologin tunna
      chown -R tunna:tunna "$CONF_DIR"
      chmod 0750 "$CONF_DIR"
      chmod 0640 "$CONF_DIR/config.yaml"

      systemctl daemon-reload
      systemctl enable nebius-tunnel-agent.service
      systemctl restart nebius-tunnel-agent.service

  - path: /etc/systemd/system/vllm.service
    permissions: "0644"
    content: |
      [Unit]
      Description=vLLM (OpenAI-compatible API behind a Nebius tunnel)
      After=network-online.target docker.service
      Wants=network-online.target
      Requires=docker.service

      [Service]
      ExecStartPre=-/usr/bin/docker rm -f vllm
      ExecStartPre=/usr/bin/docker pull vllm/vllm-openai:latest
      ExecStart=/usr/bin/docker run --rm --name vllm \
        --gpus all \
        -p 127.0.0.1:8000:8000 \
        -v /opt/vllm/hf-cache:/root/.cache/huggingface \
        vllm/vllm-openai:latest \
        --model Qwen/Qwen3-8B \
        --served-model-name qwen3-8b \
        --max-model-len 8192 \
        --host 0.0.0.0 --port 8000
      ExecStop=/usr/bin/docker stop vllm
      Restart=on-failure
      RestartSec=10
      TimeoutStartSec=0

      [Install]
      WantedBy=multi-user.target

runcmd:
  # Agent first: the tunnel is up while the backend is still pulling images.
  - [/usr/local/sbin/install-nebius-tunnel-agent]
  - mkdir -p /opt/vllm/hf-cache
  - systemctl enable --now vllm.service

This configuration enables the following actions:

Add a username and a public SSH key to the VM.
Authenticate by using the service account attached to the VM.
Create two Nebius Tunnels services:
- llm: To address the model and chat with it.
- ssh: To connect to the VM by using SSH.
Deploy the tunnel agent on the VM.
Deploy a Docker image with the Qwen/Qwen3-8B model on the VM.

Create the VM:
- Web console
- CLI
1. In the sidebar, go to Compute → Virtual machines.
2. Click Create resource → Virtual machine. The creation flow is a step-by-step wizard. The sidebar shows your progress through the configuration sections. To move between sections, click Back and Next.
3. On the Compute step, configure computing resources:
  
  In the Platform section, select:
  
  With GPUs
  
  Regular
  
  NVIDIA® H100 NVLink
  
  In the Settings section, set:
  
  Preset: 1 GPU - 16 CPUs - 200 GiB RAM.
  
  Project: The project where you create all resources for the tunnel.
  
  VM name: tunnel-llm.
4. On the Storage step:
  
  Preserve the Ubuntu 24.04 LTS for NVIDIA® GPUs (CUDA® 13) disk image.
  
  Change the disk type to Network SSD IO M3.
  
  Change the disk size to 186 GiB.
5. On the Network step, check the network and subnet. Keep None in the Public IP address field.
6. On the Configuration step, configure access, identity and VM startup settings:
  
  In the Service account field, select the tunnel-agent-sa service account.
  
  Enable a custom cloud-init configuration file.
  
  In the field that appears, paste the contents of the llm.yaml file prepared earlier.
7. On the Review step, check the full VM configuration. To change a section quickly, click next to the corresponding block. The wizard opens the relevant step with your current settings.
8. Click Create VM.
nebius compute instance create \ --name tunnel-llm \ --resources-platform gpu-h100-sxm \ --resources-preset 1gpu-16vcpu-200gb \ --boot-disk-attach-mode READ_WRITE \ --boot-disk-managed-disk-name tunnel-llm-boot \ --boot-disk-managed-disk-type network_ssd_io_m3 \ --boot-disk-managed-disk-size-gibibytes 186 \ --boot-disk-managed-disk-source-image-family-image-family ubuntu24.04-cuda13.0 \ --network-interfaces "[{\"name\": \"eth0\", \"ip_address\": {}, \"subnet_id\": \"<subnet_ID>\"}]" \ --service-account-id <service_account_ID> \ --cloud-init-user-data "$(cat llm.yaml)"
In the command, specify the subnet ID and the ID of the tunnel-agent-sa service account.

Wait for the VM to start running.

Check the result and chat with the LLM

The first VM boot pulls the LLM Docker image and model weights. The model deployment takes 10-20 minutes. The tunnel agent becomes available before the LLM starts.

To check the status of the model deployment, connect to the VM through the ssh tunnel service:
ssh -o ProxyCommand="openssl s_client -quiet -verify_quiet -servername %h -connect %h:443" \ ubuntu@ssh-<tunnel_masked_ID>.tunnel.applications.<region>.nebius.cloud
In the VM address, specify:
- tunnel_masked_ID: Mask of the tunnel ID without the applicationtunnel- prefix and regional routing code. Contains the last 15 characters of the tunnel ID. For example, for applicationtunnel-e00abcdef123456789, the masked ID of the tunnel is abcdef123456789.
- region: Region of the project where you created the resources.
For more information about the tunnel URL, see Supported traffic.
Check logs:
journalctl -u vllm -f
When the logs contain INFO: Application startup complete, the model is ready to serve requests.
Disconnect from the VM.
Send a chat completion request from your local machine:
curl https://llm-<tunnel_masked_ID>.tunnel.applications.<region>.nebius.cloud/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3-8b", "stream": false, "messages": [{"role": "user", "content": "Say hi"}] }'
The result is a JSON response with the model’s answer in the choices[0].message.content parameter. If the response is failed to connect to local service, this means that the model hasn’t started yet. Wait a few minutes and try again.

How to delete the created resources

The created VM and its boot disk are chargeable. If you don’t need them, delete the VM, so Nebius AI Cloud doesn’t charge for the resources. The boot disk is deleted automatically when you delete the VM. You can also delete free resources:

​Costs

​Steps

​Create a tunnel

​Prepare a service account

​Create a virtual machine

​Check the result and chat with the LLM

​How to delete the created resources

Costs

Steps

Create a tunnel

Prepare a service account

Create a virtual machine

Check the result and chat with the LLM

How to delete the created resources