Running vLLM inference on a Managed Service for Kubernetes® cluster with SkyPilot

​

Costs

​

Prerequisites

1. Make sure you are in a group that has at least the editor role within your tenant; for example, the default editors group. You can check this in the Administration → IAM section of the web console.
2. Install the SkyPilot CLI.

​

Steps

​

Prepare the infrastructure

1. In the web console, go to  AI orchestration → SkyPilot. If you don’t have a Managed SkyPilot server in your project, a landing page opens.
2. Click Create server. The creation page opens.
3. Enter a name for the server or keep the default one.
4. Click Create server.
5. Connect the SkyPilot CLI to your Managed SkyPilot server:
   1. In your terminal, run:
      sky api login -e "https://skypilot.nebius.cloud"
      
      The browser opens automatically.
   2. Select your Managed SkyPilot server, and click Connect.
   3. Follow the instructions to authenticate to Nebius AI Cloud and sign in to SkyPilot CLI.
   4. In your terminal, verify that the CLI displays the following:
      Logged into SkyPilot API server at: https://skypilot.nebius.cloud
      
6. Create a Managed Kubernetes cluster named vllm-k8s-cluster, and add a GPU node group. The Managed Kubernetes cluster will provide the GPU for running the vLLM server.

​

Verify that SkyPilot can access your Managed Kubernetes cluster

1. In the web console, go to  AI orchestration → SkyPilot.
2. On the Managed SkyPilot server page, click How to connect.
3. In the window that opens, click the public endpoint URL. The SkyPilot dashboard opens in the browser.
4. In the SkyPilot dashboard, go to the Infra tab, and click Refresh. The dashboard lists the Managed Kubernetes clusters available to SkyPilot, including vllm-k8s-cluster.
5. In your terminal, verify that SkyPilot can access your Managed Kubernetes cluster:
   sky check kubernetes
   
   The context for vllm-k8s-cluster is listed as enabled. For example:
   Kubernetes: enabled [compute]
     Allowed contexts:
     └── vllm-k8s-cluster: enabled.
   🎉 Enabled infra 🎉
     Kubernetes [compute]
       Allowed contexts:
       └── vllm-k8s-cluster
   

​

Prepare the vLLM task configuration and launch the vLLM server

1. Create and enter a directory for the task configuration file:
   mkdir ~/vllm-inference-tutorial
   cd ~/vllm-inference-tutorial
   
2. In this directory, create a vllm.yaml task configuration file. Replace <gpu_platform> with the GPU platform of your node group, such as H200:
   resources:
     cloud: kubernetes
     image_id: docker:vllm/vllm-openai:v0.11.0
     accelerators: "<gpu_platform>:1"
   
   run: |
     vllm serve Qwen/Qwen2.5-7B-Instruct
   
   This file defines a Kubernetes workload that requests one GPU of the specified platform and starts vLLM. The container image provides vLLM version v0.11.0 and its runtime dependencies. When the task starts, vLLM downloads the Qwen/Qwen2.5-7B-Instruct model from Hugging Face Hub and serves it through an OpenAI-compatible API.
   To customize the model server, you can add parameters to the vllm serve command in the task configuration file. For example, you can configure request logging or supply a custom chat template. For available options, see the vllm serve reference documentation.
3. Launch the task that starts the vLLM server:
   sky launch \
      -c vllm-inference \
      --infra k8s/vllm-k8s-cluster \
      vllm.yaml
   
   When prompted, confirm the launch. This command creates a vllm-inference SkyPilot environment in the Managed Kubernetes cluster named vllm-k8s-cluster and runs the task defined in vllm.yaml.
4. Wait until the logs confirm that the SkyPilot environment was launched and the vLLM server started successfully:
   ✓ Cluster launched: vllm-inference.
   ...
   INFO:     Application startup complete.
   
   You may now exit log streaming. The vLLM server continues to run.

​

Send an inference request

• To ask a question, send the following request to the vLLM server:
  sky exec vllm-inference 'curl --silent --show-error \
     http://127.0.0.1:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d "{
        \"model\": \"Qwen/Qwen2.5-7B-Instruct\",
        \"messages\": [
        {
           \"role\": \"user\",
           \"content\": \"Explain what a GPU is in simple terms (1-2 sentences).\"
        }
        ],
        \"max_tokens\": 128,
        \"temperature\": 0.2
     }"'
  
  The generated answer is in choices[0].message.content:
  Response example

  {"id":"chatcmpl-d3eb883a907f41d4a7e269dbd0918ebe","object":"chat.completion","created":1792065449,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"A GPU, or graphics processing unit, is a specialized processor designed to handle the complex calculations required for rendering images, videos, and animations, making it much more efficient at tasks related to visual output than a standard CPU.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":44,"total_tokens":89,"completion_tokens":45,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}
  
• To constrain the response to a predefined set of values, ask the model to select from the provided possible answers:
  sky exec vllm-inference 'curl --silent --show-error \
     http://127.0.0.1:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d "{
        \"model\": \"Qwen/Qwen2.5-7B-Instruct\",
        \"messages\": [
        {
           \"role\": \"user\",
           \"content\": \"Classify the sentiment of the following sentence: The application is easy to use and works perfectly.\"
        }
        ],
        \"structured_outputs\": {
           \"choice\": [\"positive\", \"negative\", \"neutral\"]
        },
        \"max_tokens\": 16
     }"'
  
  The generated answer is in choices[0].message.content:
  Response example

  {"id":"chatcmpl-3c7e5f832edd42f5a2f2d2c4f3e238dd","object":"chat.completion","created":1792582733,"model":"Qwen/Qwen2.5-7B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"positive","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":48,"total_tokens":50,"completion_tokens":2,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}
  

​

How to delete the created resources

• Delete the SkyPilot environment:
  sky down vllm-inference -y
  
• Delete the Managed SkyPilot server:
  1. In the web console, go to  AI orchestration → SkyPilot.
  2. On the Managed SkyPilot server page, go to the Settings tab.
  3. In the Danger zone section, click Delete server.
  4. In the window that opens, confirm the deletion.
• Delete the Managed Kubernetes cluster.

​

See also

• Examples: vLLM inference with SkyPilot