How to ingest metrics with Nebius Observability Agent for Kubernetes

​

Prerequisites

1. Install Nebius Observability Agent for Kubernetes.
2. Create a Managed Kubernetes cluster and connect to it by using kubectl.

​

Update the agent

1. Create a values.yaml file with metrics configuration:
   config:
     metrics:
       enabled: true
       collectAgentMetrics: false
       collectK8sClusterMetrics: false
   
2. Update your Nebius Observability Agent for Kubernetes installation:
   helm upgrade nebius-o11y-agent oci://cr.nebius.cloud/observability/public/nebius-o11y-agent-helm \
     --version $(curl https://nebius-o11y-agent.storage.eu-north1.nebius.cloud/nebius-o11y-agent-helm/latest-release) \
     --namespace observability \
     --values values.yaml
   

​

Configuration options

• config.metrics.enabled: Enable or disable metrics collection. Default: true.
• config.metrics.collectAgentMetrics: Option to collect metrics from the Nebius Observability Agent for Kubernetes itself. Default: false.
• config.metrics.collectK8sMetrics: Enable collection of Kubernetes infrastructure metrics (API servers, nodes, cAdvisor, Hubble). Default: false.
• config.metrics.excludedNamespaces: List of namespaces to exclude from metrics collection.

​

Excluding namespaces

config:
  metrics:
    enabled: true
    collectAgentMetrics: false
    excludedNamespaces:
      - kube-system
      - monitoring
      - cert-manager
      - istio-system

​

Collecting both logs and metrics

config:
  logs:
    enabled: true
    collectAgentLogs: false
    excludedNamespaces:
      - kube-system

  metrics:
    enabled: true
    collectAgentMetrics: false
    collectK8sClusterMetrics: false

​

Collected targets

​

Service endpoints targets

• kubernetes-service-endpoints (scrape interval: 15s) Collects metrics from Kubernetes services with prometheus.io/scrape: "true" annotation. This target discovers services and scrapes their endpoints.
• kubernetes-service-endpoints-slow (scrape interval: 5m) Collects metrics from services with prometheus.io/scrape_slow: "true" annotation for less frequent scraping with extended timeout (30s).

​

Pod targets

• kubernetes-pods (scrape interval: 15s) Collects metrics directly from Pods with prometheus.io/scrape: "true" annotation. Only scrapes Pods running on the same node as the agent (node-local collection).
• kubernetes-pods-slow (scrape interval: 5m) Collects metrics from Pods with prometheus.io/scrape_slow: "true" annotation for less frequent scraping with extended timeout (30s).

​

(Optional) Kubernetes infrastructure targets

• kubernetes-apiservers Collects metrics from Kubernetes API servers for cluster health monitoring.
• kubernetes-nodes Collects node-level metrics via kubelet /metrics endpoint, including node resource usage and status.
• kubernetes-nodes-cadvisor Collects container metrics via kubelet /metrics/cadvisor endpoint, providing detailed container resource usage.
• hubble (scrape interval: 15s) Collects network observability metrics from Cilium Hubble in the kube-system namespace, if available.

​

Custom targets

• additionalTargets User-defined custom scrape targets that can be configured via config.metrics.additionalTargets in the Helm values.

​

Target filtering

• Namespace filtering: Targets in namespaces listed in excludedNamespaces are automatically excluded.
• Node locality: Pod and node targets are filtered to only collect metrics from the same node where the agent is running.
• Pod state filtering: Pods in Pending, Succeeded, Failed or Completed states are excluded from collection.

​

Annotation requirements

​

Data enrichment

• k8s_cluster_id: Cluster ID
• k8s_node_group_id: Node group ID
• app.kubernetes.io/name: Application name label
• k8s.namespace.name: Namespace name
• k8s.deployment.name: Deployment name (if applicable)
• k8s.statefulset.name: StatefulSet name (if applicable)
• k8s.daemonset.name: DaemonSet name (if applicable)
• k8s.cronjob.name: CronJob name (if applicable)
• k8s.job.name: Job name (if applicable)
• k8s.node.name: Node name
• k8s.pod.name: Pod name
• k8s.pod.start_time: Pod start time
• container.image.tag: Container image tag
• k8s.container.restart_count: Restart count of the container in the Pod
• k8s_pod_uid: Pod unique identifier

​

Pod annotations for metrics scraping

apiVersion: v1
kind: Pod
metadata:
  name: my-application
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"
spec:
  ...

​

Required annotations

• prometheus.io/scrape: Set to "true" to enable metrics scraping for this Pod.
• prometheus.io/port: The port number where your application exposes metrics (as a string).
• prometheus.io/path: The path where metrics are available. Default: /metrics.

​

Example deployment with annotations

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - name: my-app
        image: my-app:latest
        ports:
        - containerPort: 8080

​

Troubleshooting

1. Verify that the agent is running:
   kubectl get pods --namespace observability
   
2. Check agent logs for errors:
   kubectl logs <Pod_name> --namespace observability
   
3. Check the status of targets by using Prometheus API exposed by the agent:
   1. Run the following command to forward the port to your local machine:
      kubectl --namespace observability port-forward pod/<Pod_name> 8080:8080
      
   2. Open http://127.0.0.1:8080/api/v1/targets in your browser to inspect the targets that the agent currently scrapes. For more details, see the api/v1/targets Prometheus documentation.