Networking add-ons in Managed Service for Kubernetes®

• CoreDNS is a cluster DNS server.
• Cilium is a networking solution that provides security and observability.

• For CoreDNS, use a custom ConfigMap.
• For Cilium, edit the default ConfigMap.

​

CoreDNS

kubectl get configmap -n kube-system coredns -o yaml

1. Create a custom ConfigMap coredns-custom.yaml. It should contain the keys with the .override and .server extensions.
   • .override keys allow you to add plugins to the default Server Block of CoreDNS. You cannot override the parameters already specified in the default ConfigMap.
   • .server keys allow you to specify additional Server Blocks for CoreDNS.
   An example of a custom ConfigMap:
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: coredns-custom
     namespace: kube-system
   data:
     log.override: |
       log
     custom.server: |
       example.io:8053 {
         forward . 8.8.8.8
       }
   
   This ConfigMap:
   • Adds the log plugin to start logging at the system level.
   • Creates a new Server Block for the example.io domain. All requests directed to example.io at port 8053 should be forwarded to another DNS server at 8.8.8.8.
   See the CoreDNS documentation for more information on the Corefile parameters and the list of available plugins.
2. Apply the custom configuration:
   kubectl apply -f coredns-custom.yaml
   

​

Cilium

• Some pods might contain sensitive data, and Cilium enforces rules that only certain internal services or authorized users are allowed to access it.
• If a node requires restricted access, Cilium ensures that only internal services with proper credentials or traffic with specific labels are allowed.

kubectl edit configmap -n kube-system cilium-config

​

Integration with Istio

1. Install Istio.
2. In the Cilium ConfigMap, set the bpf-lb-sock-hostns-only parameter to true:
   kubectl -n kube-system patch configmap cilium-config \
     --type merge \
     -p='{"data":{"bpf-lb-sock-hostns-only":"true"}}'
   
   kubectl -n kube-system rollout restart ds/cilium
   
3. Wait until all Cilium pods are restarted.

​

Host firewall

1. Connect to the cluster.
2. Run the script that enables the host firewall:
   enable_cilium_host_firewall.sh

   #!/usr/bin/env bash
   
   set -euo pipefail
   
   # This script ensures that Cilium Host Firewall feature is enabled if all nodes run with the "set-name" feature.
   
   kubectl_args=("$@")
   
   cluster_name=$(kubectl "${kubectl_args[@]}" config current-context)
   
   current_value=$(
     kubectl "${kubectl_args[@]}" get configmap cilium-config \
       -n kube-system \
       -o jsonpath='{.data.enableHostFirewall}' \
       2>/dev/null || echo ""
   )
   
   if [[ "$current_value" == "true" ]]; then
     echo "Cilium Host Firewall feature is already enabled in cluster \"$cluster_name\". Nothing to do."
     exit 0
   fi
   
   echo "Verifying that every node has \"set-name\": \"eth0\" in network-data"
   for node_ref in $(kubectl "${kubectl_args[@]}" get nodes -o name); do
     echo "Inspecting $node_ref"
     node_name="${node_ref#node/}"
     output=$(
       kubectl "${kubectl_args[@]}" debug "$node_ref" \
         --profile=general \
         --image=busybox \
         -i -- \
         chroot /host sh -c \
           'if grep -q "\"set-name\": \"eth0\"" /var/lib/cloud/instance/network-config.json 2>/dev/null; then
              echo OK
            else
              echo BAD
            fi' 2>&1 \
         | grep -Eo 'OK|BAD'
     )
   
     echo "Cleaning up debug pod"
     kubectl "${kubectl_args[@]}" delete $(kubectl "${kubectl_args[@]}" get pod -o name | grep node-debugger-"$node_name") 2>/dev/null
   
     if [[ "$output" != "OK" ]]; then
       echo "ERROR: $node_ref does not have \"set-name\": \"eth0\" in network-data."
       echo "Ensure that all node groups are upgraded with the following command:"
       echo "  nebius mk8s node-group upgrade --latest-infra-version"
       exit 1
     fi
   done
   
   echo -e "\nAll nodes verified. Enabling Cilium Host Firewall"
   kubectl "${kubectl_args[@]}" patch configmap cilium-config -n kube-system \
     --type=merge \
     --patch $'data:\n  enable-host-firewall: "true"'
   
   echo "Patched cilium-config ConfigMap; new enable-host-firewall value:"
   kubectl "${kubectl_args[@]}" get configmap cilium-config -n kube-system \
     -o yaml \
     | sed -n 's/^[[:space:]]*enable-host-firewall:.*/&/p'
   
   echo "Restarting Cilium DaemonSet to pick up the new config"
   kubectl "${kubectl_args[@]}" -n kube-system rollout restart daemonset cilium
   
   echo -e "\nDone"
   

​

How the add-ons affect autoscaling node groups