Syncing SecretStash secrets with Managed Service for Kubernetes® clusters by using External Secrets Operator

MysteryBox is now called SecretStash. The Nebius AI Cloud CLI, API and Terraform provider still use the mysterybox service identifier. Keep using mysterybox in scripts and manifests.

Applications in a Managed Service for Kubernetes cluster often read credentials, such as API keys or passwords, from Kubernetes Secrets. Instead of creating these Secrets manually or storing their values in manifests and CI/CD pipelines, you can keep the values in SecretStash and sync them with the cluster by using External Secrets Operator (ESO). ESO reads a SecretStash secret, writes its payload to a Kubernetes Secret and updates it when you add a new primary version in SecretStash. In this tutorial, you will grant a service account access to a secret, install ESO, set up authentication for ESO and sync the secret with your cluster.

Costs

Nebius AI Cloud only charges for running a Managed Kubernetes cluster. For more details, see Managed Kubernetes pricing and SecretStash pricing.

Prerequisites

Install and configure the Nebius AI Cloud CLI.
Check that your project ID is saved in the Nebius AI Cloud CLI profile configuration:
cat ~/.nebius/config.yaml
Make sure you are in a group that has the admin role within your tenant or project; for example, the default admins group. You can check this in the Administration → IAM section of the web console.
Create a Managed Kubernetes cluster with at least one node group and connect to it using kubectl.
Install Helm.
Create a secret in SecretStash. The examples use a secret with the password key.

Steps

Run all commands in the same terminal session. Some commands save values, such as resource IDs, to environment variables. The kubectl commands that you run later read these variables and pass their values to the Kubernetes manifests, so you don’t need to copy the values manually.

Grant a Nebius AI Cloud service account access to the secret

ESO reads the secret on behalf of a Nebius AI Cloud service account. This service account needs the mysterybox.payload-viewer role for the secret.

Creating a secret or a version of a secret does not automatically grant you access to view payloads in that secret or version. The editor role is enough to create a secret or a version, but viewing payloads requires the mysterybox.payload-viewer role, which is a sub-role of admin but not editor.

Get the ID of the secret and save it to an environment variable:
export SECRET_ID=$(nebius mysterybox secret get-by-name \ --parent-id <project_ID> \ --name <secret_name> \ --format jsonpath='{.metadata.id}')
Make sure that the variable isn’t empty: run echo $SECRET_ID. If the output is empty, check the secret name and the project ID.

Create a service account and save its ID to an environment variable:

export SA_ID=$(nebius iam service-account create \
  --name <service_account_name> \
  --format jsonpath='{.metadata.id}')

Create a custom group for the service account and save the group ID to an environment variable:

export GROUP_ID=$(nebius iam group create \
  --parent-id <project_ID> \
  --name <group_name> \
  --format jsonpath='{.metadata.id}')

Grant the mysterybox.payload-viewer role for the secret to the group:
nebius iam access-permit create \ --parent-id $GROUP_ID \ --resource-id $SECRET_ID \ --role mysterybox.payload-viewer
To grant access to all secrets in the project, specify the project ID in --resource-id instead of the secret ID.

Add the service account to the group:

nebius iam group-membership create \
  --parent-id $GROUP_ID \
  --member-id $SA_ID

Install ESO

Add the ESO Helm repository:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

Install ESO 2.10.0 or later with its custom resource definitions:

helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --version ">=2.10.0" \
  --set installCRDs=true

Make sure that the ESO Pods are running:
kubectl get pods -n external-secrets
All Pods should have the Running status.

Create a namespace

Specify a name for the namespace and save it to an environment variable:
export K8S_NAMESPACE=<namespace>
The kubectl commands that you run later create the Kubernetes resources in this namespace, including the Kubernetes Secret with the synced values. Pods can only use Secrets from their own namespace. If the workloads that need the secret already run in a namespace, specify its name and skip the next step.

Create the namespace:

kubectl create namespace $K8S_NAMESPACE

Set up authentication and connect ESO to SecretStash

ESO can authenticate in SecretStash using one of the following methods:

Workload identity federation (recommended): ESO exchanges a short-lived token of a Kubernetes ServiceAccount for an access token of the Nebius AI Cloud service account. You don’t need to store long-lived credentials in the cluster.
Service account credentials: ESO uses an authorized key of the Nebius AI Cloud service account. You store the key in a Kubernetes Secret.

For either method, you create a SecretStore: a Kubernetes custom resource that ESO adds to your cluster. A SecretStore tells ESO how to authenticate in SecretStash. For more details, see the ESO documentation. Set up one of the authentication methods and create a SecretStore for it:

Workload identity (recommended)
Service account credentials

Create the mysterybox-reader ServiceAccount together with a Role and a RoleBinding. ESO requests tokens for this ServiceAccount through the Kubernetes TokenRequest API, and the Role and RoleBinding allow this:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: mysterybox-reader
  namespace: $K8S_NAMESPACE
---
# Allows creating tokens for the mysterybox-reader ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: external-secrets-token-request
  namespace: $K8S_NAMESPACE
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    resourceNames: ["mysterybox-reader"]
    verbs: ["create"]
---
# Grants the Role to the ESO controller
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: allow-eso-to-request-token
  namespace: $K8S_NAMESPACE
subjects:
  - kind: ServiceAccount
    name: external-secrets
    namespace: external-secrets
roleRef:
  kind: Role
  name: external-secrets-token-request
  apiGroup: rbac.authorization.k8s.io
EOF

In the RoleBinding, external-secrets is the name and namespace of the ESO controller’s ServiceAccount when you install ESO as shown above. If you installed ESO with a different Helm release name or namespace, get the ServiceAccount name by running kubectl get serviceaccounts -n <ESO_namespace>.

Get the ID of your Managed Kubernetes cluster:

nebius mk8s cluster get-by-name \
  --name <cluster_name> \
  --format jsonpath='{.metadata.id}'

Create federated credentials. They let a workload outside Nebius AI Cloud, here the mysterybox-reader ServiceAccount in your cluster, get access tokens of a Nebius AI Cloud service account. To create them, run the following command:
nebius iam federated-credentials create \ --parent-id <project_ID> \ --name <federated_credentials_name> \ --subject-id $SA_ID \ --oidc-provider-issuer-url https://oidc.mk8s.api.nebius.cloud/<cluster_ID> \ --federated-subject-id system:serviceaccount:$K8S_NAMESPACE:mysterybox-reader
The command contains the following parameters:
- --parent-id: ID of the project.
- --subject-id: ID of the Nebius AI Cloud service account whose access tokens the ServiceAccount gets. The command takes the ID from the SA_ID variable.
- --oidc-provider-issuer-url: URL of the cluster’s OpenID Connect (OIDC) issuer. Replace <cluster_ID> with the ID that you got in the previous step. For more details, see Using an OIDC public endpoint in Managed Kubernetes clusters.
- --federated-subject-id: Kubernetes ServiceAccount in the system:serviceaccount:<namespace>:<ServiceAccount_name> format. The command uses the mysterybox-reader ServiceAccount in the namespace from the K8S_NAMESPACE variable.
Nebius AI Cloud then accepts the tokens that your cluster issues for this ServiceAccount and exchanges them for access tokens of the service account.

Create the SecretStore. The command passes the service account ID from the SA_ID variable to iamServiceAccountID:

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: mysterybox
  namespace: $K8S_NAMESPACE
spec:
  provider:
    nebiusmysterybox:
      apiDomain: api.nebius.cloud:443
      auth:
        workloadIdentity:
          serviceAccountRef:
            name: mysterybox-reader
          iamServiceAccountID: $SA_ID
EOF

Make sure that the SecretStore is valid:
kubectl get secretstore mysterybox -n $K8S_NAMESPACE
In the output, the STATUS column should show Valid. To validate the SecretStore, ESO gets an access token for the service account, so this status also confirms that the federated credentials and the RoleBinding work. Don’t proceed until the SecretStore has the Valid status. If it doesn’t, see Troubleshooting.

Don’t generate the credentials file or create the Kubernetes Secret with the credentials by using Terraform: Terraform saves the private key of the service account in its state, and anyone who can read the state can use the key to act as the service account.

For more details, see the Terraform documentation.

Generate an authorized key for the service account and save the credentials to a file:
nebius iam auth-public-key generate \ --service-account-id $SA_ID \ --output ~/.nebius/$SA_ID-credentials.json
The command creates a key pair, uploads the public key to the service account and writes the credentials, including the private key, to the output file. To limit how long the key remains valid, add the --expires-at parameter with the expiration time in RFC 3339 format, for example, --expires-at 2027-01-31T23:59:59Z. For more details, see How to use authorized keys. If a credentials file with this name already exists, delete it before you run the command. The command doesn’t fully overwrite an existing file, and ESO rejects the result with the invalid subject credentials: malformed JSON error.

Store the credentials in a Kubernetes Secret:

kubectl create secret generic mysterybox-sa-credentials \
  -n $K8S_NAMESPACE \
  --from-file=credentials.json=$HOME/.nebius/$SA_ID-credentials.json

Create the SecretStore. The command refers to the Kubernetes Secret that you created in the previous step:

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: mysterybox
  namespace: $K8S_NAMESPACE
spec:
  provider:
    nebiusmysterybox:
      apiDomain: api.nebius.cloud:443
      auth:
        serviceAccountCredsSecretRef:
          name: mysterybox-sa-credentials
          key: credentials.json
EOF

Make sure that the SecretStore is valid:
kubectl get secretstore mysterybox -n $K8S_NAMESPACE
In the output, the STATUS column should show Valid. To validate the SecretStore, ESO gets an access token for the service account, so this status also confirms that the authorized key in the Kubernetes Secret works. Don’t proceed until the SecretStore has the Valid status. If it doesn’t, see Troubleshooting.

Sync the secret with the cluster

An ExternalSecret is another ESO custom resource. It tells ESO which SecretStash secret to read and which Kubernetes Secret to write it to. For more details, see the ESO documentation.

Create the ExternalSecret. The command passes the secret ID from the SECRET_ID variable to key:

kubectl apply -f - <<EOF
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-credentials
  namespace: $K8S_NAMESPACE
spec:
  # How often ESO checks the secret for changes
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore
    name: mysterybox
  target:
    # Name of the Kubernetes Secret that ESO creates
    name: app-credentials
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: $SECRET_ID
EOF

With dataFrom.extract, every key of the secret payload becomes a key in the Kubernetes Secret. To sync only some keys, replace dataFrom with data and add an entry with property for each key:

  data:
    - secretKey: password # Key in the Kubernetes Secret
      remoteRef:
        key: $SECRET_ID
        property: password # Key in the secret payload

If you don’t specify property, ESO writes the whole payload as JSON to the Kubernetes Secret key.

Make sure that ESO has synced the secret:

kubectl get externalsecret app-credentials -n $K8S_NAMESPACE

The result looks like the following:

NAME              STORETYPE     STORE        REFRESH INTERVAL   STATUS         READY   LAST SYNC
app-credentials   SecretStore   mysterybox   1h                 SecretSynced   True    5s

If the status isn’t SecretSynced, see Troubleshooting.

Check the keys of the Kubernetes Secret:
kubectl get secret app-credentials -n $K8S_NAMESPACE -o jsonpath='{.data}'
The output contains the keys of the secret payload with base64-encoded values.

Your workloads can now use the app-credentials Kubernetes Secret, for example, as environment variables or a mounted volume. By default, ESO syncs the primary version of the secret. When you make another version primary, ESO updates the Kubernetes Secret within the interval set in refreshInterval. To sync a specific version instead, add version: <version_ID> under remoteRef or extract. To get the version ID, list the versions of the secret. For example, to pin the version in the ExternalSecret that you created:

  dataFrom:
    - extract:
        key: $SECRET_ID
        version: <version_ID>

If you sync only some keys, add version to each entry:

  data:
    - secretKey: password
      remoteRef:
        key: $SECRET_ID
        property: password
        version: <version_ID>

When a version is pinned, ESO doesn’t pick up a new primary version until you change version in the ExternalSecret and apply it again.

Troubleshooting

If the ExternalSecret doesn’t have the SecretSynced status, get the details of the error:

kubectl describe externalsecret app-credentials -n $K8S_NAMESPACE

Check the Events section of the output and the following possible causes:

Nothing is reconciled: Make sure that the ESO Pods are running and the ESO custom resource definitions are installed:
kubectl get pods -n external-secrets kubectl get crds | grep external-secrets.io
The SecretStore isn’t found: Make sure that the SecretStore and the ExternalSecret are in the same namespace and that secretStoreRef.name matches the SecretStore name.
The SecretStore isn’t valid with workload identity federation: Make sure that the issuer URL and the system:serviceaccount:<namespace>:mysterybox-reader subject in the federated credentials match your cluster and ServiceAccount, and that the RoleBinding refers to the ESO controller’s ServiceAccount.
The SecretStore isn’t valid with service account credentials: Get the details by running kubectl describe secretstore mysterybox -n $K8S_NAMESPACE.
- If the events contain Public Key not exists, expired or deactivated, the authorized key has expired or been deleted. Delete the old credentials file, generate new credentials and recreate the mysterybox-sa-credentials Kubernetes Secret. ESO caches access tokens, so the SecretStore can stay valid for several hours after the key stops working.
- If the events contain invalid subject credentials: malformed JSON, delete the credentials file, generate the credentials again and recreate the Kubernetes Secret.
Access is denied: Make sure that the service account is in a group with the mysterybox.payload-viewer role for the secret or its project.
The payload is empty or has unexpected keys: Check the payload of the secret:
nebius mysterybox payload get --secret-id mbsec-*** --version-id mbsecver-***
Keys that you specify in property must match the keys of the payload.
A Kubernetes resource has an empty namespace or ID: If you opened a new terminal session when configuring the sync of secrets, the environment variables aren’t set. Set K8S_NAMESPACE, SA_ID and SECRET_ID again, then delete and recreate the affected resources.

Security

Audit Logs

Key Management Service

SecretStash (formerly MysteryBox)

Syncing SecretStash secrets with Managed Service for Kubernetes® clusters by using External Secrets Operator

Costs

Prerequisites

Steps

Grant a Nebius AI Cloud service account access to the secret

Install ESO

Create a namespace

Set up authentication and connect ESO to SecretStash

Sync the secret with the cluster

Troubleshooting

​Costs

​Prerequisites

​Steps

​Grant a Nebius AI Cloud service account access to the secret

​Install ESO

​Create a namespace

​Set up authentication and connect ESO to SecretStash

​Sync the secret with the cluster

​Troubleshooting

Costs

Prerequisites

Steps

Grant a Nebius AI Cloud service account access to the secret

Install ESO

Create a namespace

Set up authentication and connect ESO to SecretStash

Sync the secret with the cluster

Troubleshooting