SecretStore

The first thing we need to do when using External Secrets Operator (ESO) is choose the provider that will store the secrets — such as AWS Secrets Manager, SSM Parameter Store, GCP Secrets, among others.

It doesn't make sense to study all providers in depth: each one has different details, and the documentation covers this very well. What they all have in common is that they need to be configured through a SecretStore.

Example with AWS Secrets Manager

We won't go into detail about IAM in AWS, just the necessary concepts you need to know.

One of the most used providers is AWS Secrets Manager. To use it, we need to:

Create the SecretStore resource
Create a secret with AWS credentials in the cluster (if not using IRSA). We'll see this approach later.
Ensure the IAM Role has correct permissions

Before starting, you need to have IAM keys in AWS and know which role we're going to use.

Quick Important IAM Concepts

IAM User >>> Assumes a Role >>> And the Role has permissions.

A user generally has the following keys that are used to identify them:

access-key-id
secret-access-key

A second approach we'll discuss here is having a Kubernetes cluster service account assume an AWS role directly, playing the role of IAM User.

Service Account (RBAC Kubernetes) >>> Assumes an AWS Role >>> And the Role has permissions.

If you observe carefully, both cases need the role that will be assumed. That's where we put the necessary permissions.

Example with IAM User

Create an IAM user in AWS called external-secrets-iam (or any other name) and generate the access credentials. Many tutorials show how to do this. We need these credentials as a secret inside Kubernetes to reference them.

# Creating an aws-external-secrets-credentials secret in the external-secrets namespace
kubectl create secret generic aws-external-secrets-credentials --from-literal=access-key=xxxxxxxxxxxx --from-literal=secret-access-key=xxxxxxxxxxxxxxxxxxxx -n external-secrets

Here we have the SecretStore example using these credentials. We haven't created the role yet. An IAM User can assume multiple roles, so we need to reference which one will be used, but we haven't created it yet.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      role: arn:aws:iam::123456789012:role/external-secrets # Role we'll use (we'll create it later)
      region: us-east-2
      auth:
        secretRef: # Reference to the previously created secret
          accessKeyIDSecretRef:
            name: aws-external-secrets-credentials
            key: access-key
          secretAccessKeySecretRef:
            name: aws-external-secrets-credentials
            key: secret-access-key

This role needs read permissions in Secrets Manager so ESO can synchronize the secrets we'll map in the future.

Parameter Store is also supported, but a new SecretStore must be created pointing to the correct service.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-parameter-store
spec:
  provider:
    aws:
      service: ParameterStore
      role: arn:aws:iam::YOUR_ACCOUNT_ID:role/external-secrets
      region: us-east-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-external-secrets-credentials
            key: access-key
          secretAccessKeySecretRef:
            name: aws-external-secrets-credentials
            key: secret-access-key

The same IAM Role can be used, as long as it has permissions for both services — which is what we'll do here.

Policy (Permission in the Role)

Create a role in AWS called external-secrets, since that's the name used in the examples above, and add permissions for the main ESO services.

It's a good practice to limit the role's access only to the secrets/parameters it should access. We can do this with Resource Tags. If it's not necessary, just remove the condition.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SecretsManagerAccess",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecrets",
        "secretsmanager:ListSecretVersionIds",
        "secretsmanager:BatchGetSecretValue"
      ],
      "Resource": [ 
        "*"
      ],
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/external-secrets": "true"
        }
      }
    },
    {
      "Sid": "SSMParameterStoreAccess",
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter",
        "ssm:GetParameters",
        "ssm:GetParametersByPath",
        "ssm:DescribeParameters"
      ],
      "Resource": [ 
        "*"
      ],
      "Condition": {
        "StringEquals": {
          "ssm:ResourceTag/external-secrets": "true"
        }
      }
    }
  ]
}

Push Secret

ESO also supports the PushSecret resource, which allows creating and updating secrets in the provider (like Secrets Manager). For this, we need to add extra permissions. I generally like to do this in a separate policy.

NOTE: Use your region and account ID.

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:CreateSecret",
    "secretsmanager:PutSecretValue",
    "secretsmanager:TagResource",
    "secretsmanager:DeleteSecret"
  ],
  "Resource": [
    "arn:aws:secretsmanager:us-east-2:YOUR_ACCOUNT_ID:secret:dev-*"
  ]
}

The ideal is to be more restrictive, so I prefer to use tag-based control on delete:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:CreateSecret",
        "secretsmanager:PutSecretValue",
        "secretsmanager:TagResource"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-east-2:YOUR_ACCOUNT_ID:secret:dev-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DeleteSecret"
      ],
      "Resource": [
        "arn:aws:secretsmanager:us-east-2:YOUR_ACCOUNT_ID:secret:dev-*"
      ],
      "Condition": {
        "StringEquals": {
          "secretsmanager:ResourceTag/managed-by": "external-secrets"
        }
      }
    }
  ]
}

This dev-* prefix can follow any pattern: app name, namespace, cluster, etc.

For Parameter Store (Push), the idea is similar:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter*",
        "ssm:PutParameter*",
        "ssm:AddTagsToResource",
        "ssm:ListTagsForResource"
      ],
      "Resource": [
        "arn:aws:ssm:us-east-2:YOUR_ACCOUNT_ID:secret-manager/dev-*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:DeleteParameter*"
      ],
      "Resource": [
        "arn:aws:ssm:us-east-2:YOUR_ACCOUNT_ID:secret-manager/dev-*"
      ],
      "Condition": {
        "StringEquals": {
          "ssm:ResourceTag/managed-by": "external-secrets"
        }
      }
    }
  ]
}

Finally, configure so that the IAM User external-secrets-iam can assume the external-secrets role. This is done in the role, not in the user:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:user/external-secrets-iam"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

IAM Role with IRSA

The configuration with static keys (accessKey/secretAccessKey) in a Kubernetes Secret has some risks:

Security: They are long-lived credentials. If leaked, they can be used from anywhere.
Management: Manual rotation is laborious and often forgotten.

IRSA (IAM Roles for Service Accounts) is the native and secure way to give AWS permissions to pods in EKS without using AWS credentials in the pod. A pod uses a service account and if this service account has permission to assume the role in AWS, we don't need to pass credentials via secretRef.

Superior Security: No long-lived keys stored. The pod uses temporary tokens generated and validated by STS.
Simplified Management: No need to create, rotate, or distribute keys manually.

By adding a trust relationship in the role, we can say that the service account (external-secrets) created by external-secrets in the namespace (external-secrets) can assume the external-secrets role in AWS. Once again remembering this is a configuration of the external-secrets role.

You need to configure an OIDC Provider used by the cluster for federation because it is your IDP. You probably already have the OIDC Provider configured. If not, look for better information about this or we'll stray too far from the subject.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AssumeRulesExternalSecrets",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::YOUR_ACCOUNT_ID:oidc-provider/oidc.eks.us-east-2.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXX"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-2.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXX:sub": "system:serviceaccount:external-secrets:external-secrets"
                }
            }
        }
    ]
}

This way, you can remove the secretRef in SecretStore and use JWT.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore # We'll explain why we changed to ClusterSecretStore instead of SecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: "us-east-2"
      auth:
        jwt:
          serviceAccountRef: # We specify which service account will be used and in which namespace it will be.
            name: external-secrets
            namespace: external-secrets

However, the external-secrets service account needs to have an annotation on it saying which role it can assume. This can be defined in values.yaml during helm chart installation.

#values.yaml
###...
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::1234567889911:role/external-secrets
###...

This would be reflected in the external-secrets service account created by the chart.

❯ kubectl describe sa -n external-secrets external-secrets

Name:                external-secrets
Namespace:           external-secrets
Labels:              app.kubernetes.io/instance=external-secrets
                     app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=external-secrets
                     app.kubernetes.io/version=v0.18.1
                     helm.sh/chart=external-secrets-0.18.1
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::1234567889911:role/external-secrets ### Here...
                     meta.helm.sh/release-name: external-secrets
                     meta.helm.sh/release-namespace: external-secrets
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

Apply the SecretStore in the best way according to the solution you chose.

kubectl apply -f clustersecretstore.yaml

ClusterSecretStore vs SecretStore

Both are External Secrets Operator (ESO) resources used to define how and with whom ESO connects to a provider (like AWS Secrets Manager, SSM, etc.). The difference between them is basically the scope where they can be used.

SecretStore

Scope: Namespace

Usage: Can only be referenced by ExternalSecret within the same namespace

Ideal when: You want to separate secrets by teams, environments, or apps with isolated policies

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-store
  namespace: app-dev
  ...

An ExternalSecret in the app-prod namespace cannot use the SecretStore above.

Why would we do this? We can have different roles where the permission within each one is different for the tags. For example, a role that only allows reading secrets with the tag team=frontend and another role team=backend.

ClusterSecretStore

Scope: Cluster-wide (global)

Usage: Can be referenced by ExternalSecret in any namespace

Ideal when: You want a single configuration for multiple namespaces, centralize access to the provider, or standardize role usage.

Perfect for clusters with multiple apps/namespaces using the same provider and role.

If you're defining just one for the cluster, ClusterSecretStore is what you're looking for.

Example with AWS Secrets Manager​

Quick Important IAM Concepts​

Example with IAM User​

Policy (Permission in the Role)​

Push Secret​

IAM Role with IRSA​

ClusterSecretStore vs SecretStore​

SecretStore​

ClusterSecretStore​

Example with AWS Secrets Manager

Quick Important IAM Concepts

Example with IAM User

Policy (Permission in the Role)

Push Secret

IAM Role with IRSA

ClusterSecretStore vs SecretStore

SecretStore

ClusterSecretStore