Extra
What happens if we delete a secret in AWS that is mapped?
❯ k get secrets
NAME TYPE DATA AGE
app-auth kubernetes.io/basic-auth 2 104m
minha-secret Opaque 1 37m
minha-secret-k8s Opaque 4 129m
❯ k describe es external-secret-auth
Name: external-secret-auth
Namespace: default
Labels: <none>
Annotations: <none>
API Version: external-secrets.io/v1
Kind: ExternalSecret
Metadata:
Creation Timestamp: 2025-07-01T01:13:27Z
Generation: 1
Resource Version: 3407507
UID: acf4b93d-b1ee-4c36-b0fa-0a1ed172ae10
Spec:
Data From:
Extract:
Conversion Strategy: Default
Decoding Strategy: None
Key: test/app
Metadata Policy: None
Refresh Interval: 1m
Secret Store Ref:
Kind: ClusterSecretStore
Name: aws-secrets-manager
Target:
Creation Policy: Owner
Deletion Policy: Retain
Name: app-auth
Template:
Engine Version: v2
Merge Policy: Replace
Metadata:
Labels:
App: my-app
Type: kubernetes.io/basic-auth
Status:
Binding:
Name: app-auth
Conditions:
Last Transition Time: 2025-07-01T01:13:27Z
Message: secret synced
Reason: SecretSynced
Status: True # Pay attention here!
Type: Ready
Refresh Time: 2025-07-01T02:58:27Z
Synced Resource Version: 1-dd100cb940ef5edeb926b19bde61ff16c82209bcb92fb4574e67bbcf
Events: <none>
I deleted the test/app.
If you delete a secret in AWS that is being used by an ExternalSecret, what happens will depend on the configuration and default behavior of ESO:
- ESO tries to sync every refreshInterval.
- If it doesn't find the secret in the provider, it doesn't delete the Secret from Kubernetes by default.
- But the .status.conditions of the ExternalSecret enters an error state with something like:
❯ k describe es external-secret-auth
Name: external-secret-auth
Namespace: default
Labels: <none>
Annotations: <none>
API Version: external-secrets.io/v1
Kind: ExternalSecret
Metadata:
Creation Timestamp: 2025-07-01T01:13:27Z
Generation: 1
Resource Version: 3408307
UID: acf4b93d-b1ee-4c36-b0fa-0a1ed172ae10
Spec:
Data From:
Extract:
Conversion Strategy: Default
Decoding Strategy: None
Key: test/app
Metadata Policy: None
Refresh Interval: 1m
Secret Store Ref:
Kind: ClusterSecretStore
Name: aws-secrets-manager
Target:
Creation Policy: Owner
Deletion Policy: Retain
Name: app-auth
Template:
Engine Version: v2
Merge Policy: Replace
Metadata:
Labels:
App: my-app
Type: kubernetes.io/basic-auth
Status:
Binding:
Name: app-auth
Conditions:
Last Transition Time: 2025-07-01T03:00:27Z
Message: could not get secret data from provider # Message
Reason: SecretSyncedError # Error
Status: False # Already set to false status
Type: Ready
Refresh Time: 2025-07-01T02:59:27Z
Synced Resource Version: 1-dd100cb940ef5edeb926b19bde61ff16c82209bcb92fb4574e67bbcf
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning UpdateFailed 2m18s external-secrets error processing spec.dataFrom[0].extract, err: operation error Secrets Manager: GetSecretValue, https response error StatusCode: 400, RequestID: 4dee00d1-6413-4f21-adad-5e7b5bebf05d, InvalidRequestException: You can't perform this operation on the secret because it was marked for deletion.
ESO doesn't delete the Secret from K8s just because the provider secret disappeared. This is intentional, to avoid breaking the app due to an external error. If you restore the secret, the status will become true and the error message will disappear.
Best practice tip: Always have alerts on the .status.conditions of ExternalSecrets with Prometheus or another system. Validate if .status.conditions[].status == False and reason == SecretSyncedError.
ClusterExternalSecret
If you have 20 namespaces and want to replicate the same ExternalSecret, without copying the YAML 20 times... there's ClusterExternalSecret.
apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
name: sync-para-todos
spec:
namespaceSelector:
matchLabels:
sync-db: "true"
template:
metadata:
name: db-secret
spec:
refreshInterval: 1m
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: minha-secret-k8s
creationPolicy: Owner
data:
- secretKey: DB_USER
remoteRef:
key: test/app
property: username
- secretKey: DB_PASS
remoteRef:
key: test/app
property: password
- ESO will create an ExternalSecret named db-secret in each namespace with sync-db: "true".
- This ExternalSecret will have exactly the template content.
- Each namespace will receive its own Secret named minha-secret-k8s.
If you want something more dynamic, you can still use Go Template to change the secret name per namespace.
###...
spec:
refreshInterval: 1m
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: app-secret-{{ .metadata.namespace }}
creationPolicy: Owner
###...
Multi-Provider Secret (Multi-Backend)
Have one SecretStore for AWS, another for GCP, another for Vault and combine everything in the same ExternalSecret, pulling data from each via data or dataFrom.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: multi-provider-secret
namespace: default
spec:
refreshInterval: 1h
target:
name: minha-secret-final
creationPolicy: Owner
# The secretStoreRef at spec level (global) is optional if all fields use a specific store in data.
data:
- secretKey: DB_USER
remoteRef:
key: my-app/credentials
property: username
secretStoreRef:
name: aws-secretstore
kind: ClusterSecretStore
- secretKey: GCP_API_KEY
remoteRef:
key: gcp-secret-id
secretStoreRef:
name: gcp-secretstore
kind: ClusterSecretStore
- secretKey: VAULT_TOKEN
remoteRef:
key: secret/data/my-app
property: token
secretStoreRef:
name: vault-secretstore
kind: ClusterSecretStore
And have a single secret combining multiple parts expecting a result like this in the secret.
data:
DB_USER: <comes from AWS>
GCP_API_KEY: <comes from GCP Secrets Manager>
VAULT_TOKEN: <comes from HashiCorp Vault>