External Secrets

logo

In the complex Kubernetes ecosystem, managing sensitive information such as passwords, API keys, and authentication tokens represents a critical challenge for both security and operational efficiency of applications.

The External Secrets Operator (ESO) is a robust solution, consolidated in the CNCF, ideal for synchronizing secrets from external sources directly into the Kubernetes cluster.

It is a Kubernetes operator that extends the Kubernetes API with Custom Resource Definitions (CRDs). Its main function is to read information from external secret providers, such as AWS Secrets Manager, HashiCorp Vault, Google Secret Manager, and Azure Key Vault, and automatically inject them as native Kubernetes Secrets.

At its core, ESO acts as a secure bridge. Development and operations teams can keep their secrets centralized and managed in a secure and auditable location, while applications within Kubernetes can consume these secrets in the standard way, through environment variables or volumes, without the need for complex modifications.

The External Secrets architecture is based on two main CRDs:

SecretStore or ClusterSecretStore: Defines how to connect and authenticate to an external secret provider. The ClusterSecretStore version allows the configuration to be accessible by all cluster namespaces, while SecretStore is limited to a single namespace.
ExternalSecret: Describes which secret should be fetched from the external provider and how it should be created or updated in Kubernetes. This resource references a SecretStore to obtain the necessary access credentials.

Advantages

Centralization and External Management: Allows secrets to be managed by specialized systems that offer advanced features for auditing, automatic rotation, and granular access policies.
Enhanced Security (GitOps): By preventing secret values from being stored directly in Kubernetes YAML manifests, External Secrets is a fundamental ally for GitOps practices. Git repositories contain only the reference to the external secret, not its value, drastically reducing the risk of exposure.
Automatic Synchronization: The operator continuously monitors the external secret provider. Any changes to secrets are automatically synchronized with the corresponding Secrets in Kubernetes, ensuring applications always have the most recent credentials.
Application Compatibility: Since the end result is a native Kubernetes Secret, applications don't need any custom logic to interact with the secret provider. They consume secrets the way they always have.
Wide Provider Support: The tool is compatible with a vast range of secret providers, both from public clouds and on-premises solutions, offering flexibility for different architectures.
Active Community and Open Source Project: As a CNCF (Cloud Native Computing Foundation) project, External Secrets has a vibrant community and active development, ensuring good support, constant updates, and a solid user base.

Disadvantages

Despite its numerous advantages, it's important to consider some points before adopting External Secrets:

Initial Configuration Complexity: Creating SecretStores and defining the correct permissions both in the cluster and in external providers can be complex for teams less experienced with Kubernetes.
Still Uses Native Secrets: For applications that require the highest level of security, the fact that External Secrets materializes the secret into a native Secret in Kubernetes (which by default is only Base64 encoded in etcd) can be a disadvantage. Although etcd can be encrypted, some security policies may prohibit storing secrets in the cluster. This can be considered an advantage or disadvantage - the fact that the secret materializes as an object in the cluster. If we were to inject a secret directly from Vault or any other provider during pod initialization and that connection failed, the pod wouldn't start, but with a secret already in Kubernetes we eliminate one more dependency.
Dependency on an Additional Component: Adds another controller to the cluster that needs to be monitored, maintained, and updated.
Performance at Large Scale: In very large clusters with a massive number of secrets being synchronized, the operator's performance can become a point of attention, consuming resources and generating a considerable volume of logs.
Sync Time: There is latency between when the external secret understands that a secret has changed.

Why Did It Become the Most Used Tool?

The rise of External Secrets to the position of most widely used tool for secrets management in Kubernetes can be attributed to a combination of factors:

Solution to a Universal Problem: Secrets management is a universal pain point in container environments, and External Secrets offers an elegant and effective solution that integrates perfectly with the Kubernetes ecosystem.
"GitOps-Friendly" Approach: With the popularization of GitOps as a standard for continuous delivery, the ability to keep secrets out of Git repositories has become a fundamental requirement.
Balance Between Security and Usability: The tool finds an excellent balance by allowing secrets to be managed in secure external vaults, without imposing a large development overhead on application teams, who can continue to consume secrets in a familiar way.

External Secrets vs. Sealed Secrets vs. GitOps

While the External Secrets Operator (ESO) has established itself as one of the most popular and robust solutions, Sealed Secrets offers a distinct and simpler approach that may be ideal for certain scenarios. The choice between them fundamentally depends on your architecture, the scale of your operation, and your GitOps workflow.

External Secrets: Its main advantage lies in centralization and unified secrets management, acting as a HUB that facilitates auditing, key rotation, etc., but which brings with it the dependencies of external services and extra complexity as we've already seen.

Sealed Secrets: Has a complete focus on GitOps, but comes with a completely different approach. Instead of connecting to an external vault, it allows you to securely encrypt secrets so they can be stored directly in a Git repository.

How Sealed Secrets Works

A controller is installed in the Kubernetes cluster, which generates a key pair (public and private).

The public key is used to encrypt (seal) your Kubernetes Secret manifest files. The result is a new SealedSecret object, which is safe to be committed to Git.

The controller in the cluster is the only one that has the private key and, therefore, the only one capable of decrypting (unsealing) the SealedSecret and creating the native Secret in the cluster.

Main Advantages of Sealed Secrets

Pure GitOps Workflow: Allows everything, including secrets, to be versioned and managed through Git. This simplifies the deployment process and ensures a complete declarative state of your infrastructure.
Self-Contained: There are no dependencies on external services for runtime secrets management. Everything needed is in the cluster and the Git repository.
Simplicity: It is relatively simpler to configure and use compared to External Secrets, especially for smaller teams and less complex environments.
Repository Security: The original secret is never exposed in the Git repository, only its encrypted version.

Disadvantages of Sealed Secrets

Decentralized Management: Each cluster has its own key pair, which can make secrets management difficult across multiple clusters.
Key Rotation: Rotating the Sealed Secrets encryption key requires re-encrypting all existing secrets, which can be a manual and error-prone process.
Fewer Advanced Features: Does not offer the same auditing and secrets lifecycle management features as dedicated secret providers.

It's important to remember that limitations can be worked around and manual processes automated, but it's not a feature that comes with the tool out of the box.

Characteristic	External Secrets Operator	Sealed Secrets
Source of Truth	External secret provider (Vault, AWS SM, etc.)	Git repository
Workflow	Synchronization with external source	Encryption and commit to Git (GitOps)
Dependencies	External secret provider	Self-contained in cluster
Scalability	High, ideal for multiple clusters	Less scalable for multiple clusters
Secret Rotation	Managed by external provider, automatic synchronization	Manual, requires re-encryption
Complexity	Moderate to high	Low to moderate
Ideal Use Case	Enterprise environments, multiple clusters, need for centralized auditing.	Projects that follow GitOps strictly, simpler environments, storing all configuration in Git.

For most organizations that already use a centralized secret provider or operate at a scale that demands robust management and auditing, External Secrets is indeed the most complete and secure choice.

However, Sealed Secrets has immense value and should not be dismissed. It is an excellent option for:

Teams that adopt GitOps as their main philosophy and want to have a single declarative source of truth in their repositories.
Simpler scenarios, with a limited number of clusters, where the overhead of managing an external secrets vault is not justified.
Open source applications or projects where simplicity and the absence of external dependencies are priorities.

Advantages​

Disadvantages​

Why Did It Become the Most Used Tool?​

External Secrets vs. Sealed Secrets vs. GitOps​

How Sealed Secrets Works​

Main Advantages of Sealed Secrets​

Disadvantages of Sealed Secrets​