DeploymentRuntimeConfig

DeploymentRuntimeConfig is a fundamental resource that allows you to completely customize how Crossplane providers are executed in your cluster. It offers granular control over deployment aspects, computational resources, security, and integration with Kubernetes infrastructure.

This is a subject that often goes unnoticed when we talk about providers, but it's essential for production environments.

Purpose and Use Cases

• Resource Control

  • Define specific CPU/memory limits and requests for each provider
  • Configure different resource profiles (development vs production)
  • Optimize performance based on each provider's specific workload

• Security Settings

  • Apply custom SecurityContexts
  • Configure specific Service Accounts for providers
  • Define network policies and workload isolation
  • Implement Pod Identity and Workload Identity for cloud service authentication
  • Configure IRSA (IAM Roles for Service Accounts) in AWS EKS
  • Use Azure AD Workload Identity

• Deployment Customization

  • Modify deployment strategies (RollingUpdate, Recreate)
  • Configure affinity/anti-affinity rules for intelligent placement
  • Apply tolerations for execution on specific nodes

And we have many other utilities!

Structure and Configuration

The structure is initially very simple, but quite verbose. The three main sections are:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-aws-config
spec:
  deploymentTemplate: {} # Customizes how the provider pod is executed (resources, volumes, etc)
  serviceAccountTemplate: {} # Defines the service account the provider will use for authentication
  serviceTemplate: {} # Configures the service that exposes provider metrics and endpoints

Purpose of Main Keys

deploymentTemplate

• Controls aspects of the provider's Deployment/Pod (CPU, memory, volumes, nodeSelector, etc.)
• Used for 90% of customization cases
• Allows configuring everything you would do in a normal Kubernetes Deployment

serviceAccountTemplate

• Creates/configures automatically the ServiceAccount for the provider
• Essential for Pod Identity, IRSA, and Workload Identity
• Avoids having to create ServiceAccounts manually

serviceTemplate

• Configures the Service that exposes provider metrics
• Useful for monitoring and debugging
• Less used than the other two templates

A typical DeploymentRuntimeConfig allows configuring:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-aws-config
spec:
  deploymentTemplate:
    spec:
      replicas: 2
      template:
        spec:
          containers:
          - name: package-runtime
            resources:
              requests:
                cpu: 100m
                memory: 128Mi
              limits:
                cpu: 500m
                memory: 512Mi
            securityContext:
              runAsNonRoot: true
              runAsUser: 65532
          nodeSelector:
            workload-type: crossplane
          tolerations:
          - key: "crossplane-dedicated"
            operator: "Equal"
            value: "true"
            effect: "NoSchedule"

Application in Providers

To apply a DeploymentRuntimeConfig to a provider, you reference it in the installation:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1
  runtimeConfigRef:
    name: provider-aws-config

Affinity and Anti-Affinity

Configure where providers should or should not be executed:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-ha-config
spec:
  deploymentTemplate:
    spec:
      template:
        spec:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100
                podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      pkg.crossplane.io/provider: provider-aws
                  topologyKey: kubernetes.io/hostname

Volume Mounts and ConfigMaps

Mount specific configurations in providers:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-custom-config
spec:
  deploymentTemplate:
    spec:
      template:
        spec:
          containers:
          - name: package-runtime
            volumeMounts:
            - name: custom-config
              mountPath: /etc/provider-config
              readOnly: true
          volumes:
          - name: custom-config
            configMap:
              name: provider-configuration

Pod Identity and Workload Identity

Configure authentication without credentials using managed identities:

AWS EKS with IRSA (IAM Roles for Service Accounts)

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: aws-irsa-config
spec:
  deploymentTemplate:
    spec:
      template:
        spec:
          serviceAccountName: crossplane-provider-aws
          containers:
          - name: package-runtime
            env:
            - name: AWS_ROLE_ARN
              value: "arn:aws:iam::123456789012:role/crossplane-provider-role"
            - name: AWS_WEB_IDENTITY_TOKEN_FILE
              value: "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
            volumeMounts:
            - name: aws-iam-token
              mountPath: "/var/run/secrets/eks.amazonaws.com/serviceaccount"
              readOnly: true
          volumes:
          - name: aws-iam-token
            projected:
              sources:
              - serviceAccountToken:
                  audience: sts.amazonaws.com
                  expirationSeconds: 86400
                  path: token
---
# Service Account with annotations for IRSA
apiVersion: v1
kind: ServiceAccount
metadata:
  name: crossplane-provider-aws
  namespace: crossplane-system
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/crossplane-provider-role"

Azure AKS with Workload Identity

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: azure-workload-identity-config
spec:
  deploymentTemplate:
    spec:
      template:
        spec:
          serviceAccountName: crossplane-provider-azure
          containers:
          - name: package-runtime
            env:
            - name: AZURE_CLIENT_ID
              value: "12345678-1234-1234-1234-123456789012"
            - name: AZURE_TENANT_ID
              value: "87654321-4321-4321-4321-210987654321"
            - name: AZURE_FEDERATED_TOKEN_FILE
              value: "/var/run/secrets/azure/tokens/azure-identity-token"
            volumeMounts:
            - name: azure-identity-token
              mountPath: "/var/run/secrets/azure/tokens"
              readOnly: true
          volumes:
          - name: azure-identity-token
            projected:
              sources:
              - serviceAccountToken:
                  audience: api://AzureADTokenExchange
                  expirationSeconds: 3600
                  path: azure-identity-token
---
# Service Account with labels for Workload Identity
apiVersion: v1
kind: ServiceAccount
metadata:
  name: crossplane-provider-azure
  namespace: crossplane-system
  annotations:
    azure.workload.identity/client-id: "12345678-1234-1234-1234-123456789012"
  labels:
    azure.workload.identity/use: "true"

GCP GKE with Workload Identity

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: gcp-workload-identity-config
spec:
  deploymentTemplate:
    spec:
      template:
        spec:
          serviceAccountName: crossplane-provider-gcp
          containers:
          - name: package-runtime
            env:
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: "/var/run/secrets/tokens/gcp-ksa/token"
            volumeMounts:
            - name: gcp-ksa-token
              mountPath: "/var/run/secrets/tokens/gcp-ksa"
              readOnly: true
          volumes:
          - name: gcp-ksa-token
            projected:
              sources:
              - serviceAccountToken:
                  audience: https://gcp.googleapis.com/
                  expirationSeconds: 3600
                  path: token
---
# Service Account with annotation for Workload Identity
apiVersion: v1
kind: ServiceAccount
metadata:
  name: crossplane-provider-gcp
  namespace: crossplane-system
  annotations:
    iam.gke.io/gcp-service-account: "[email protected]"

Important Limitation: Multi-Account Scenarios

The above examples work only for one account/subscription/project per provider. For multi-account/multi-tenant scenarios, you need different approaches:

Approach 1: Multiple Providers with Specific DeploymentRuntimeConfigs

# Provider for AWS Production Account
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-production
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1
  runtimeConfigRef:
    name: aws-production-config
---
# Provider for AWS Development Account
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-development
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1
  runtimeConfigRef:
    name: aws-development-config
---
# RuntimeConfig for Production
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: aws-production-config
spec:
  serviceAccountTemplate:
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/crossplane-prod-role"
---
# RuntimeConfig for Development
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: aws-development-config
spec:
  serviceAccountTemplate:
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: "arn:aws:iam::987654321098:role/crossplane-dev-role"

Approach 2: Cross-Account Roles with ProviderConfigs

For a more scalable solution, use a single provider with multiple ProviderConfigs:

# Single provider
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v1
  runtimeConfigRef:
    name: aws-cross-account-config
---
# RuntimeConfig that supports assume role
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: aws-cross-account-config
spec:
  serviceAccountTemplate:
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: "arn:aws:iam::111111111111:role/crossplane-master-role"
---
# ProviderConfig for Production Account
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: aws-production
spec:
  credentials:
    source: IRSA
  assume_role:
    role_arn: "arn:aws:iam::123456789012:role/crossplane-target-role"
    session_name: "crossplane-production"
---
# ProviderConfig for Development Account
apiVersion: aws.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: aws-development
spec:
  credentials:
    source: IRSA
  assume_role:
    role_arn: "arn:aws:iam::987654321098:role/crossplane-target-role"
    session_name: "crossplane-development"

Approach 3: Multi-Tenant with External Secrets

For complex scenarios, combine with External Secrets Operator:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: multi-tenant-config
spec:
  deploymentTemplate:
    spec:
      template:
        spec:
          containers:
          - name: package-runtime
            volumeMounts:
            - name: multi-account-secrets
              mountPath: "/etc/crossplane/accounts"
              readOnly: true
          volumes:
          - name: multi-account-secrets
            csi:
              driver: secrets-store.csi.k8s.io
              readOnly: true
              volumeAttributes:
                secretProviderClass: crossplane-accounts

• Simple Scenario (1-3 accounts) Use Multiple providers with specific RuntimeConfigs
• Medium Scenario (4-10 accounts) Use Cross-account roles with ProviderConfigs
• Enterprise Scenario (10+ accounts) Use Multi-tenant with External Secrets + automation

It's still possible to create a provider with a generic configuration that serves all scenarios, but customize the ProviderConfig with an XRD. We'll see this approach later.

DeploymentRuntimeConfig is a powerful tool for optimizing and customizing Crossplane provider execution, allowing you to adapt platform behavior to your organization and infrastructure's specific needs.