Kubernetes GitOps with FluxCD - Part 2 - Secret Management using SOPS

Table of Contents

In Kubernetes-based GitOps workflows, securely managing sensitive information presents a unique challenge. While GitOps principles encourage storing all cluster configurations in Git repositories, committing plaintext secrets creates significant security risks. SOPS (Secrets OPerationS) offers an elegant solution to this problem when integrated with FluxCD.

In our previous post, we explored how to perform the initial setup of FluxCD in a Kubernetes cluster. Building on that foundation, we’ll now address one of the most critical aspects of GitOps implementation: Secret Management.

SOPS, developed by Mozilla, enables encrypting specific values within YAML, JSON, and other configuration formats while keeping the file structure intact. When combined with FluxCD’s native support for decryption, this allows teams to safely store encrypted secrets directly in their Git repositories alongside other infrastructure definitions. The secrets are only decrypted at runtime within the Kubernetes cluster, maintaining security while preserving the GitOps workflow.

This article explores how to implement a robust secret management strategy using SOPS and FluxCD. We’ll cover the setup process, encryption workflows, integration with different key management systems, and best practices for maintaining secure GitOps operations in production environments.

1. Install SOPS and GPG

Your installation method may differ depending on your operating system. Below is the command for openSUSE Tumbleweed:

1sudo zypper in sops gpg2

For other distributions:

  • Ubuntu/Debian: sudo apt install sops gnupg2
  • macOS: brew install sops gnupg
  • Arch Linux: sudo pacman -S sops gnupg

2. Generate Keypair

Next, we’ll generate a GPG keypair that will be used for encryption and decryption:

 1export KEY_NAME="cluster.local"
 2export KEY_COMMENT="For Flux Secrets"
 3
 4gpg --batch --full-generate-key <<EOF
 5%no-protection
 6Key-Type: 1
 7Key-Length: 4096
 8Subkey-Type: 1
 9Subkey-Length: 4096
10Expire-Date: 0
11Name-Comment: ${KEY_COMMENT}
12Name-Real: ${KEY_NAME}
13EOF

This creates a 4096-bit RSA key with no expiration date. The output will look similar to:

gpg: directory '/home/apurv/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/apurv/.gnupg/openpgp-revocs.d/A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4.rev'

Now retrieve the key fingerprint, which we’ll need for subsequent steps:

 1gpg --list-secret-keys "${KEY_NAME}"
 2
 3gpg: checking the trustdb
 4gpg: marginals needed: 3  completes needed: 1  trust model: pgp
 5gpg: depth: 0  valid:   2  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 2u
 6sec   rsa4096 2025-02-25 [SCEAR]
 7      A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4
 8uid           [ultimate] cluster.local (For Flux Secrets)
 9ssb   rsa4096 2025-02-25 [SEA]
10      8B514FAAC98DF16F61EE487A45E294E6CACB03D4

Take note of the fingerprint (in this example: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4).

3. Store Private Key as Kubernetes Secret

Now we’ll export the private key and store it as a Kubernetes secret that FluxCD can access:

1gpg --export-secret-keys --armor "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4" | kubectl create secret generic sops-gpg \
2--namespace=flux-system \
3--from-file=sops.asc=/dev/stdin
4
5secret/sops-gpg created

Let’s verify the secret was created properly:

1apurv@oxygen:~> kubectl -n flux-system get secrets
2
3NAME          TYPE     DATA   AGE
4flux-system   Opaque   3      25h
5sops-gpg      Opaque   1      38s

Since the local key is unprotected (we used %no-protection for demonstration purposes), we should delete it from our local machine:

1gpg --delete-secret-keys "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4"

4. Configure In-Cluster Secrets Decryption

For production environments, the recommended approach is to store secrets in a separate repository with restricted access. However, for this tutorial, we’ll patch the existing cluster manifest to add support for the decryption provider.

Edit cluster/default/flux-system/kustomization.yaml

 1apiVersion: kustomize.config.k8s.io/v1beta1
 2kind: Kustomization
 3resources:
 4 - gotk-components.yaml
 5 - gotk-sync.yaml
 6+patches:
 7+  - patch: |-
 8+      apiVersion: kustomize.toolkit.fluxcd.io/v1
 9+      kind: Kustomization
10+      metadata:
11+        name: flux-system
12+        namespace: flux-system
13+      spec:
14+        decryption:
15+          provider: sops
16+          secretRef:
17+            name: sops-gpg

This patch configures FluxCD to use SOPS for decryption, referencing our previously created GPG key secret. Let’s commit and push our changes, then verify the deployment:

1git commit -m "Configured SOPS decryption for main cluster repo" && git push origin
2
3flux get kustomizations flux-system --watch
4
5NAME            REVISION                SUSPENDED       READY   MESSAGE                              
6flux-system     main@sha1:c35e4e45      False           True    Applied revision: main@sha1:c35e4e45

5. Export Public Key and Configure SOPS

We’ll export the public key to the cluster directory:

1gpg --export --armor "A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4" > ./cluster/default/.sops.pub.asc

Next, create a SOPS configuration file to specify which parts of the YAML files should be encrypted:

1cat <<EOF > ./cluster/default/.sops.yaml
2creation_rules:
3  - path_regex: .*.yaml
4    encrypted_regex: ^(data|stringData)$
5    pgp: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4
6EOF

This configuration tells SOPS to only encrypt the data and stringData sections of YAML files, leaving metadata intact for Kubernetes to process. Commit these configuration files:

1git add . && git commit -m "Added SOPS public key and config"

6. Validate the Setup

Let’s verify our setup by creating a sample secret and ensuring it works properly:

Create ./cluster/default/samplesecret.yaml:

1apiVersion: v1
2kind: Secret
3metadata:
4  name: samplesecret
5  namespace: default
6type: Opaque
7stringData:
8  message: This is a secret message

Now encrypt this file with SOPS:

1sops --encrypt --in-place samplesecret.yaml

After encryption the yaml file looks like below.

 1apiVersion: v1
 2kind: Secret
 3metadata:
 4    name: samplesecret
 5    namespace: default
 6type: Opaque
 7stringData:
 8    message: ENC[AES256_GCM,data:H0iMCWDGpqRSQZtjSXTbMlXaRvXQRxqs,iv:/r5ylcoqWd1wnOp1p9ksDUEs+kkPkdQz31I66LXrpxo=,tag:r2m9+dyo9tnBz+ty42cfrA==,type:str]
 9sops:
10    kms: []
11    gcp_kms: []
12    azure_kv: []
13    hc_vault: []
14    age: []
15    lastmodified: "2025-02-25T12:00:11Z"
16    mac: ENC[AES256_GCM,data:AuJ9hHq2+Yars67Rw8QhkpjCu8iwqGVsWDJ2/1oEx430yaGSCNXXcZnntek9rsI3xuEzyVwTJcXIE/1yW1bCvPPdWUOel+dY+mLkLSUxio238FbDoGI8mv+7FLpDe4Tn9GZdBbaMyGF6FwXXfQJ8021tLSqK6IJQN8nW2XbT0L0=,iv:KkmRa+HzZ0lI0vsEdqX+ZY6HIpudA/0ABny6OruwuQ8=,tag:6gaSJssqOEQk53DvXCZT0Q==,type:str]
17    pgp:
18        - created_at: "2025-02-25T12:00:11Z"
19          enc: |-
20            -----BEGIN PGP MESSAGE-----
21
22            hQIMA0XilObKywPUARAArblXLVn7VaJ+Sw2jNqptBPoPaDq7CA4V2LqUoGQUdIiX
23            6pHZoZmfWCpvXAUTqLQ1xEnZ32XHl+lwEVxLzV6OckkYcZKxMGz1lHZGmK7QghtV
24            Gm32lFvXVOnMfI9uZ96WH9WW+8Dng3VgyBAK/putNG+N3NLkeXa3vrWaaNaHY/Dj
25            aBPK1FqzFAYLbFlGjadA1+1xpFfA4JnE5dLX5HTDIo9DKldJWxxqY6cGo6jTg1lO
26            j+vmLHQicdKrApbMqdq1KyRLWTU21B0cRlfIIK37rX9xESXjUlwPmF27sew2KPsx
27            iuLyofDu/fVD9kFUC/Zkrmog0IgBWaKuUniOBT1KkcPJoJ5AX9m3qyTGjTGNwvVU
28            DWATppNCqnO2NhN2D13sOxL5/BSkJT4HZgdj8oIiQQjv0QwLW6nwlEHcmveiX6aK
29            CRUOR51gp+ja+fAlZnevYUpJMJZFz4TX1LTVXqUd8dJeXePaAsCPzO38Ywrpe83D
30            Z9n9ABYawCohwSb+Nd1/eoU0RBZwRfcD0PumTbWj8mbSMn6cPHJuAY1j1WZKH1/y
31            PM583Vv1zA4NlUJwhAiqI/X0kl22+Qh4tq+tdrhPVLw7P+m7diHG2pyd6jKDZ/K9
32            D3tY+eeQWcM3EV0hRBN7yA87Rs07lCy0m72PqAtD07qoUbO6jLXipIuE7TgPuZ3U
33            aAEJAhBexKkWWprEJjk+jpt4h8aXyG7wUAeafCWr2kIWz0/kOSnG0STCjuL1kDbu
34            +Ysah6EqMijU1sQBJv9Jn5oQ9eTAiHwN2Brh8F1nCPT+E6Ih6lbiJSLAD8duEa7V
35            Nsgo8cYB0ebx
36            =R3G0
37            -----END PGP MESSAGE-----            
38          fp: A2E9A640A0E47EB72E6A4F0517B6FD7A7486D6E4
39    encrypted_regex: ^(data|stringData)$
40    version: 3.9.4

Lets push these changes.

1git add samplesecret.yaml && git commit -m "Added sample secret"
2git push origin

Let’s verify that the secret is created in the cluster:

1kubectl get secrets
2NAME           TYPE     DATA   AGE
3samplesecret   Opaque   1      52s

Verify the content of the secret:

1kubectl get secret samplesecret -o jsonpath='{.data.message}' | base64 --decode
2
3This is a secret message

What next ?

Future posts will explore advanced GitOps patterns with FluxCD, including:

  • Helm chart automation
  • Image update automation
  • Notification and alerting configuration

Stay tuned for each of these topics.

References