Skip to main content

Secrets Provider for Kubernetes

It is possible to consume Strongbox secrets in a Kubernetes environment through the use of a secrets provider. The secrets provider is a container that access secrets in Strongbox and populates files and/or the Kubernetes Secrets store.

This way an existing Kubernetes application does not have to use the Strongbox APIs directly, and can still be given access to secrets stored in Strongbox.

The kubernetes-secrets-provider repository (https://gitlab.com/avassa-public/kubernetes-secrets-provider.git) builds a container called avassa-mount-secrets. The container can be run in two different modes, as an initContainer and as a regular container.

The secrets provider can provide Strongbox secrets both as one or more files and as Kubernetes secrets.

How secrets are mapped and accessed is controlled through Kubernetes pod annotations. The annotations tells the secrets provider how to access an Avassa system and also which secrets should be mapped and how they should be mapped. Access to the annotations are provided thought the Downward API by mounting the metadata.annotations as a volume in the container.

A pod specification can, for example, look like this:

apiVersion: v1
kind: Pod
metadata:
  name: auth-demo
  annotations:
    avassa.io/api-host: api.theater-operation.avassa.io
    avassa.io/tenant: theater-operation
    avassa.io/service: theater-ops
    avassa.io/role: theater-ops-role
    avassa.io/ca-cert: |-
      -----BEGIN CERTIFICATE-----
      MIICGzCCAcCgAwIBAgITAIGw58I85P6bWhwIX2Qo8hcHHTAKBggqhkjOPQQDAjBj
      MRgwFgYDVQQDEw9BdmFzc2EgQVBJIHJvb3QxEjAQBgNVBAcTCVN0b2NraG9sbTEL
      MAkGA1UEBhMCU0UxDzANBgNVBAoTBkF2YXNzYTEVMBMGA1UECxMMZGlzdHJpYnV0
      aW9uMCIYDzIwMjIwNDIxMDcxMTMwWhgPMjAyNTA4MzAxMDQ3MzBaMGMxGDAWBgNV
      BAMTD0F2YXNzYSBBUEkgcm9vdDESMBAGA1UEBxMJU3RvY2tob2xtMQswCQYDVQQG
      EwJTRTEPMA0GA1UEChMGQXZhc3NhMRUwEwYDVQQLEwxkaXN0cmlidXRpb24wWTAT
      BgcqhkjOPQIBBggqhkjOPQMBBwNCAAQFdfEMHr6X8fuudI7fI01UtEzbTW0DHLhg
      v9G80P39l8D9FKY8IaRmILkn+wKgO/jCOGDBTXQHICnALCEZqWNUo08wTTAOBgNV
      HQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBATAnBgNVHR8EIDAeMBygGqAY
      hhZodHRwOi8vY3JsLmF2YXNzYS5uZXQvMAoGCCqGSM49BAMCA0kAMEYCIQCf/oBh
      6PSuZ23ODIgtdmOY4tC2UmSP10OY+bikzCqjFAIhAJb6gc/Mb5hKbgtwRJ2BdIPk
      s/ujLEF3Tgc70oUona1Y
      -----END CERTIFICATE-----
    # create a file for each entry in the secrets map
    avassa.io/vault.credentials.admin: files
    # store the secrets map in the file credentials.json
    avassa.io/vault.credentials.oper: json/credentials.json
    # store a single entry in the secrets map in the file user.txt
    avassa.io/vault.credentials.oper.user: user.txt
spec:
  serviceAccountName: nginx-sa
  volumes:
  - name: shared-data
    emptyDir: {}
  - name: podinfo
    downwardAPI:
      items:
        - path: annotations
          fieldRef:
            fieldPath: metadata.annotations
  - name: avassa-secrets
    emptyDir:
      medium: Memory
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
    - name: avassa-secrets
      mountPath: /avassa/secrets
  initContainers:
  - name: init-avassa-secrets
    image: avassa-mount-secrets:0.1
    imagePullPolicy: Never
    volumeMounts:
    - name: podinfo
      mountPath: /avassa/podinfo
    - name: avassa-secrets
      mountPath: /avassa/secrets
  hostNetwork: false
  dnsPolicy: Default

Before this pod can be loaded some preparation is needed.

Preparing the Avassa System​

Before the avassa-mount-secrets container can be used the Avassa system must be setup to allow the Kubernetes cluster to authenticate using Kubernetes Service Account Tokens.

Before Avassa can be configured some setup in Kubernetes is required.

Configure Kubernetes​

First, create a service account for the application by applying the following using kubectl.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: theater-operations
  namespace: default

Assign the system:auth-delegator role to the service account token to allow the remote Strongbox authenticator to use the applications service account token to invoke the token review endpoint to validate the service account token.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: theater-operations
  namespace: default

In order for the avassa-mount-secrets container to update secrets in the Kubernetes Secrets store, it must be given permission to perform operations towards the store.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: my-secrets-clusterrole
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-secrets-global
  namespace: default
roleRef: # referring to your ClusterRole
  kind: ClusterRole
  name: my-secrets-clusterrole
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: theater-operations
  namespace: default

Assign a service account to a pod using the serviceAccountName setting, eg:

apiVersion: v1
kind: Pod
metadata:
  name: curtain-control
spec:
  serviceAccountName: theater-operations
  containers:
  - name: alpine
    image: alpine
    command:
      - /bin/sh
      - "-c"
      - "sleep 60m"

Configure Strongbox for Kubernetes authentication​

Strongbox also needs to be configured to accept service account tokens as valid credentials. First extract the API certificate from Kubernetes (used for the api-ca-cert parameter below).

kubectl config view --raw --minify --flatten --output \
    'jsonpath={.clusters[].cluster.certificate-authority-data}' | \
    base64 --decode
-----BEGIN CERTIFICATE-----
MIIDBjCCAe6gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
a3ViZUNBMB4XDTIxMDkyNjA4NTE1NloXDTMxMDkyNTA4NTE1NlowFTETMBEGA1UE
AxMKbWluaWt1YmVDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALPj
RABeQOTAY9Vj4IORJ1oeAdHCXRvu6sLWokuX0rZuKj80YIJSiqXOsh2Djp8aD1fG
EU89+fJNVV/3Ba5pe9NODbp597xAs/wpzmlurM/O99czoOxNsfd6pu0KpsRZgzBv
UXzc9offqfr+bbFBrb6VD0fZ8q5ZJ+BupQcdc46/pm31AEG3MfmGv0jVU4d/RZTm
+s4vrjrCEb255XTdWFHMi/bKMRNKRCoTHVAi01FbdHEivzVlFzObgp+UbldZf1v9
GvqXzSmXLX87nn4Fa3JroPhNCajYiJikEBV5hIAe6RpGg42zr8GDrDj0e5CvpTOu
F8leq8hjEvknEk3XQBECAwEAAaNhMF8wDgYDVR0PAQH/BAQDAgKkMB0GA1UdJQQW
MBQGCCsGAQUFBwMCBggrBgEFBQcDATAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW
BBTBXYN94F5bU0shR57gmFiNtyRahDANBgkqhkiG9w0BAQsFAAOCAQEAl9mr8t+M
FCAsn2ZIuILwZeiOhNXqyftFR2la47jJmYFtV6FZwMG7zNWOZlXYKrfw8mrRsaAB
a2OvItNipywU54mJaN4OV9PuOadz3KjB3sZ78yXsteXgH3+TxNOwe4LW1mxeyxdL
2JBCPmgrAt0VqekN1q0RRFyI/5JkU6Q+y1rpFV+cLvJGnX47Icgbgf/BGM4Bo30q
1U0iTWrfcVUNQwpYtbHYPnbDD3F8E4FuX+G8J2QNgjHXGKHdCkM/s9IBKr/SLgiM
kv/++NI+D3yABup4kHRnAWPLxAsBVB/v4p8o6yLB3jwIDxHwC3LU4tyQ2n3a/RFR
ceaTK5VvI8G+xg==
-----END CERTIFICATE-----

Then extract a Service Account token that can be used to access the Token Review service in Kubernetes (used in the token-review-jwt parameter below).

kubectl describe secret theater-operations
Name:         theater-operations-token-ljckk
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: theater-operations
              kubernetes.io/service-account.uid: 66b26fef-6e9a-47ba-bea1-8ec1936b9551

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1111 bytes
namespace:  7 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImVGRXpUeGJORzI3Q3A0R1RDT1d1dWhINHU2c09CSDlqdjZINWZ6dXdYejAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InRoZWF0ZXItb3BlcmF0aW9ucy10b2tlbi1samNrayIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJ0aGVhdGVyLW9wZXJhdGlvbnMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2NmIyNmZlZi02ZTlhLTQ3YmEtYmVhMS04ZWMxOTM2Yjk1NTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDp0aGVhdGVyLW9wZXJhdGlvbnMifQ.DKzkbgRseQWIkSv6q7QHOlS3z9f8Y0NFNWFQsfMnyLBIIq2xsvSfRCAqh2Ps4vUr5xzCqlA16p0llFQs-MVwTvN6StXaAw6MfgAXQcqSOWYmPi5nKRTIyXQtsz5rEBZaz6hKVilek6KhIYCNeCAgpLilVcz7i55qtdHsz1JNUgWd7HqUDPwBLfx4BILY3B3L7myX3EE7WgpxLqqAKQ40mL9TVD2CpOa6Zw1X1ovP5ANTgUVjrDAosDmlL-BKNMsJOmb4LRzMr5OlY6oFRbuSOsZZ_NtI7WOE5zA8yoy18jSOqg33FYwMfFixjvjZRm1XQMV5DYD7JZH_GmrAJksUVw

Then configure Strongbox to accept service account tokens as valid credentials.

In Kubernetes default tokens use 'kubernetes/serviceaccount' as issuer, however ephemeral tokens use the api-servers service-account-issuer setting, which is often a URL. Both may have to be configured as valid issuers.

supctl create strongbox authentication kubernetes <<EOF
name: theater-ops
host: 192.168.0.147
jwks-uri: "https://192.168.0.147/openid/v1/jwks"
server-name-indication: kubernetes
api-ca-cert: |-
    -----BEGIN CERTIFICATE-----
    MIIDBjCCAe6gAwIBAgIBATANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwptaW5p
    a3ViZUNBMB4XDTIxMDkyNjA4NTE1NloXDTMxMDkyNTA4NTE1NlowFTETMBEGA1UE
    AxMKbWluaWt1YmVDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALPj
    RABeQOTAY9Vj4IORJ1oeAdHCXRvu6sLWokuX0rZuKj80YIJSiqXOsh2Djp8aD1fG
    EU89+fJNVV/3Ba5pe9NODbp597xAs/wpzmlurM/O99czoOxNsfd6pu0KpsRZgzBv
    UXzc9offqfr+bbFBrb6VD0fZ8q5ZJ+BupQcdc46/pm31AEG3MfmGv0jVU4d/RZTm
    +s4vrjrCEb255XTdWFHMi/bKMRNKRCoTHVAi01FbdHEivzVlFzObgp+UbldZf1v9
    GvqXzSmXLX87nn4Fa3JroPhNCajYiJikEBV5hIAe6RpGg42zr8GDrDj0e5CvpTOu
    F8leq8hjEvknEk3XQBECAwEAAaNhMF8wDgYDVR0PAQH/BAQDAgKkMB0GA1UdJQQW
    MBQGCCsGAQUFBwMCBggrBgEFBQcDATAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW
    BBTBXYN94F5bU0shR57gmFiNtyRahDANBgkqhkiG9w0BAQsFAAOCAQEAl9mr8t+M
    FCAsn2ZIuILwZeiOhNXqyftFR2la47jJmYFtV6FZwMG7zNWOZlXYKrfw8mrRsaAB
    a2OvItNipywU54mJaN4OV9PuOadz3KjB3sZ78yXsteXgH3+TxNOwe4LW1mxeyxdL
    2JBCPmgrAt0VqekN1q0RRFyI/5JkU6Q+y1rpFV+cLvJGnX47Icgbgf/BGM4Bo30q
    1U0iTWrfcVUNQwpYtbHYPnbDD3F8E4FuX+G8J2QNgjHXGKHdCkM/s9IBKr/SLgiM
    kv/++NI+D3yABup4kHRnAWPLxAsBVB/v4p8o6yLB3jwIDxHwC3LU4tyQ2n3a/RFR
    ceaTK5VvI8G+xg==
    -----END CERTIFICATE-----
token-review-jwt: eyJhbGciOiJSUzI1NiIsImtpZCI6ImVGRXpUeGJORzI3Q3A0R1RDT1d1dWhINHU2c09CSDlqdjZINWZ6dXdYejAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InRoZWF0ZXItb3BlcmF0aW9ucy10b2tlbi1samNrayIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJ0aGVhdGVyLW9wZXJhdGlvbnMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2NmIyNmZlZi02ZTlhLTQ3YmEtYmVhMS04ZWMxOTM2Yjk1NTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDp0aGVhdGVyLW9wZXJhdGlvbnMifQ.DKzkbgRseQWIkSv6q7QHOlS3z9f8Y0NFNWFQsfMnyLBIIq2xsvSfRCAqh2Ps4vUr5xzCqlA16p0llFQs-MVwTvN6StXaAw6MfgAXQcqSOWYmPi5nKRTIyXQtsz5rEBZaz6hKVilek6KhIYCNeCAgpLilVcz7i55qtdHsz1JNUgWd7HqUDPwBLfx4BILY3B3L7myX3EE7WgpxLqqAKQ40mL9TVD2CpOa6Zw1X1ovP5ANTgUVjrDAosDmlL-BKNMsJOmb4LRzMr5OlY6oFRbuSOsZZ_NtI7WOE5zA8yoy18jSOqg33FYwMfFixjvjZRm1XQMV5DYD7JZH_GmrAJksUVw
valid-issuers:
  - https://kubernetes.default.svc.cluster.local
  - kubernetes/serviceaccount
EOF

A Kubernetes login role must also be configured, eg

supctl create strongbox authentication kubernetes theater-ops roles <<EOF
name: theater-ops-role
token-policies:
 - theater-ops-policy
verbose-logging: true
bound-service-account-names:
 - "*"
bound-service-account-namespaces:
 - default
EOF

And the theater-ops-policy must be configured to give the pod access to the secret.

supctl create policy policies <<EOF
name: theater-ops-policy
rest-api:
  rules:
    - path: /v1/*/strongbox/vaults/credentials/secrets/admin
      operations:
        read: allow
    - path: /v1/*/strongbox/vaults/credentials/secrets/oper
      operations:
        read: allow
EOF
"""

The API CA certificate needs to be extracted.

supctl do get-api-ca-cert
cert: |-
  -----BEGIN CERTIFICATE-----
  MIICGzCCAcCgAwIBAgITAIGw58I85P6bWhwIX2Qo8hcHHTAKBggqhkjOPQQDAjBj
  MRgwFgYDVQQDEw9BdmFzc2EgQVBJIHJvb3QxEjAQBgNVBAcTCVN0b2NraG9sbTEL
  MAkGA1UEBhMCU0UxDzANBgNVBAoTBkF2YXNzYTEVMBMGA1UECxMMZGlzdHJpYnV0
  aW9uMCIYDzIwMjIwNDIxMDcxMTMwWhgPMjAyNTA4MzAxMDQ3MzBaMGMxGDAWBgNV
  BAMTD0F2YXNzYSBBUEkgcm9vdDESMBAGA1UEBxMJU3RvY2tob2xtMQswCQYDVQQG
  EwJTRTEPMA0GA1UEChMGQXZhc3NhMRUwEwYDVQQLEwxkaXN0cmlidXRpb24wWTAT
  BgcqhkjOPQIBBggqhkjOPQMBBwNCAAQFdfEMHr6X8fuudI7fI01UtEzbTW0DHLhg
  v9G80P39l8D9FKY8IaRmILkn+wKgO/jCOGDBTXQHICnALCEZqWNUo08wTTAOBgNV
  HQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBATAnBgNVHR8EIDAeMBygGqAY
  hhZodHRwOi8vY3JsLmF2YXNzYS5uZXQvMAoGCCqGSM49BAMCA0kAMEYCIQCf/oBh
  6PSuZ23ODIgtdmOY4tC2UmSP10OY+bikzCqjFAIhAJb6gc/Mb5hKbgtwRJ2BdIPk
  s/ujLEF3Tgc70oUona1Y
  -----END CERTIFICATE-----

Accessing Secrets​

Once both Kubernetes and Avassa has been set up for authentication the avassa-mount-secrets container can be used to access secrets stored in Strongbox.

Directives to the container is provided as annotations in the pod, ie

apiVersion: v1
kind: Pod
metadata:
  name: auth-demo
  annotations:
    avassa.io/api-host: api.theater-operation.avassa.io
    avassa.io/tenant: theater-operation
    avassa.io/service: theater-ops
    avassa.io/role: theater-ops-role
    avassa.io/ca-cert: |-
      -----BEGIN CERTIFICATE-----
      MIICGzCCAcCgAwIBAgITAIGw58I85P6bWhwIX2Qo8hcHHTAKBggqhkjOPQQDAjBj
      MRgwFgYDVQQDEw9BdmFzc2EgQVBJIHJvb3QxEjAQBgNVBAcTCVN0b2NraG9sbTEL
      MAkGA1UEBhMCU0UxDzANBgNVBAoTBkF2YXNzYTEVMBMGA1UECxMMZGlzdHJpYnV0
      aW9uMCIYDzIwMjIwNDIxMDcxMTMwWhgPMjAyNTA4MzAxMDQ3MzBaMGMxGDAWBgNV
      BAMTD0F2YXNzYSBBUEkgcm9vdDESMBAGA1UEBxMJU3RvY2tob2xtMQswCQYDVQQG
      EwJTRTEPMA0GA1UEChMGQXZhc3NhMRUwEwYDVQQLEwxkaXN0cmlidXRpb24wWTAT
      BgcqhkjOPQIBBggqhkjOPQMBBwNCAAQFdfEMHr6X8fuudI7fI01UtEzbTW0DHLhg
      v9G80P39l8D9FKY8IaRmILkn+wKgO/jCOGDBTXQHICnALCEZqWNUo08wTTAOBgNV
      HQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBATAnBgNVHR8EIDAeMBygGqAY
      hhZodHRwOi8vY3JsLmF2YXNzYS5uZXQvMAoGCCqGSM49BAMCA0kAMEYCIQCf/oBh
      6PSuZ23ODIgtdmOY4tC2UmSP10OY+bikzCqjFAIhAJb6gc/Mb5hKbgtwRJ2BdIPk
      s/ujLEF3Tgc70oUona1Y
      -----END CERTIFICATE-----

The annotations are made available to the secrets provider container through the Kubernetes downward API as a mounted file system. Some annotations can also be provided through environment variables to allow two instances of the secrets provider to be run at the same time (both as an init container and an application container).

Init Container​

When using the avassa-mount-secrets container as an init container it can pre-populate files and secrets in the Kubernetes Secrets store to prepare an environment for a regular container to run. The avassa-mount-secrets container will populate the secrets from Avassa Strongbox and then exit, allowing an application container to start.

Application Container​

It is also possible to run the avassa-mount-secrets container as an init container where it can periodically update secrets from the Avassa System. By default it will check for changes every 5 minutes.

If an application requires the secrets to be present before it is started, and also requires secrets to be updated over time, then two instances of the avassa-mount-secrets container can be used, one as an init container, and one as an application container. The mode that the container should run as can then be set using environment variables AVASSA_MODE and AVASSA_UPDATE_INTERVAL.

Mounting a Secret as a File​

Secrets can be mounted as a set of files. Each secret is mounted as a file where the key is used as filename, and the value is stored as contents of the file. The avassa-mount-secrets container is instructed to mount a secret in this way through the use of an annotation on the format

avassa.io/vault.<vault-name>.<secret-name>: files

Suppose the following secret is stored in Strongbox in the vault credentials

name: admin
data:
  password: verysecret
  user: admin@avassa.io

then the following annotation will result in two files being created: user and password.

avassa.io/vault.credentials.admin: files

It is also possible to mount the value of a key in a secret in a named file using an annotation on the format:

avassa.io/vault.<vault-name>.<secret-name>.<key-id>: <file-name>

For example

avassa.io/vault.credentials.admin.user: user.txt

Finally, an entire secret can be mounted as a single file with the secret dict rendered in json format. This is achieved with an annotation on the format:

avassa.io/vault.<vault-name>.<secret-name>: json/<file-name>

For example:

avassa.io/vault.credentials.admin: json/credentials.json

Populating Kubernetes Secret Store​

An Avassa Strongbox secret can be mapped into the Kubernetes Secrets store. This is achieved by storing a strongbox-map value in the Kubernetes store. The map indicates that a key should be created in the store taking the value from a key in some Strongbox vault secret. The format is

 <secret-key>: <vault-name>/<secret-name>/<key-id>

For example:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  strongbox-map: |-
    username: credentials/admin/user
    password: credentials/admin/password

An annotation is also needed to instruct the avassa-mount-secrets container to populate the map in some given Kubernetes secret on the form

avassa.io/k8s-secrets: <name of kubernetes secret>

For example

avassa.io/k8s-secrets: db-credentials

Example​

Putting it all together in a pod description

Mount as files​

Mount secrets as files in a Kubernetes Init container.

apiVersion: v1
kind: Pod
metadata:
  name: auth-demo
  annotations:
    avassa.io/api-host: api.theater-operation.avassa.io
    avassa.io/tenant: theater-operation
    avassa.io/service: theater-ops
    avassa.io/role: theater-ops-role
    # create a file for each entry in the secrets map
    avassa.io/vault.credentials.admin: files
    # store the secrets map in the file credentials.json
    avassa.io/vault.credentials.oper: json/credentials.json
    # store a single entry in the secrets map in the file user.txt
    avassa.io/vault.credentials.oper.user: user.txt
spec:
  serviceAccountName: nginx-sa
  volumes:
  - name: shared-data
    emptyDir: {}
  - name: podinfo
    downwardAPI:
      items:
        - path: annotations
          fieldRef:
            fieldPath: metadata.annotations
  - name: avassa-secrets
    emptyDir:
      medium: Memory
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
    - name: avassa-secrets
      mountPath: /avassa/secrets
  initContainers:
  - name: init-avassa-secrets
    image: avassa-mount-secrets:0.1
    imagePullPolicy: Never
    volumeMounts:
    - name: podinfo
      mountPath: /avassa/podinfo
    - name: avassa-secrets
      mountPath: /avassa/secrets
  hostNetwork: false
  dnsPolicy: Default

Populate Kubernetes Secret​

Populate a Kubernetes Secret in a Kubernetes Init container.

apiVersion: v1
kind: Pod
metadata:
  name: auth-demo-2
  annotations:
    avassa.io/api-host: api.theater-operation.avassa.io
    avassa.io/tenant: theater-operation
    avassa.io/service: theater-ops
    avassa.io/role: theater-ops-role
    avassa.io/ca-cert: |-
      -----BEGIN CERTIFICATE-----
      MIICGDCCAb+gAwIBAgISTk4CwNh068Jmvz45tgEmXsk7MAoGCCqGSM49BAMCMGMx
      GDAWBgNVBAMTD0F2YXNzYSBBUEkgcm9vdDESMBAGA1UEBxMJU3RvY2tob2xtMQsw
      CQYDVQQGEwJTRTEPMA0GA1UEChMGQXZhc3NhMRUwEwYDVQQLEwxkaXN0cmlidXRp
      b24wIhgPMjAyMjA0MjEwNzI4MTBaGA8yMDI1MDgzMDExMDQxMFowYzEYMBYGA1UE
      AxMPQXZhc3NhIEFQSSByb290MRIwEAYDVQQHEwlTdG9ja2hvbG0xCzAJBgNVBAYT
      AlNFMQ8wDQYDVQQKEwZBdmFzc2ExFTATBgNVBAsTDGRpc3RyaWJ1dGlvbjBZMBMG
      ByqGSM49AgEGCCqGSM49AwEHA0IABJ6ctcD03qg7jC273SXMi6D4AzLSe/zEEYd6
      WnDxgLzpXJTAbchIURSTLGDDMA4AmbjHwcrhXfn1u4iL1sSbhP6jTzBNMA4GA1Ud
      DwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEBMCcGA1UdHwQgMB4wHKAaoBiG
      Fmh0dHA6Ly9jcmwuYXZhc3NhLm5ldC8wCgYIKoZIzj0EAwIDRwAwRAIgYY2gpw8f
      w5y+6PW92w+lfKgwz5lzniP3IPbZqFi3qGcCIDEU+8lYtp/EG53QFxjqT1cco5D0
      9XKqQ/8Vp/8BEg+i
      -----END CERTIFICATE-----
    avassa.io/k8s-secrets: db-credentials
spec:
  serviceAccountName: nginx-sa
  volumes:
  - name: podinfo
    downwardAPI:
      items:
        - path: annotations
          fieldRef:
            fieldPath: metadata.annotations
  containers:
  - name: alpine
    image: alpine
    command:
      - /bin/sh
      - "-c"
      - "sleep 60m"
    env:
      - name: DB_USERNAME
        valueFrom:
          secretKeyRef:
            name: db-credentials
            key: username
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: db-credentials
            key: password
  initContainers:
  - name: init-avassa-secrets
    image: avassa-mount-secrets:0.1
    imagePullPolicy: Never
    volumeMounts:
    - name: podinfo
      mountPath: /avassa/podinfo
  hostNetwork: false
  dnsPolicy: Default

Continuous Update​

Pre-populate the files user.txt and password.txt in an init container and continue to periodically update the files if the secrets are changed in Strongbox.

apiVersion: v1
kind: Pod
metadata:
  name: auth-demo-3
  annotations:
    avassa.io/api-host: api.theater-operation.avassa.io
    avassa.io/tenant: theater-operation
    avassa.io/service: theater-ops
    avassa.io/role: theater-ops-role
    avassa.io/ca-cert: |-
      -----BEGIN CERTIFICATE-----
      MIICGDCCAb+gAwIBAgISTk4CwNh068Jmvz45tgEmXsk7MAoGCCqGSM49BAMCMGMx
      GDAWBgNVBAMTD0F2YXNzYSBBUEkgcm9vdDESMBAGA1UEBxMJU3RvY2tob2xtMQsw
      CQYDVQQGEwJTRTEPMA0GA1UEChMGQXZhc3NhMRUwEwYDVQQLEwxkaXN0cmlidXRp
      b24wIhgPMjAyMjA0MjEwNzI4MTBaGA8yMDI1MDgzMDExMDQxMFowYzEYMBYGA1UE
      AxMPQXZhc3NhIEFQSSByb290MRIwEAYDVQQHEwlTdG9ja2hvbG0xCzAJBgNVBAYT
      AlNFMQ8wDQYDVQQKEwZBdmFzc2ExFTATBgNVBAsTDGRpc3RyaWJ1dGlvbjBZMBMG
      ByqGSM49AgEGCCqGSM49AwEHA0IABJ6ctcD03qg7jC273SXMi6D4AzLSe/zEEYd6
      WnDxgLzpXJTAbchIURSTLGDDMA4AmbjHwcrhXfn1u4iL1sSbhP6jTzBNMA4GA1Ud
      DwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEBMCcGA1UdHwQgMB4wHKAaoBiG
      Fmh0dHA6Ly9jcmwuYXZhc3NhLm5ldC8wCgYIKoZIzj0EAwIDRwAwRAIgYY2gpw8f
      w5y+6PW92w+lfKgwz5lzniP3IPbZqFi3qGcCIDEU+8lYtp/EG53QFxjqT1cco5D0
      9XKqQ/8Vp/8BEg+i
      -----END CERTIFICATE-----
    avassa.io/vault.credentials.oper.user: user.txt
    avassa.io/vault.credentials.oper.password: password.txt
spec:
  serviceAccountName: nginx-sa
  volumes:
  - name: avassa-secrets
    emptyDir:
      medium: Memory
  - name: podinfo
    downwardAPI:
      items:
        - path: annotations
          fieldRef:
            fieldPath: metadata.annotations
  containers:
  - name: alpine
    image: alpine
    command:
      - /bin/sh
      - "-c"
      - "sleep 60m"
    volumeMounts:
    - name: avassa-secrets
      mountPath: /avassa/secrets
  - name: avassa-secrets
    image: avassa-mount-secrets:0.1
    imagePullPolicy: Never
    volumeMounts:
    - name: podinfo
      mountPath: /avassa/podinfo
    - name: avassa-secrets
      mountPath: /avassa/secrets
    env:
      - name: AVASSA_MODE
        value: application
      - name: AVASSA_UPDATE_INTERVAL
        value: 10s
  initContainers:
  - name: init-avassa-secrets
    image: avassa-mount-secrets:0.1
    imagePullPolicy: Never
    volumeMounts:
    - name: podinfo
      mountPath: /avassa/podinfo
    - name: avassa-secrets
      mountPath: /avassa/secrets
    env:
      - name: AVASSA_MODE
        value: init
  hostNetwork: false
  dnsPolicy: Default

Troubleshooting​

The avassa-mount-secrets container logs to stdout. These logs can be inspected for errors using the kubectl command. For example, in a pod called auth-demo the output from the init-avassa-secrets container can be view using the following command

kubctl logs auth-demo -c init-avassa-secrets