Consuming Vault Secrets in Acorn Apps

Oct 12, 2022 by Bill Maxwell
Consuming Vault Secrets in Acorn Apps

Intro

In this post I will discuss one of the many ways to use Hashicorp’s Vault in a Kubernetes environment with Acorn. Vault is a very popular tool used to secure, store, and tightly control access to passwords and other sensitive data. Vault can be used on premise and in the cloud and there are open source, commercially supported, and hosted versions available.

Acorn is a new tool from Acorn Labs that enables deploying applications on Kubernetes without the user needing to know Kubernetes.

To demonstrate how to use Vault and Acorn, we will use Vault’s Kubernetes authentication backend to allow containers to authenticate to Vault with a Kubernetes service account token. We will also use the Vault Agent Injection service to write the secret values to a file inside containers launched by Acorn.

Pre-reqs

In order to follow along in this post, you will need to have:

  • A Kubernetes cluster 1.24+
  • jq CLI tool
  • A Vault server accessible to/from the Kubernetes cluster
  • vault CLI
  • helm CLI
  • acorn CLI

Fully configuring a production Vault server is outside the scope of this post. For the examples below, I’m using a Vault server running in dev mode on my machine.

Vault secret data

To get started, let’s write our first bit of secret data to our Vault server.

  1. log into Vault

vault login [USER]

  1. write some secret data

vault kv put secret/data/myapp/config username='hammy' password='squirrel'

  1. Check the data
vault kv get -format=json secret/data/myapp/config | jq '.data.data' #{ # "password": "squirrel", # "username": "hammy" #}

Now that we have some sensitive data stored in Vault lets work on getting that to our Acorn application containers. Next we will install the Vault Agent Injection service.

Vault Agent Injection service install

We will use the official Hashicorp Vault Helm chart to install the Vault Agent Injection service. The service watches for containers being deployed with specific annotations and modifies their deployment so the Vault secrets can be written in the container.

First, we will add the Helm repo to our setup.

helm repo add hashicorp https://helm.releases.hashicorp.com # "hashicorp" has been added to your repositories

Then update our helm repositories to ensure we get the latest versions.

helm repo update #Hang tight while we grab the latest from your chart repositories... #...Successfully got an update from the "hashicorp" chart repository #Update Complete. ⎈Happy Helming!

Then launch the Injection pod with the command below. The main argument is the address of the external Vault server. I’m going to use my IP address, but you can also use DNS or if you’d like you could create an external service in Kubernetes. Whichever method you decide, use that value for your server address. I’m also using the acorn namespace, but in a production setting it is best to deploy in its own namespace.

export VAULT_SERVER=192.168.1.119

Install the chart.

helm install -n acorn vault hashicorp/vault \ --set "injector.externalVaultAddr=http://${VAULT_SERVER}:8200"

Now we need to generate the Kubernetes service account token needed for Vault to validate container credentials. To do that, we will create a Kubernetes service-account-token secret

cat > vault-secret.yaml <<EOF apiVersion: v1 kind: Secret metadata: name: vault-token annotations: kubernetes.io/service-account.name: vault type: kubernetes.io/service-account-token EOF

Create the secret in Kubernetes.

kubectl apply -n acorn -f ./vault-secret.yaml #secret/vault-token created

Vault k8s auth backend

Now that we have installed the Vault Agent Injection service, we will setup Vault to allow authentication with tokens from containers in our cluster.

VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-token")).name')

Then we will describe the secret so we can see that a token value is set.

kubectl describe secret $VAULT_HELM_SECRET_NAME

Now lets enable the auth backend on our Vault server.

vault auth enable kubernetes

Once the backend is enabled in Vault we need to provide Vault with the connection information to the cluster. First we need the JWT that allows Vault to validate credentials coming from the containers in the cluster with Kubernetes.

TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)

Then we will need the CA certificate for the cluster so that Vault can have a secure connection to the Kubernetes API server.

KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

Last, Vault will need the Kubernetes API server endpoint to communicate with.

KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

Now lets configure Vault to

vault write auth/kubernetes/config \ token_reviewer_jwt="$TOKEN_REVIEW_JWT" \ kubernetes_host="$KUBE_HOST" \ kubernetes_ca_cert="$KUBE_CA_CERT" \ issuer="https://kubernetes.default.svc.cluster.local"

::To determine the correct value for issuer name for your cluster you can try::

Configure Vault policy

Earlier in the post we stored data in Vault at this path secret/data/acornwebapp/config. Now we are going to want our containers to access that data. To allow access to the data from our containers we need to create a Vault policy for the path like so.

vault policy write myapp - <<EOF path "secret/data/myapp/config" { capabilities = ["read"] } EOF

Now lets grant assign the policy to a role for our containers to use.

vault write auth/kubernetes/role/myapp \ bound_service_account_names=acorn \ bound_service_account_namespaces=myapp \ policies=myapp \ ttl=24h

This allows the acorn service account, the default for all apps, from the myapp namespace to access the paths allowed in the myapp policy. The namespace myapp will be specified later as the --target-namespace when we start our Acorn app.

Acorn run

Now that we have Vault setup and some secret data stored, let’s use those secrets in some of our Acorn containers. To illustrate we will use a very simple Acornfile.

containers: app: { image: "alpine" }

The Vault Agent Injector service we installed previously looks for annotations on containers to know if secrets need to be injected or not. The primary annotations are:

  • vault.hashicorp.com/agent-inject
  • vault.hashicorp.com/role
  • vault.hashicorp.com/agent-inject-secret-FILEPATH.txt

Let’s run our app with our Acornfile from above.

acorn run --annotation containers:vault.hashicorp.com/agent-inject="true" \ --annotation containers:vault.hashicorp.com/role="myapp" \ --annotation containers:vault.hashicorp.com/agent-inject-secret-credentials.txt="secret/data/myapp/config" \ --target-namespace myapp .

Now when we exec into our container we can see the values printed in the file /vault/secrets/credentials.txt. Notice that because we changed FILEPATH to credentials our data is in the credentials.txt file.

acorn exec -it [APP] cat /vault/secrets/credentials.txt # data: map[password:squirrel username:hammy] # metadata: map[created_time:2022-10-05T16:42:54.29411Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

As you can see in the output our secret data is written into the container. You can also use the annotation vault.hashicorp.com/agent-inject-template-FILEPATH.txt to apply formatting to the data written in the file within the container.

Wrapping up

In this post we looked at one way you can use Vault with Acorn in your Kubernetes clusters. This configuration allows Acorn applications to access sensitive data without any direct knowledge of Vault. There is a lot of configuration to ensure access to the sensitive data is tightly controlled, but once configured it is easy for applications to access the data. In future posts, we will explore using the Vault CSI driver within the cluster to access sensitive information from the Vault server. Stay tuned.