Serverless Secrets with Google Cloud Run and Hashicorp Vault

calendar Jan 12, 2021 |  6 minutes  |  Brett Wright howto cloudrun serverless secrets hashicorp vault google go golang
Share on: linkedin copy

Want to run serverless applications and use your secrets from Hashicorp Vault?
Google's Cloud Run and Cloud Functions are brilliant for running workloads without having to mange any infrastructure. But more often than not, the serverless application needs to communicate with other services. This will most likely require credentials, certificates or other critical information to do so.

Imagine using Google Cloud Run or Cloud Functions and injecting secrets from Vault at runtime. This post will go through in detail how it is possible using the GCP Auth method in Vault which allows a Google Service Account to authenticate to Vault.

Assumptions

This article assumes that you have some general knowledge about Vault and that you known how to create secrets and polices in Vault. That you have a GCP Artifact Registry setup

How it works

Let's say we have a Go application called test-app, which has been put into a container and deployed to Cloud Run. Our test-app needs access to the secrets in the Vault path "secret/data/foo". With the help of a little Go module, go-vault, we have created, we can easily get our secrets into the container at runtime using an init function as in the example below.

 1package main
 2
 3import (
 4	"fmt"
 5	"log"
 6	"net/http"
 7
 8	"github.com/BESTSELLER/go-vault/gcpss"
 9)
10
11var secret string
12
13func init() {
14	vaultAddr := "https://vault.yourdomain.com"
15	vaultSecret := "secret/data/foo"
16	vaultRole := "test-app"
17	var err error
18
19	secret, err = gcpss.FetchVaultSecret(vaultAddr, vaultSecret, vaultRole)
20	if err != nil {
21		fmt.Println(err)
22	}
23
24}
25
26func main() {
27	fmt.Println(secret)
28}

So how does the module do this?

  1. The client sends a get request to the GCP metadata server and returns a signed JWT.
  2. The client sends a post request with the JWT and the role name to the Vault.
  3. Vault then verify IAM account with Google IAM.
  4. Vault then returns login details which includes a Vault token.
  5. The client can then use the Vault token to retrieve secrets from Vault

Image pulled from Hashicorp.

Configuration

Before we can pull Vault secrets directly from our serverless applications we need to a bit on configuration in both Vault and GCP.

Google Configuration

We need to create two service accounts within GCP. One service account so Vault can query Google IAM and the second service account that will be used authenticate against Vault and to retrieve secrets.

  • Lets begin with creating the service account that will be used by Vault to query Google IAM.
1gcloud iam service-accounts create vault-auth \
2    --description="Vault gcp auth"
  • We now need to give the service account, we just created, permission to generate service accounts keys within Google.
1PROJECT_ID="your-project-id-here"
2gcloud projects add-iam-policy-binding $PROJECT_ID \
3    --member="serviceAccount:Vault-auth@${PROJECT_ID}.iam.gserviceaccount.com" \
4    --role="roles/iam.serviceAccountKeyAdmin"
  • Create and download service account key for this account, which will be used in a later step.
1gcloud iam service-accounts keys create ./key.json \
2  --iam-account "Vault-auth@${PROJECT_ID}.iam.gserviceaccount.com"
  • We now can create the second service account. This will be used to run our Cloud Run application and be used to authenticate and pull secrets from Vault.
1gcloud iam service-accounts create test-app \
2    --description="This account is used to cloud run test-app and login to Vault"
  1. The service account need the roles/run.invoker permission to run Cloud Run.
1gcloud projects add-iam-policy-binding $PROJECT_ID \
2    --member="serviceAccount:test-app@${PROJECT_ID}.iam.gserviceaccount.com" \
3    --role="roles/run.invoker"

Vault GCP Auth Configuration

Now that we have nailed the Google service accounts and roles configuration, we need to connect Vault to our new GCP service account.

  • Firstly we need to enable the Google auth method within Vault.
1vault auth enabled gcp
  • Next we configure the GCP auth with the credentials we created earlier. This enables Vault to query Google IAM to ensure that this is valid service account.
1vault write auth/gcp/config \
2    credentials=@./key.json
  • Now let's create a role in Vault called test-app and assign the polices to this role.
1vault write auth/gcp/role/test-app \
2    type="iam" \
3    policies="test-app" \
4    max_jwt_exp="3600"
5    bound_service_accounts="test-app@${PROJECT_ID}.iam.gserviceaccount.com"

Build, Deploy and Run time

Now that we have created the necessary service accounts in Google and configured Vault to access the our GCP project through our test-app role, we can move on to building, deploying and running our test-app in Cloud Run. In the interest of saving time, we have already created a little test-app written in Go. This will display the secret as json. This in for testing purposes only and please don't do this in production.

  • Lets go head and clone the repo.
1git clone https://github.com/BESTSELLER/blog-google-serverless-secrets
  • Once we have clone the repo we can build the docker image and push this to the Google container registry, as Cloud Run can only deploy docker images from Google Sources.
1DOCKER_REGISTRY_URL=europe-docker.pkg.dev/${PROJECT_ID}
2
3gcloud auth configure-docker europe-docker.pkg.dev
4
5docker build -t $DOCKER_REGISTRY_URL/docker/test-app:latest .
6docker push $DOCKER_REGISTRY_URL/docker/test-app:latest
  • We can now deploy the docker image to Google Cloud Run.
 1VAULT_ADDR="https://vault.yourdomain.com"
 2VAULT_ROLE="test-app"
 3VAULT_SECRET="secret/data/foo"
 4
 5gcloud run deploy test-app \
 6    --image=$DOCKER_REGISTRY_URL/docker/test-app:latest \
 7    --allow-unauthenticated \
 8    --max-instances=100 \
 9    --port=8080 \
10    --platform=managed \
11    --region=europe-west4 \
12    --project=${PROJECT_ID} \
13    --set-env-vars VAULT_ADDR=${VAULT_ADDR},VAULT_ROLE=${VAULT_ROLE},VAULT_SECRET=${VAULT_SECRET} \
14    --service-account=test-app@${PROJECT_ID}.iam.gserviceaccount.com
  • Once we have the docker image running in Cloud Run we can query this using the below command.
1curl $(gcloud run services describe test-app --platform managed --project $PROJECT_ID --region europe-west4 --format json | jq -r '.status.url' )
  • If you get a 403 error message when trying to Cloud Run app run the below command. This will allow us to access the Cloud Run application public, you should only have do this the first time you deploy your app.
1gcloud run services add-iam-policy-binding test-app \
2    --member="allUsers" \
3    --role="roles/run.invoker" --platform managed --project $PROJECT_ID --region

Troubleshooting

  • If you are have trouble with the above you can troubleshoot this by spinning up a gce instance running as test-app service account.
 1gcloud beta compute --project=${PROJECT_ID} instances create cloud-run-test \
 2    --zone=europe-west4-a \
 3    --machine-type=e2-medium \ 
 4    --subnet=default \ 
 5    --network-tier=PREMIUM \ 
 6    --maintenance-policy=MIGRATE \ 
 7    --service-account=test-app@${PROJECT_ID}.iam.gserviceaccount.com \ 
 8    --scopes=https://www.googleapis.com/auth/cloud-platform \
 9    --image=debian-10-buster-v20201216 \ 
10    --image-project=debian-cloud \ 
11    --boot-disk-size=10GB \ 
12    --boot-disk-type=pd-standard \
13    --boot-disk-device-name=cloud-run-test \ 
14    --no-shielded-secure-boot \
15    --shielded-vtpm \
16    --shielded-integrity-monitoring  \
17    --reservation-affinity=any
  • You can then ssh into the newly created gce instance and run though the below script.
 1#!/bin/sh
 2
 3VAULT_ADDR="https://vault.yourdomain.com"
 4VAULT_ROLE="test-app"
 5VAULT_SECRET="test/data/foo"
 6
 7# Gets google service JWT via the gcp metadata server
 8JWT=$(curl --header "Metadata-Flavor: Google" --get  "http://metadata/computeMetadata/v1/instance/service-accounts/default/identity" --data-urlencode "audience=http://vault/${VAULT_ROLE}" --data-urlencode "format=full")
 9echo "JWT: " $JWT
10
11# Uses the JWT to a vault token
12TOKEN=$(curl --request POST --data '{"role":"'${VAULT_ROLE}'", "jwt": "'$JWT'"}' ${VAULT_ADDR}/v1/auth/gcp/login | jq -r '.auth.client_token')
13echo "Token: " $TOKEN
14
15# Gets Vault secret 
16curl \
17    --header "X-Vault-Token: $TOKEN" \
18    "${VAULT_ADDR}/v1/${VAULT_SECRET}"

Final Words

Cloud Run have a first class integration with Googles Secret Manager, but as we are already using Vault, we would prefer not to manage our secrets multiple places. With a bit of configuration and an init function we are now able to keep our secrets in Vault. If you, like us, are using Vault i would suggest that you try it out. And as always we are open for pull request or other feedback.

https://www.vaultproject.io/docs/auth/gcp
https://www.youtube.com/watch?v=s_f0vV6Cmas&ab_channel=GoogleCloudPlatform
https://github.com/sethvargo/secrets-in-serverless/tree/master/hashicorp-vault

About the author

Brett Wright

I'm an Australian living and working in Aarhus Denmark. Currently working for BESTSELLER written infrastructure as code, golang and automating everything.