Harpocrates - How to easily pull secrets from your Vault

When using HashiCorp Vault to store your secret, you know that it can sometimes be a pain to get your secrets from Vault to your application running in Kubernetes.

Do you save the secrets in your git repository (spoiler - you don't, that's how you end up paying for Bitcoin mining)?

Do you create a Kubernetes secret based upon the secrets from Vault, risking that you forget to update it in either places?

Do you utilize some kind of sync service to keep your Kubernetes secrets in sync with your Vault secrets?

For me, all of above, wasn't really a great solution as it meant the Vault secrets had to be saved outside of Vault.

In this post, I will try to explain which solution we came up with.

The Problem

Let's say you need to get the url from secret1 and username and password from secret2 and everything from secret3, you would have to run these 4 commands to and then somehow combine it into one file so that your application can read it.

1vault kv get -field=url secret/secret1 > secret1
2vault kv get -field=username secret/secret2 > secret2
3vault kv get -field=password secret/secret2 >> secret2
4vault kv get secret/secret3 > secret3

If your application need to have all the secrets in one file, you would then have to combine them somehow.

And what if your application expects that the secret keys where prefixed with something? Then you would also need to figure out how to do that.

Our Solution

We made Harpocrates. Harpocrates is a small application that can be used to pull secrets from HashiCorp Vault.

With Harpocrates, it is all in one command:

1harpocrates -f /path/to/file.yaml

The file will contain some information about which secrets you wanna fetch:

 1format: json
 2output: "/secrets"
 3prefix: PREFIX_
 4secrets:
 5  - secret/data/secret1:
 6      keys:
 7       - url
 8  - secret/data/secret2:
 9      keys:
10        - username
11        - password
12  - secret/data/secret2

The output of the above would be something like:

1{
2  "PREFIX_url": "http://google.com",
3  "PREFIX_username:": "root",
4  "PREFIX_password": "toor",
5  "PREFIX_version": 1,
6  "PREFIX_debug_level": "DEBUG"
7}

You can customize on almost every level:

Setting a prefix at a global level, changing that for a specific secret or even on a key level.

We tried to create an example with most of the options you have.

So far we support the following formats:

• JSON, which is simple key-values.

  1{
  2  "KEY": "value",
  3  "FOO": "bar"
  4}
  

• source ready env file e.g.

  1export KEY=value
  2export FOO=bar
  

• Raw key values.

  1KEY=value
  2FOO=bar
  

• Raw value in a separate file.

  1value
  

Usage

We designed Harpocrates to be used as an init- or sidecar container in Kubernetes. An example of how to use it, can be found here

To make it easier to add Harpocrates to your Kubernetes deployments, we have created an Orb for CircleCI

As part of your CircleCI config, you can add our orb by adding this:

1orbs:
2  secret-injector: bestsellerit/secret-injector@1.3.2

Then just before you normally would apply your deployment to kubernetes, you call this step:

1- secret-injector/inject:
2    app-name: alfeios
3    deploy-file: ./deployment.yml
4    secret-file: secrets.yaml
5    container-name: aa
6    deploy-type: Deployment

Read the full documentation on GitHub.

Next step

We have been running it internally for about a year now and we are ready for the world to know about it! So what's next for Harpocrates? You tell us!

Take it for a spin and we'll be looking forward to your feedback :)

About the author

Lasse Gaardsholt

My name is Lasse Gaardsholt, I work as a Systems Engineer in a engineering team at BESTSELLER.

I write Go, put stuff in Kubernetes and solve problems. If you wanna see some of the stuff I have made, please look at my GitHub page.