Every application needs configuration data like database passwords, AWS access keys, and social app IDs before it can run.
What’s the easiest way to do it? Hardcode those values, push them to Git and everyone who has a copy of the source code can run the application. The only problem is, you don’t want the whole world to know your usernames and passwords.
They need to be kept secret. They need to be kept safe.

Point III. of the 12-factor methodology calls for a strict separation of configuration from code.

This means removing your database configuration and similar files from version control and copying them directly to your server and CI. That, I don’t need to tell you, is a tedious job. Additionally, this approach is highly prone to errors and makes it harder to collaborate with other developers. You and all your collaborators need to remember to update configuration files everywhere.

Having multiple environments (such as production, staging, and acceptance) with different configuration files makes this even more difficult.

Heroku solved this problem through its config setup. All configuration files can be set with heroku config:set independently for each environment.

If you have your own servers like us, Heroku’s strategy of using environment variables won’t work as easily. This is where Vault comes into play. We use HashiCorp’s Vault to store and retrieve our secrets.

We use Rails for web development. Luckily, there are a couple of gems that handle the setup of application environment variables. We agreed on using Figaro.

The easiest and least error-prone way to do it is moving all your secrets to the config/secrets.yml file. There you can list all application secrets and make them use environment variables. Here’s an example:

PROTIP: By using Figaro’s bang methods we make sure all environment variables are set. If an environment variable is not set it will throw a Figaro::MissingKey error.

With the secrets.yml file all ready, we created three Figaro config files: config/application.yml, config/application.staging.yml and config/application.production.yml and remembered to add those files to .gitignore. Here’s an example of a config file:

And here’s an example of how to actually use those secrets:

Now that our application is all 12-factored up, we need a way to share environment variables between developers and servers. Remember, we added application.*.yml files to gitignore so they don’t end up in the repository. No one has access to those files except for the person who wrote them. We are using Vault for sharing those files.

Here’s a summary of what Vault does:

Basically, it is an all-in-one solution for storing your critical information somewhere safe.

After setting up Vault on a separate server, we configured it to use consul.io as a backend. For authentication backends we are using github and app-id methods. A simple file system is used for an audit backend. You can look into Vault’s documentation for more information.

As we have hundreds of projects under our belt, naming conventions are a must if we want our developers to know where the secrets are stashed. We agreed on rails/#{git_repository_name}/#{environment} for a path to store secrets within Vault. We are using the Git repository name here as a part of the path because that isn’t something that ever changes, so everyone knows the location of the secrets.

Next, a policy needs to be created to give someone writing and/or reading privileges on a specific Vault path. Here’s an example of such a policy:

As Vault is easily integrated with GitHub, we are using GitHub teams to apply different application policies to users. Simply create a new GitHub team, add all needed users, and link the Vault policy to that team. This has an added benefit of only having to remove someone from your GitHub organization in order to revoke their access to all secrets. This can come in handy when someone leaves the company.

Except to users, we also need to grant access to servers and CI clients. We use what Vault calls app-id authentication. App-id authentication uses two strong keys to authenticate a Vault server. We give our machines read-only access to secrets within Vault.

As I mentioned before, Vault has no web interface, and its command line tool has a steep learning curve. We mitigated that problem by creating our own secrets gem. It uses the official Vault Ruby gem and is built around the requirements described above.

It has a couple of simple commands:

This will create a .secrets file with the project configuration. The command will ask for everything you don’t supply via options.

Here’s an example of a .secrets file:

And if you set up your environment variables correctly you can push and pull secrets:

This can also be set up on a server, so your deployment scripts pull secrets on every deployment. We use mina for our deployment purposes, and we created a mina-secrets plugin which simplifies our deployment with secrets.

So there it is, our approach to building a production-grade, scalable system for managing secrets. Using Vault makes the system secure. Using Github for the authentication makes it convenient for managing access. And using the secrets gem makes sharing secrets quick and dead simple for developers. Don’t sacrifice security for the sake of convenience any more.

Hope you enjoy using it.