Every application needs configuration data like database passwords, AWS access keys, and social app IDs before it can run. What’s the easiest way to do it? Hardcode those values, push them to Git and everyone who has a copy of the source code can run the application. The only problem is, you don’t want the whole world to know your usernames and passwords. They need to be kept secret. They need to be kept safe.
Point III. of the 12-factor methodology calls for a strict separation of configuration from code.
This means removing your database configuration and similar files from version control and copying them directly to your server and CI. That, I don’t need to tell you, is a tedious job. Additionally, this approach is highly prone to errors and makes it harder to collaborate with other developers. You and all your collaborators need to remember to update configuration files everywhere.
Having multiple environments (such as production, staging, and acceptance) with different configuration files makes this even more difficult.
If you have your own servers like us, Heroku’s strategy of using environment variables won’t work as easily. This is where Vault comes into play. We use HashiCorp’s Vault to store and retrieve our secrets.
12-factoring it up
We use Rails for web development. Luckily, there are a couple of gems that handle the setup of application environment variables. We agreed on using Figaro.
The easiest and least error-prone way to do it is moving all your secrets to the config/secrets.yml file. There you can list all application secrets and make them use environment variables. Here’s an example:
# config/secrets.yml
default: &default
secret_key_base: <%= Figaro.env.secret_key_base! %>
database_name: <%= Figaro.env.database_database! %>
database_username: <%= Figaro.env.database_username! %>
database_password: <%= Figaro.env.database_password! %>
database_host: <%= Figaro.env.database_host! %>
bugsnag_api_key: <%= Figaro.env.bugsnag_api_key! %>
devise_secret_key: <%= Figaro.env.devise_secret_key! %>
development:
<<: *default
test:
<<: *default
staging:
<<: *default
production:
<<: *default
aws_access_key_id: <%= Figaro.env.aws_access_key_id! %>
aws_secret_access_key: <%= Figaro.env.aws_secret_access_key! %>
aws_region: <%= Figaro.env.aws_region! %>
aws_bucket: <%= Figaro.env.aws_bucket! %>
PROTIP: By using Figaro’s bang methods we make sure all environment variables are set. If an environment variable is not set it will throw a Figaro::MissingKey error.
With the secrets.yml file all ready, we created three Figaro config files: config/application.yml, config/application.staging.yml and config/application.production.yml and remembered to add those files to .gitignore. Here’s an example of a config file:
# config/application.yml
secret_key_base: 05d822712453f3433298f3...
devise_secret_key: 682a7bd0fefc30d2fda448062ccd828d3f13...
database_username: postgres
database_password:
database_host: localhost
bugsnag_api_key: ’5780d02c163...’
development:
database_name: application_dev
test:
database_name: application_test
And here’s an example of how to actually use those secrets:
# config/database.yml
default: &default
adapter: postgresql
encoding: unicode
pool: 5
database: <%= Rails.application.secrets.database_name %>
username: <%= Rails.application.secrets.database_username %>
password: <%= Rails.application.secrets.database_password %>
host: <%= Rails.application.secrets.database_host %>
development:
<<: *default
test:
<<: *default
production:
<<: *default
staging:
<<: *default
Sharing secrets
Now that our application is all 12-factored up, we need a way to share environment variables between developers and servers. Remember, we added application.*.yml files to gitignore so they don’t end up in the repository. No one has access to those files except for the person who wrote them. We are using Vault for sharing those files.
Here’s a summary of what Vault does:
- Vault secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing.
- Vault handles leasing, key revocation, key rolling, and auditing.
- Vault presents a unified API to access multiple backends: HSMs, AWS IAM, SQL databases, raw key/value, and more.
Basically, it is an all-in-one solution for storing your critical information somewhere safe.
After setting up Vault on a separate server, we configured it to use consul.io as a backend. For authentication backends we are using github and app-id methods. A simple file system is used for an audit backend. You can look into Vault’s documentation for more information.
As we have hundreds of projects under our belt, naming conventions are a must if we want our developers to know where the secrets are stashed. We agreed on rails/#{git_repository_name}/#{environment} for a path to store secrets within Vault. We are using the Git repository name here as a part of the path because that isn’t something that ever changes, so everyone knows the location of the secrets.
Next, a policy needs to be created to give someone writing and/or reading privileges on a specific Vault path. Here’s an example of such a policy:
path "rails/application/*" {
policy = "write"
}
As Vault is easily integrated with GitHub, we are using GitHub teams to apply different application policies to users. Simply create a new GitHub team, add all needed users, and link the Vault policy to that team. This has an added benefit of only having to remove someone from your GitHub organization in order to revoke their access to all secrets. This can come in handy when someone leaves the company.
Except to users, we also need to grant access to servers and CI clients. We use what Vault calls app-id authentication. App-id authentication uses two strong keys to authenticate a Vault server. We give our machines read-only access to secrets within Vault.
Real world usage
As I mentioned before, Vault has no web interface, and its command line tool has a steep learning curve. We mitigated that problem by creating our own secrets gem. It uses the official Vault Ruby gem and is built around the requirements described above.
It has a couple of simple commands:
$ secrets init
This will create a .secrets file with the project configuration. The command will ask for everything you don’t supply via options.
Here’s an example of a .secrets file:
# file where your secrets are kept depending on your environment gem
:secrets_file: config/application.yml
# vault ’storage_key’ where your secrets will be kept
:secrets_storage_key: rails/my_project/
And if you set up your environment variables correctly you can push and pull secrets:
$ secrets push
$ secrets pull
This can also be set up on a server, so your deployment scripts pull secrets on every deployment. We use mina for our deployment purposes, and we created a mina-secrets plugin which simplifies our deployment with secrets.
Conclusion
So there it is, our approach to building a production-grade, scalable system for managing secrets. Using Vault makes the system secure. Using Github for the authentication makes it convenient for managing access. And using the secrets gem makes sharing secrets quick and dead simple for developers. Don’t sacrifice security for the sake of convenience any more.
Hope you enjoy using it.
