Have you ever noticed how there are no awards for backend development? Sure, there are “web” awards, but they mostly focus on design and the visual aspects (e.g. Awwwards, the Webby Awards). Mobile has many awards such as the Mobile Industry Awards, even HR receives awards (the HR Excellence Awards).

No awards for backend, though. It’s as if the world thinks that what we do is easy and we don’t need any special knowledge or skills to write backend applications.

It is a job that nobody notices when it’s done right, but when it’s not, everything crumbles. Sort of like the Night Watch in Game of Thrones. If you are interested in pursuing this not very glamorous, but badass role, read on to find out what it entails.

There are a multitude of programming languages available for backend development. Most of them have more than one framework for building a web application. From Ruby’s Ruby on Rails and Python’s Django to C#’s ASP.NET and Java’s Spring, the list goes on and on. Sorry if I didn’t mention your favorite framework, this of course barely scratches the surface.

If you’re only starting out, it’s best to choose a programming language you are most familiar with because all frameworks work in a similar fashion, some with more and some with less magic in them. However, they all have one thing in common – they have to work within the confines of the web’s request/response infrastructure.

The state of your application will need to be stored for long-term storage and retrieval. For that, you will need some kind of a database. You can use a relational database like MySql or Postgresql or a NOSQL like MongoDB or Redis, depending on the application’s needs.

In my experience, we opt for a relational database 90 percent of the time because we end up needing the relational connections.

It’s possible to use other types of databases, too, such as the Graph database or the database for event sourcing.

With the basic tools for building a web application covered, you need to figure out what kind of architecture you are going to use. The two currently prevailing architectures are Monolithic and Microservices.

In a monolithic architecture, all the application’s functionality is coded in one big repository. Whenever you make a change you can potentially bring down the whole application, but the changes themselves are easier to make.

On the other hand, a microservices application is broken down into multiple smaller services that (should) work independently of one another. Changes in one service won’t affect other services, but keeping them all in sync becomes a problem once they start to stack up.

At some point, your application will need to respond to requests coming from the world wide web. You need to decide how it will respond, with an outdated XML or a JSON.

When you use JSON for your response schema, you will need to structure it so that all the responses follow the same structure. Whether you choose GraphQL, JSON:API spec or another schema, is completely up to you. There is no correct answer here as long as you respond with JSON. It might be smart to learn what a json schema is, too.

Since you want your application to be accessible to the whole Internet, you need to deploy it somewhere. There are plenty of options, from buying your own barebone machine you can customize and take care of, through paid services like AWS or Digital Ocean, which maintain the hardware for you, to SaaS services like Heroku where you can deploy an application in one command line.

I should probably note that you will also need to learn about Kubernetes and Docker if you want your application to scale well. However, we’ll leave these for now as they’re a topic for a separate blog post.

An admin dashboard is a good way for you to create and edit your content. To build one, though, you will need to learn some frontend languages – HTML, CSS and some Javascript. Fortunately, there are complete frameworks like Bootstrap that will help you build the pages, but you still need to understand how they work in harmony.

At one point, some of the requests will just take too long and users will start complaining. For example, those where you resize uploaded images or export a portion of your data. These types of requests can be migrated to background jobs so that they exit early. Users don’t have to wait for a response and the application responds asynchronously.

You will need to learn about queues and lambdas.

Emails should also be sent in the background, but designing them is a whole other beast to master. I recommend just going with the simplest design you can think of.

You want your application to be available all the time. To achieve this, you need to learn about metrics and how to read graphs to figure out which part of your application is slow. Reading application logs is a science in itself, but yet another one you need to tackle. You will also need to learn how to set up alerts when your application goes offline so you can quickly restart it.

The challenges you’ll face on your backend journey don’t end here. You might find yourself working with third party API’s. I can’t begin to explain my frustration with those. Responding with XML, responding with the wrong HTTP status codes, or just plain not working as the documentation states, it’s just always bad.

Next, there is this pressure of achieving 99.999% availability, which is nearly impossible in practice.

It’s difficult to build a backend “portfolio” because backend is not “showy” or “pretty”. Backend applications don’t win awards. Still, I wouldn’t give up backend development for anything.

First, it’s so much easier to test backend applications than Javascript or mobile apps. Next, you absolutely don’t care or have to worry about what browser the end user is using, and finally, you don’t need to know the latest and “best ever” javascript framework that seems to come out every three months. You also don’t need to make fun of Javascript, that’s purely optional.

Backend is the backbone of the web. The sword in the darkness. Give your backend developer friend a high five when you see them.

Every application needs configuration data like database passwords, AWS access keys, and social app IDs before it can run.
What’s the easiest way to do it? Hardcode those values, push them to Git and everyone who has a copy of the source code can run the application. The only problem is, you don’t want the whole world to know your usernames and passwords.
They need to be kept secret. They need to be kept safe.

Point III. of the 12-factor methodology calls for a strict separation of configuration from code.

This means removing your database configuration and similar files from version control and copying them directly to your server and CI. That, I don’t need to tell you, is a tedious job. Additionally, this approach is highly prone to errors and makes it harder to collaborate with other developers. You and all your collaborators need to remember to update configuration files everywhere.

Having multiple environments (such as production, staging, and acceptance) with different configuration files makes this even more difficult.

Heroku solved this problem through its config setup. All configuration files can be set with heroku config:set independently for each environment.

If you have your own servers like us, Heroku’s strategy of using environment variables won’t work as easily. This is where Vault comes into play. We use HashiCorp’s Vault to store and retrieve our secrets.

We use Rails for web development. Luckily, there are a couple of gems that handle the setup of application environment variables. We agreed on using Figaro.

The easiest and least error-prone way to do it is moving all your secrets to the config/secrets.yml file. There you can list all application secrets and make them use environment variables. Here’s an example:

PROTIP: By using Figaro’s bang methods we make sure all environment variables are set. If an environment variable is not set it will throw a Figaro::MissingKey error.

With the secrets.yml file all ready, we created three Figaro config files: config/application.yml, config/application.staging.yml and config/application.production.yml and remembered to add those files to .gitignore. Here’s an example of a config file:

And here’s an example of how to actually use those secrets:

Now that our application is all 12-factored up, we need a way to share environment variables between developers and servers. Remember, we added application.*.yml files to gitignore so they don’t end up in the repository. No one has access to those files except for the person who wrote them. We are using Vault for sharing those files.

Here’s a summary of what Vault does:

Basically, it is an all-in-one solution for storing your critical information somewhere safe.

After setting up Vault on a separate server, we configured it to use consul.io as a backend. For authentication backends we are using github and app-id methods. A simple file system is used for an audit backend. You can look into Vault’s documentation for more information.

As we have hundreds of projects under our belt, naming conventions are a must if we want our developers to know where the secrets are stashed. We agreed on rails/#{git_repository_name}/#{environment} for a path to store secrets within Vault. We are using the Git repository name here as a part of the path because that isn’t something that ever changes, so everyone knows the location of the secrets.

Next, a policy needs to be created to give someone writing and/or reading privileges on a specific Vault path. Here’s an example of such a policy:

As Vault is easily integrated with GitHub, we are using GitHub teams to apply different application policies to users. Simply create a new GitHub team, add all needed users, and link the Vault policy to that team. This has an added benefit of only having to remove someone from your GitHub organization in order to revoke their access to all secrets. This can come in handy when someone leaves the company.

Except to users, we also need to grant access to servers and CI clients. We use what Vault calls app-id authentication. App-id authentication uses two strong keys to authenticate a Vault server. We give our machines read-only access to secrets within Vault.

As I mentioned before, Vault has no web interface, and its command line tool has a steep learning curve. We mitigated that problem by creating our own secrets gem. It uses the official Vault Ruby gem and is built around the requirements described above.

It has a couple of simple commands:

This will create a .secrets file with the project configuration. The command will ask for everything you don’t supply via options.

Here’s an example of a .secrets file:

And if you set up your environment variables correctly you can push and pull secrets:

This can also be set up on a server, so your deployment scripts pull secrets on every deployment. We use mina for our deployment purposes, and we created a mina-secrets plugin which simplifies our deployment with secrets.

So there it is, our approach to building a production-grade, scalable system for managing secrets. Using Vault makes the system secure. Using Github for the authentication makes it convenient for managing access. And using the secrets gem makes sharing secrets quick and dead simple for developers. Don’t sacrifice security for the sake of convenience any more.

Hope you enjoy using it.

As you work with your favorite programming language, you often look for tools to make your job easier. In the Ruby world, these tools (gems) are created on a daily basis, but only some of them are good enough to win our hearts.

Here are some of the core gems that have a special place in all of our Gemfiles. Some of them help with debugging code, others provide certain functionality like authentication. That’s useful in almost any project. In the end, they all help us work faster and produce quality software.

Whether you need simple HTTP authentication, or you’re looking for some complex Facebook and Google+ login system, this authentication solution is great and it will significantly speed up your work. It’s built on top of the Rails MVC stack and its modular architecture allows you to pick what you need.

The successor of the abandoned cancan project, this authorization gem lets you define what a user can do on your website by using abilities defined in a single place.

In newer projects we started experimenting with pundit. It is faster, more modular and it wont bloat a single file.

We find this gem to be the best when you need to extract some longer-running actions (e.g., sending newsletters to a large number of users) into a background task. You can reuse methods from your models, or you can create complex classes for all your background task needs. With delayed_job, even progress bars are possible in Ruby on Rails.

It’s quite annoying to set up email delivery in a development environment, and that’s where Letter Opener is great. It displays the email in your browser so you can easily inspect it, click any links (e.g., account confirmation) and not spam your email account with tons of test emails.

Easily configured, this gem allows you to do fast remote deployments of you website, as multiple environments are supported and all the work is done in only one SSH session (compared to Capistrano’s multiple SSH sessions). For more info, check out this article about Mina, written by my colleague Gabrijel Škoro.

You often have to deal with attachments that need to be processed in various ways. With paperclip, that becomes quite simple (with the support of ImageMagick) and allows you all sorts of validations and transformations, providing support for customizations as well.

If you feel adventurous, you can try refile from the creators of carrierwave.

Plain Rails generated forms (and related helpers) aren’t really that helpful when you’re designing a great user interface, which is essential for every website. This is where Simple Form saves the day with its powerful yet simple DSL.

If you’re doing testing (and you should!), this gem helps you to easily integrate rspec framework into your Rails project by generating the necessary configuration files and spec helpers.

When testing your projects, you need data. The best way to generate test data is to use a factory generator. The factory_girl_rails gem helps you to easily integrate Factory Girl, our favorite factory generator, into your Rails projects.

Debug your code like a champ. With pry, you can see what went wrong in any part of your Rails project, in your tests, third-party gems, and it can even be used in the Rails console.

Ever needed a progress bar for some long-running task in your Rails application? You searched Google and couldn’t find anything that easily integrates with Rails? Well, we created a progress_job gem that helps with that problem.

Progress_job is a gem that builds upon delayed_job to give you a simple progress bar you can use in your views. Create a class with your long-running task inside, update the job in each iteration and have ajax calls that will update the progress bar.

Progress_job depends on delayed_job, so I will write this tutorial using the delayed_job_active_record gem.

After adding the gem in your Gemfile

and running bundle install,

run the progress_job generator to add migrations necessary for progress_job to work.

This will add three columns to the delayed_jobs table:

Then you can create a custom class extending ProgressJob::Base which will give you access to some handy methods for manipulating the job.

Progress_job also gives you a route from which you can get all the info on a progress_job, and it is located at:

Now all you need is an ajax call which will check the route every few seconds and update the progress bar visible on the screen.

I’ve created a demo app. You can view its source.

In May 2011, the Information Commissioner’s Office first announced that websites have to explicitly seek consent for cookies (cookies are text files that record your activity online). Later known as the EU cookie law, it took effect in May 2012 with a grace period of a year.

Developers were outraged, every website in the EU would become illegal, and users would not be happy with popups every second or so. Interestingly, 24 hours before the prosecutions started, there had been a clarification of the law that said you only needed to have implied consent from users to use cookies.

The cookie law is a piece of privacy legislation that requires websites to obtain consent from visitors to store or retrieve any information from a computer or any other web-connected device, like a smartphone or tablet.

It has been designed to protect online privacy, by making consumers aware of how information about them is collected by websites, and enable them to choose whether or not they want to allow it to take place.

If you are based in the EU and own a website, you are now expected to comply with the law.

The easiest way to comply with the cookie law is to follow the accepted standard: implied consent.

Implied consent does not require a visitor to explicitly opt-in to the use of cookies on a website. However, it does mean there has to be an action taken by the user to confirm that consent is given, after there has been sufficient attempt to clearly inform the user about the use of cookies on the website.

These actions may be moving from one page to another, clicking on particular buttons on the website or choosing to continue to use the website.

The maximum penalty is £500,000. Whilst this is the ‘maximum’, its worth pointing out that this would be a rather extreme case of failure to comply. A formal warning and enforcement notice are far more likely, but should be avoided nonetheless.

Since we deploy a lot of web applications and websites for our clients and ourselves, we wanted to develop a standard system for resolving this issue.

We took inspiration from Google’s approach, as you probably noticed when you visited our site. We added a simple floating element to our page that says that we use cookies, and if you don’t disable them in your browser settings, you are OK with us using cookies to enhance your experience.

As we work primarily with Ruby on Rails, we created the cookies_eu gem to make any web application simple to comply by just adding the gem to the Gemfile. It’s open source, and you’re welcome to use it for your own Ruby on Rails applications.

If you don’t use Ruby on Rails, you can probably roll your own solution quickly in the technology you favor. We’ve provided a sample of a cookie privacy page that needs to be located somewhere on your webpage, which you can see here.