Background and scheduled jobs
Last modified on Wed 22 Jul 2020

Our goal is to have applications that process incoming web requests in a timely fashion. Some requests don't just manipulate data from your database, but also perform time-consuming operations, eg:

Performing time-consuming tasks during the request processing negatively impact response time.

Background jobs

The solution is to extract time-consuming tasks and execute them asynchronously.

A common idiom is to extract time-consuming tasks into units of work called jobs. The main process (the request) can defer (schedule) a job for later execution (in the background).

There are many Ruby projects dealing with background jobs * Sidekiq * DelayedJob * Resque * ...

After trying them all, we decided on Sidekiq. Sidekiq offers the optimal performance, versatility and support for our purposes.

To incorporate Sidekiq into your project, you'll need the following infrastructure:

Instructions on how to configure Sidekiq to work with Redis can be found here. After you setup a worker (background job), you can hook it up wherever it is required and choose when it's going to be executed.

Error handling

Another great benefit of using background jobs is improving fault tolerance, especially if you have an API that communicates with another API.

We can never be 100% sure that the dependent API works OK, which could cause errors in the database or 500 HTTP errors.

Sidekiq background jobs provide options that are defined as class methods which can be used to improve error handling.

Best practices

Some of the best practices have been covered in the official Sidekiq Wiki pages.

Active Job

Active Job is a framework for declaring jobs and making them run on a variety of queuing backends.

To setup Sidekiq as a queuing library for Active Job read this Wiki.

Scheduled jobs

Sometimes a service needs to be run recurrently and not in an HTTP request lifecycle:

This is where sidekiq-scheduler comes in handy. It also provides the cron option for specifying when the service will be called if you prefer or require it.

Sidekiq Monitoring

APIs usually require a dashboard interface for admin users. It makes sense to also include a monitoring tool that has a lot of useful info in regards to background jobs.

You can use built-in methods in your config/routes.rb file to enable this feature.

NOTE:

If you have nginx configured to only look up assets in the current/public folder, the Monitoring dashboard will load without the sidekiq/web assets. To solve this, you have to create a symlink to the Sidekiq assets folder using the link_sidekiq_assets mina task:

task :deploy do
  invoke :'git:ensure_pushed'
  deploy do
    invoke :'git:clone'
    invoke :'deploy:link_shared_paths'
    invoke :'bundle:install'
    invoke :'secrets:pull'
    invoke :'rails:db_migrate'
    invoke :'rails:assets_precompile'
    invoke :'deploy:cleanup'

    on :launch do
      invoke :link_sidekiq_assets
      invoke :restart_application
      invoke :restart_sidekiq
    end
  end
end

In case your Sidekiq monitoring tool is namespaced behind an authenticated route:

authenticate :administrator do
  mount Sidekiq::Web => '/admin/sidekiq'
end

then you also need to:

set :sidekiq_web_namespace, :admin

in your config/deploy.rb file.

Interesting articles: