Our goal is to have applications that process incoming web requests in a timely fashion. Some requests don't just manipulate data from your database, but also perform time-consuming operations, eg:
- send emails
- send push notifications
- execute transactions
- generate PDFs
- interact with 3rd party APIs
- perform other long-running tasks, eg. updating database indices (Algolia)
Performing time-consuming tasks during the request processing negatively impact response time.
The solution is to extract time-consuming tasks and execute them asynchronously.
A common idiom is to extract time-consuming tasks into units of work called jobs. The main process (the request) can defer (schedule) a job for later execution (in the background).
There are many Ruby projects dealing with background jobs * Sidekiq * DelayedJob * Resque * ...
After trying them all, we decided on Sidekiq. Sidekiq offers the optimal performance, versatility and support for our purposes.
To incorporate Sidekiq into your project, you'll need the following infrastructure:
Instructions on how to configure Sidekiq to work with Redis can be found here. After you setup a worker (background job), you can hook it up wherever it is required and choose when it's going to be executed.
Another great benefit of using background jobs is improving fault tolerance, especially if you have an API that communicates with another API.
We can never be 100% sure that the dependent API works OK, which could cause errors in the database or 500 HTTP errors.
Sidekiq background jobs provide options that are defined as class methods which can be used to improve error handling.
Some of the best practices have been covered in the official Sidekiq Wiki pages.
Active Job is a framework for declaring jobs and making them run on a variety of queuing backends.
To setup Sidekiq as a queuing library for Active Job read this Wiki.
Sometimes a service needs to be run recurrently and not in an HTTP request lifecycle:
- every N (seconds/minutes/hours/days)
- on a certain day of every month
- on a certain hour of every day
This is where sidekiq-scheduler comes in handy.
It also provides the
cron option for specifying when the service will be called if you prefer or require it.
APIs usually require a dashboard interface for admin users. It makes sense to also include a monitoring tool that has a lot of useful info in regards to background jobs.
You can use built-in methods in your
config/routes.rb file to enable this feature.
If you have nginx configured to only look up assets in the
current/public folder, the Monitoring dashboard will load without the
sidekiq/web assets. To solve this, you have to create a symlink to the Sidekiq assets folder using the
link_sidekiq_assets mina task:
task :deploy do invoke :'git:ensure_pushed' deploy do invoke :'git:clone' invoke :'deploy:link_shared_paths' invoke :'bundle:install' invoke :'secrets:pull' invoke :'rails:db_migrate' invoke :'rails:assets_precompile' invoke :'deploy:cleanup' on :launch do invoke :link_sidekiq_assets invoke :restart_application invoke :restart_sidekiq end end end
In case your Sidekiq monitoring tool is namespaced behind an authenticated route:
authenticate :administrator do mount Sidekiq::Web => '/admin/sidekiq' end
then you also need to:
set :sidekiq_web_namespace, :admin