How We Escaped the Dependency Upgrade Maze on 20+ Projects

Open-source dependencies are practical, convenient, time-saving – and sure to make your life miserable if left unattended. Read on to learn how we optimized our dependency maintenance process and got all our Ruby on Rails projects updated to the latest versions with minimum time loss.

Modern applications and frameworks are composed of hundreds of individual dependencies.  Using third-party libraries, frameworks, and other ready-made software components is a time-saver during development. However, dependencies can also incur an often overlooked maintenance cost.

As a digital agency, we often find ourselves in a situation where we need to perform maintenance on applications with severely outdated dependencies that haven’t been updated in years. These apps present a series of challenges – upgrade efforts are estimated in months, the development pace is slower, and they are often susceptible to numerous security vulnerabilities.

In this article we will present our process and tooling improvements that helped us increase dependency health for more than 20 Ruby on Rails projects under our maintenance in the last couple of years.

Modern software development relies heavily on open-source dependencies. The developer community maintains countless dependencies for all possible use cases in the diverse software landscape. Feature development used to take a lot of effort, and now a large portion of it can be easily replaced by adding a single line to the Gemfile and setting some initializer options. It’s an elegant solution – why write lines and lines of code when the dependency’s author already designed, implemented, reviewed, and tested the exact functionality your project requires; all you have to do is include it.

This is not reserved only for the Ruby ecosystem; other software development camps also heavily rely on the work of open-source authors.

However, the problem with using third-party components is that we tend to take care of our own code garden and rarely think about maintaining downstream dependencies. Our codebases evolve, but so do our dependencies. 

Open-source authors spend considerable time maintaining their libraries during their support life cycle. They introduce new features, fix bugs, and even introduce performance enhancements. All we have to do to reap the benefits is to upgrade to the current version, yet we seldom do so.

One of the benefits of agency work is the variety of projects and situations that we encounter. During the last 18 years, we’ve worked with countless client applications, and at any given moment, we’re also actively maintaining around ten internal ones.

It’s not uncommon for us to take over a larger client project that is two or more Ruby on Rails versions behind. Just recently, we started supporting a Ruby on Rails project started seven years ago, and dependency upgrades were the first order of business. Bringing the application to the latest major Ruby and Rails version took more than 500 hours of development work. That’s a big chunk of time to pay for a one-time upgrade. 

However, this situation is far from uncommon – even GitHub took a year and a half to upgrade Rails from 3.2 to 5.2 before they adopted a continual upgrade strategy.

Understanding framework changes, auxiliary gem changes, and how they interact with your codebase is a big undertaking. Running bundle update devise for a single dependency is trivial, but understanding the change described in a dependency changelog requires a lot of context. Good automated test coverage helps build confidence in the upgrade, but we’re still often pushing out a risky code change we aren’t entirely comfortable with.

Always focusing on client-requested feature work, we were also guilty of often sweeping our dependency updates under the carpet. When Rails 7 was introduced around two years ago, we were faced with the exact amount of our technical debt.

Wanting to capitalize on all the newly related features (ActiveRecord encryption, ActiveRecord load_async) across all projects under our care, we decided to conduct a fleet dependency health review.

Most applications shared some common characteristics:

Bringing just one project up to date seemed like a big chore and getting all of them back on track seemed infeasible. We have performed considerable dependency upgrades before, but planning, estimating, and safely executing a mass wave of upgrades on applications under our care was uncharted territory.

We realized that we couldn’t continue on the same upgrade path. Our team needed to improve our dependency upgrade culture and processes to repay the debt and prevent it from accruing again. It was time to figure out how to perform the upgrades safely and do a better job of tracking and executing dependency upgrades on internal and client projects in the future.

After interviewing several internal project teams and looking at industry best practices, we identified the weakest points in our dependency upgrade culture and came up with a set of resources and guidelines to help us get to the next level.

The most significant change we introduced was mandating small but frequent dependency upgrades on all projects. Regular (and minor) dependency upgrades are more predictable, safer, and easier to plan than large ones. With a little inventiveness, all upgrades can be performed in bite-size chunks.

Upgrading dependencies is a never-ending process since new versions are released daily. To stay on top, we need to be as efficient as possible. Not all dependency upgrades have the same impact, and we should prioritize them accordingly:

We emphasized the urgency of the security upgrade by requiring all project teams to perform CVE triage and resolution within one business day. Each additional day of exposure to a CVE in the wild increases the likelihood that an automated vulnerability scanner will find the application and exploit it. 

Rails upgrades should be carefully researched and planned in multiple stages as defined by our guide. Revolutionary upgrades are strongly discouraged. We want to move in small, predictable, and reversible steps to reduce the likelihood of regressions. 

Project dependency health should be easy to determine programmatically. Tracking it for all projects under our care over time allows us to gauge our progress and hold data-backed discussions with stakeholders.

Implementing these changes required not only changing the process but also some new tooling.

Tracking dependency health over time can be performed manually for individual projects, but keeping up with the 20+ active projects we support requires some automation. We decided to invest some time into building basic and effective tooling that would enable us to easily keep an eye on the health of our applications.

For this purpose, we built an application called Revisor. It periodically aggregates Ruby dependency health data of all the projects in a GitHub organization and exposes it in a simple dashboard.

The core logic is simple: we query all Ruby repositories in our Github organization daily and evaluate their Gemfile.lock. Each project is then assigned a dependency health score, which is based on the following formula:

The data is presented on a simple dashboard for software developers, but we also decided to periodically showcase the information on project Slack channels so stakeholders can see our progress.

Ranking the projects proved to be a great incentive for both the development teams and stakeholders. Development teams take pride in having the least amount of outdated dependencies and patching the newest CVE first. On the other hand, stakeholders sometimes get curious about how other project teams are getting better results and what it would take to catch up with them.

Before we launched our initiative, most project stakeholders weren’t accustomed to dependency upgrades. We discussed and executed general software maintenance items before, but we didn’t put much emphasis on dependency upgrades besides the occasional Rails upgrade. They were suddenly faced with an additional (recurring!) amount of work that would reduce our feature output in the mid-term.

On the other hand, upgrading severely outdated projects requires months of development time and blocks most feature delivery.

With internal projects, we had strong support from technical leadership as we had seen the effects and cost of deferring upgrades firsthand. Introducing our new standard to client projects was a longer discussion, but our incentives turned out to be very much aligned. We all want to make our dependency upgrade process as lean and predictable as possible so that feature output can be continuous. 

Regular, smaller dependency upgrades allow us to meet all of those requirements. Keeping the application healthy for optimal development speed is just the icing on the cake.

Introducing culture and process changes is a noble goal in itself, but we also wanted to get some empirical data on the changes we introduced. The idea was to visualize the past state and evaluate whether our decisions led us to a better overall outcome.

Revisor provides us with a current project dependency health score and answers the question “What’s the current dependency health of the project?”. On the other hand, to answer “How has our dependency health progressed over the last year?” requires historical data. 

When we posed this second question, Revisor was still in its infancy, so we relied on another data source – Rubygems.

Rubygems holds historical data (version, release date) about all published gems. This allows us to obtain a publicly available database dump and determine which version of a specific gem was the most recent one at any point in time. The last remaining input is the exact version of the gem that the project had used at a point in time, which can be determined through git history.

Our script iterates over the defined date range, checks out Gemfile.lock from a specific date, and counts the number of outdated gems in the lock file on that date. A gem is considered outdated if a higher version was released before the specific date we’re evaluating. Outdated gem counts per date are then printed to standard output for further plotting.

Interpreting the graphs is not an exact science, but generally speaking, we expect the number of outdated gems to rise over time as new gem versions are released. The effort of the development team is seen when the number of outdated gems is reduced, either through dependency upgrades or dependency removals. We can observe the frequency and scale of those changes to determine if the trend points downward. With these graphs, both the development team and the stakeholders get a clear picture of the project dependency health trajectory.

Improving our dependency update posture took a while, but I can happily say that our efforts have paid off. In the end, we’ve managed to accomplish all of our goals:

What’s more, we have also amassed a great deal of upgrade experience over the past two years. We’re now better versed in researching, planning, and executing complex dependency upgrades.

The number of projects we maintain as an agency presents a unique set of challenges. However, our foray into improving dependency upgrade processes and internal tooling has allowed us to bring our service to the next level. 

Starting with a very grim report sheet – a large portion of our projects more than two major Ruby and Ruby on Rails versions behind, we’ve now brought all of our projects up to date with the latest major Ruby and Ruby on Rails versions. 

Dependency upgrades are a core part of our service, and most projects are on a bi-weekly upgrade schedule. By increasing the frequency of dependency upgrades, we drastically reduced their difficulty but also brought additional value to our clients – their apps are now faster, safer, and up to date.