Avoid year old dependencies with CircleCI Dependabot

In a previous post we covered the creation and usage of a small homebuild dependabot, but only for CircleCI. The goal was updating CI specific dependencies like images and orbs. If you want the details and the reason behind creating our own small dependabot check out the post here. Even though the first iteration was open source and free for you to use, I will admit, we had not made it easy for anyone other than ourselves to use it. Since the first release we have had the desire to make a way easier to use for others, but as it is not really our main business, we have kept the development of dependabot-circleci to slack and spare time. We often do small hackathons in the team, where we gather and hack on something specific for a whole day. This time we decided to play around with both dependabot-circleci but also Cloud Run, the main goal of the hackathon was how we could build a multi-tier app in Googles serverless infrastructure.

tl;dr: If you want to enable dependabot-circleci right away and get dependency updates automatically, check out the Getting Started

Dependabot-circleci take one

Before we go straight into the cool new dependabot-circleci, let’s look at how the old one was designed and how the flow was. It consisted of a single application with multiple routes; the webhook and the check engine. Everything running in a single Cloud Run instance.

The Webhook

The first part is the webhook, it is triggered every time a user pushes a commit in an enabled repository. The webhook checks if the dependabot-circleci config is valid and returns this as a GitHub check to the user/repository. You have probably seen either the small red cross or the green correction sign in your GitHub repositories:

The Check engine

The check engine is where the magic happens. It loops through all the repositories and for each it does the following:

1. Checks if the dependabot-circleci config is valid.
2. Scans the CircleCI config for Orbs and images
3. For each orb and image it checks if there is a newer version.
4. If a newer version is found, a pull request for each updated dependency will be created in that specific repository.

There is a bit more features to it, like superseeding pull requests, schedules and more. But for this article i will try to keep it simple. All this is triggered once a day by Cloud Scheduler.

Design flaws

That might look like a solid application, but I will admit there was some design flaws. We especially had problems with Githubs rate limits, to be specific their secondary rate limits. Yeah, that was my thoughts as well... They have more than one set of rate limits. To elaborate, the secondary rate limit increments every time you rapidly create content e.g. PUT, POST, PATCH or DELETE request towards their APIs. It did not help that we queried all repositories in our organisation every time the check engine was triggered, meaning that even if the repository did not have a dependabot-circleci config we would look for it anyway.

Dependabot-circleci take two

With all this knowledge in mind we decided to create a new multi-tiered design based on Cloud Run and Cloud SQL. We decided to keep the main parts but fixing some of the flaws along the way. Let's dive right into it

The Webhook

Again, the first part, the webhook, looks pretty similar to what the original had. Two major changes, first it is running in a Cloud Run function for itself, meaning it can scale depending on how many checks is being requsted. Second if the dependabot-circleci config is valid it will now add the repository to the Cloud SQL layer, this part enables us to query only for the enabled repositories instead of all repositories in an organization.

The Controller

The controller layer is single container which queries our database for organizations and repositories, for each organization it spawns a worker container and provides it with a list of repositories.

The workers

The worker containers implement the check engine from the first dependabot-circleci, this is again where the real magic happens. This time though it only runs through the repositories the controller has provided and does the same checks for Orb and image updates. The biggest difference is that a worker is spawned for each organization meaning that we can run checks in parallel and independent of each other.

And suddenly we had a multi-tiered app using Cloud Run. But most important it is now super easy for you to use dependabot-circleci, simply install the Github application in your organization and add a dependabot-circleci config to your repository and prepare for incoming pull requests. For a bit more details check out the readme.

Final Words

If you are using CircleCI, like us, I believe this is a brilliant addition to your toolbox. You won't end up in situation with year old dependencies.

Next steps for dependabot-circleci is to add more features such as support for private orbs, release notes in pull requests and support for CircleCI orb structure (if you are building your own orbs). Just to name a few :) And as always, we are open to suggestions, features, issues and pull requests, checkout the repository and remember to star it.

About the author

Peter Brøndum

My name is Peter Brøndum, I work as a Tech Lead and Scrum Master in a platform engineering team at BESTSELLER. Our main priority is building a development highway, with paved roads, lights and signs, so our colleagues can deliver value even faster. Besides working at BESTSELLER, I — amongst other things, am automating my own home, and yes, that is, of course, running on Kubernetes as well.