Avoid year old dependencies with CircleCI Dependabot
In a previous post we covered the creation and usage of a small homebuild dependabot, but only for CircleCI. The goal was updating CI specific dependencies like images and orbs. If you want the details and the reason behind creating our own small dependabot check out the post here.
Even though the first iteration was open source and free for you to use, I will admit, we had not made it easy for anyone other than ourselves to use it. Since the first release we have had the desire to make a way easier to use for others, but as it is not really our main business, we have kept the development of
dependabot-circleci to slack and spare time. We often do small hackathons in the team, where we gather and hack on something specific for a whole day. This time we decided to play around with both
dependabot-circleci but also Cloud Run, the main goal of the hackathon was how we could build a multi-tier app in Googles serverless infrastructure.
tl;dr: If you want to enable dependabot-circleci right away and get dependency updates automatically, check out the Getting Started
Dependabot-circleci take one
Before we go straight into the cool new
dependabot-circleci, let’s look at how the old one was designed and how the flow was.
It consisted of a single application with multiple routes;
the webhook and
the check engine. Everything running in a single Cloud Run instance.
The first part is the webhook, it is triggered every time a user pushes a commit in an enabled repository. The webhook checks if the
dependabot-circleci config is valid and returns this as a GitHub check to the user/repository. You have probably seen either the small red cross or the green correction sign in your GitHub repositories:
The Check engine
The check engine is where the magic happens. It loops through all the repositories and for each it does the following:
- Checks if the
dependabot-circleciconfig is valid.
- Scans the
CircleCIconfig for Orbs and images
- For each orb and image it checks if there is a newer version.
- If a newer version is found, a pull request for each updated dependency will be created in that specific repository.
There is a bit more features to it, like superseeding pull requests, schedules and more. But for this article i will try to keep it simple. All this is triggered once a day by Cloud Scheduler.
That might look like a solid application, but I will admit there was some design flaws. We especially had problems with Githubs rate limits, to be specific their secondary rate limits. Yeah, that was my thoughts as well... They have more than one set of rate limits. To elaborate, the secondary rate limit increments every time you rapidly create content e.g.
DELETE request towards their APIs.
It did not help that we queried all repositories in our organisation every time the
check engine was triggered, meaning that even if the repository did not have a
dependabot-circleci config we would look for it anyway.
Dependabot-circleci take two
With all this knowledge in mind we decided to create a new multi-tiered design based on Cloud Run and Cloud SQL. We decided to keep the main parts but fixing some of the flaws along the way. Let's dive right into it
Again, the first part,
the webhook, looks pretty similar to what the original had. Two major changes, first it is running in a Cloud Run function for itself, meaning it can scale depending on how many checks is being requsted. Second if the
dependabot-circleci config is valid it will now add the repository to the Cloud SQL layer, this part enables us to query only for the enabled repositories instead of all repositories in an organization.
The controller layer is single container which queries our database for organizations and repositories, for each organization it spawns a worker container and provides it with a list of repositories.
The worker containers implement the
check engine from the first
dependabot-circleci, this is again where the real magic happens. This time though it only runs through the repositories the controller has provided and does the same checks for Orb and image updates. The biggest difference is that a worker is spawned for each organization meaning that we can run checks in parallel and independent of each other.
And suddenly we had a multi-tiered app using Cloud Run. But most important it is now super easy for you to use
dependabot-circleci, simply install the Github application in your organization and add a
dependabot-circleci config to your repository and prepare for incoming pull requests. For a bit more details check out the readme.
If you are using CircleCI, like us, I believe this is a brilliant addition to your toolbox. You won't end up in situation with year old dependencies.
Next steps for
dependabot-circleci is to add more features such as support for private orbs, release notes in pull requests and support for CircleCI orb structure (if you are building your own orbs). Just to name a few :)
And as always, we are open to suggestions, features, issues and pull requests, checkout the repository and remember to star it.
About the author
My name is Peter Brøndum, I work as a Tech Lead and Scrum Master in a platform engineering team at BESTSELLER. Our main priority is building a development highway, with paved roads, lights and signs, so our colleagues can deliver value even faster. Besides working at BESTSELLER, I — amongst other things, am automating my own home, and yes, that is, of course, running on Kubernetes as well.