The Crawler Pattern

In the ever-evolving landscape of software development and data management, it’s crucial to have efficient and scalable tools that can handle bulk operations with ease. Enter the crawler pattern, a powerful technique that has become a cornerstone of my approach to mass repository management. This chapter is dedicated to unveiling the intricacies of this pattern, exploring its numerous applications, and showcasing its remarkable potential to transform how we handle repositories on a large scale.

Throughout this chapter, I’ll share personal anecdotes and insights from my journey with the crawler pattern, detailing how it became a game-changer in streamlining and optimizing repository management tasks. Whether you are new to the concept or simply looking to refine your approach, the examples provided herein will guide you through efficiently managing, updating, and modifying repositories in bulk, crafting multiple pull requests simultaneously while...

Technical requirements

To follow along with the hands-on material in this chapter, you will need to follow the steps in the previous chapter or access the resources from that chapter and refer back to it if anything is ambiguous to you. We will continue to use our self-hosted runners to make these changes. You’ll need to create a new private repository called CommandCentre.GitHub for hosting the scripts used to manage repositories en masse. Initialize the repository with a readmefile. We will be stepping through the process of creating everything else in this chapter.

Introducing the crawler pattern

Note

I can’t take credit for the establishment of this pattern; I stumbled upon a variant of this pattern in use by a colleague of mine, Graeme Christie (https://github.com/graemechristie), who used it to roll out config for a GitHub app being used internally. It is a very useful pattern to have in your toolkit when working within organizations.

The crawler pattern is a technique where desired repositories or objects are input into a GitHub workflow matrix, triggering an individual action for each repository within a runner. We can then perform actions in an ephemeral environment on the repository, allowing us to use the runners as disposable workers.

I find this is a useful way for organizations to manage multiple repositories that need to implement widespread changes. Some organizations facing mass change rollout face a dilemma:

• They can tediously manually apply these changes, dismissing the advantages of DevOps.
• They can...

Making bulk repository changes

Sometimes, making administrative modifications in GitHub can be unduly challenging, especially when managing branch protections, permissions, and repository features. Additionally, several minor adjustments should be straightforward but prove difficult or slow to do so through a UI. By harnessing the capabilities of actions and our scalability feature, we can devise methods to synchronize permissions across repositories or retrieve information swiftly.

This section will review two workflows that everyone will use: adding permissions to all repositories you target and adding branch protection. Let’s look at both of these workflows in detail in the following sections.

Adding permissions to all repositories

We will leverage the topic query action in this segment to pinpoint the user-targeted repositories requiring modifications. While our current approach utilizes the topic query action, it’s worth noting that this method can be conveniently...

Making bulk content changes

In this section, we’ll focus on rolling out new workflows and making content changes in bulk, but first, let’s take a look at what this workflow looks like.

Understanding the workflow

The process of finalizing a new workflow is illustrated in the following diagram:

Figure 12.10 – Content deploy flow

The preceding diagram shows the same flow as the repository changes in the creation of the Adding branch protection section, except that, instead, we’re copying content from the command center and applying it to a repository using a pull request process. Doing this involves a few extra actions, but most of the foundation is the same as before, so I won’t be going over that in much detail and will refer to it as a get-repos job. You can use any method of feeding the matrix you see fit. Let’s put the preceding flow into play by rolling out content across our repositories.

Rolling out...

Summary

In this chapter, you were given a firsthand account of the author’s experiences with the crawler pattern, a technique that has transformed how we approach repository management tasks. The chapter revealed how the crawler pattern aids in streamlining massive repository operations, especially when handling vast amounts of data and changes.

Central to our discussion were the challenges inherent to repository management, particularly in onboarding scenarios. With the potential need to make hundreds or even thousands of changes, repository management can easily become a cumbersome task. The crawler pattern is an invaluable asset in such contexts, turning seemingly daunting tasks into straightforward, manageable operations.

You were taken on a journey, starting with an introduction to the crawler pattern and progressing to its application in real-world scenarios. You are now equipped with a comprehensive understanding of the crawler pattern’s utility and potential...