The Links 'R'; Us Project

"The hardest part of the software task is arriving at a complete and consistent specification, and much of the essence of building a program is in fact the debugging of the specification."

- Frederick P. Brooks ^[3]

In this chapter, we will be discussing Links 'R' Us, a Go project that we will be building from scratch throughout the remaining chapters in this book. This project has been specifically designed to combine everything you have learned so far with some of the more technical topics that we will be touching on in the following chapters: databases, pipelines, graph processing, gRPC, instrumentation, and monitoring.

The following topics will be covered in this chapter:

A brief overview of the system that we will be building and its primary function
Selecting an appropriate SDLC model for the project
Functional and...

System overview – what are we going to be building?

Throughout the next chapters, we will be assembling, piece by piece, our very own search-engine. As with all projects, we need to come up with a cool-sounding name for it. Let me introduce you to Links 'R' Us!

So, what are the core functionalities of the Links 'R' Us project? The primary, and kind of obvious, functionality is being able to search for content. However, before we can make our search engine available to the public, we first need to seed it with content. To this end, we need to provide the means for users to submit URLs to our search engine. The search engine would then crawl those links, index their content, and add any newly encountered links to its database for further crawling.

Is this all we need for launching Links 'R' Us? The short answer is no! While user searches would...

Selecting an SDLC model for our project

Before delving into the details of the Links 'R' Us project, we need to consider the SDLC models we discussed in Chapter 1, A Bird's-Eye View of Software Engineering, and select one that makes more sense for this type of project. The choice of a suitable model is of paramount importance: it will serve as our guide for capturing the requirements for the project, defining the components and the interface contracts between them, and appropriately dividing the work to be done in logical chunks that can be built and tested independently of each other.

In this section, we will outline the main reasoning behind the selection of an Agile framework for our project and elaborate on a set of interesting approaches for speeding up our development velocity using a technique known as elephant carpaccio.

...

Requirements analysis

To perform a detailed requirements analysis for the Links 'R' Us project, we need to essentially come up with answers for two key questions: what do we need to build and how well would our proposed design fare against a set of goals?

To answer the what question, we need to list all of the core functionalities that our system is expected to implement as well as describe how the various actors will interact with it. This forms the Functional Requirements (FRs) for our analysis.

To answer the latter question, we have to state the Non-Functional Requirements (NFRs) for our solution. Typically, the list of non-functional requirements includes items such as Service-Level Objectives (SLOs) and capacity and scalability requirements, as well as security-related considerations for our project.

...

System component modeling

As the first step in mapping the project's architecture, we will begin by creating a UML component diagram. The main goal here is to identify and describe the structural connections between the various components that comprise our system.

A component is defined as an encapsulated standalone unit that constitutes an integral part of a system or a sub-system. Components communicate with each other by exposing and consuming one or several interfaces.

One key point of component-based design is that components should always be considered as abstract, logical entities that expose a particular behavior. This design approach is closely aligned with the SOLID principles and offers us the flexibility to freely change or even swap component implementations at any point throughout the project's development.

The following diagram breaks down the Links...

Monolith or microservices? The ultimate question

Before commencing development of the Links 'R' Us service, we need to decide whether our system components will be developed as parts of a big, monolithic service or whether we will just bite the bullet and implement a service-oriented architecture right from the start.

While the concept of using microservices does indeed seem enticing from the outside, it comes with a lot of operational overhead. Besides the mental effort required for building and wiring all components together, we would additionally need to worry about questions like the following:

How does each service get deployed? Are we doing rolling deployments? What about dark or test releases? How easy is it to roll back to a previous deployment when something goes wrong?
Are we going to use a container orchestration layer such as Kubernetes ^[6]? How does traffic...

Summary

This concludes the presentation of the Links 'R' Us project. I hope that, by this point, you have acquired a general understanding of what we are going to be building over the next few chapters. If you find yourself wondering about the technical implementation details associated with some of the project components, that's perfectly normal. The main purpose of this chapter was to introduce a high-level overview of the project. We will analyze the construction of each one of these components in extensive detail in the pages that follow!

To make the concepts and code for the following chapters easier to follow, we will be splitting each chapter into two core parts:

In the first half of each chapter, we will be performing a deep dive into a particular technical topic, for example, a survey of popular types of databases (relational, NoSQL, and so on), how you...

Questions

What is the difference between a functional and a non-functional requirement?
Describe the main components of a user story.
What things could possibly go wrong in the Links 'R' Us scenario if we blindly crawl any link that a user submits to the system?
Name the key components of an SLO.
What is the purpose of a UML component diagram?