MongoDB High Availability

2.5 (2 reviews total)
By Afshin Mehrabani
    Advance your knowledge in tech with a Packt subscription

  • Instant online access to over 7,500+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Understanding the MongoDB Architecture and Processes

About this book

MongoDB is one of the pioneers in implementing the NoSQL concept by using "Document" as the infrastructure to save and restore data from a database. Using MongoDB in high pressure situations needs a predefined plan, and this book with its step-by-step approach can help you to make a highly available MongoDB server using the latest features.

Starting with the basic concepts and their implementation, you will study real-world use cases which will help you understand the practical aspects of MongoDB. Each step contains an in-depth presentation and several screenshots. This book teaches you all the tips and tricks to make a highly available MongoDB server using different clustering solutions. This book has a perfect balance of concepts and their practical implementation along with solutions to make a highly available MongoDB server in a production environment and under high throughput with clear instructions and guidance.

Publication date:
July 2014
Publisher
Packt
Pages
164
ISBN
9781783986729

 

Chapter 1. Understanding the MongoDB Architecture and Processes

To be able to diagnose a MongoDB server or change the default preferences to provide better performance in a database, we need to understand the primitive MongoDB settings and management tools. MongoDB consists of binary files and services that make the infrastructure of the server, and each file performs a specific task.

In this chapter, we will go through the MongoDB processes to discover why they exist and what each process does exactly.

 

Utilizing MongoDB components


This section contains a brief description of the MongoDB components, file names, and the main purpose of each of them. We will further discuss each item in detail. The following diagram shows you the four main components of MongoDB:

MongoDB components can mainly be classified into the following categories:

  • Core components

  • Import and export tools

  • Diagnostic tools

  • File storage (GridFS) tools

Each file can be placed into one of the aforementioned categories. The core component files are used to run the MongoDB server and start it. The files are also used to manage the MongoDB server from the command-line interface or manage clustering tasks.

Using import and export tools, developers can create dump files from their database in different formats such as BSON, JSON or CSV and restore them into another database again. Certain tools are used to create or restore BSON files, and some tools are responsible for generating and importing other common formats such as JSON or CSV.

MongoDB has many built-in diagnostic tools to manage and control the currently running server. Finally, we can use GridFS tools to interact with filesystem and GridFS components. In the next section, we will give you more details for each category and the processes inside them.

Understanding the core components

Inside this category, you can find the core processes that are required to start the MongoDB server. The MongoDB engine uses these tools to accept requests from different clients.

The components and executable files in this group are as follows:

  • mongod

  • mongo

  • mongos

Understanding mongod

The mongod component is the primary process that is needed for MongoDB to start the server. It manages requests or queries and handles connections.

The following screenshot is the result of executing mongod from the command-line interface:

The preceding screenshot illustrates the result of running mongod from the command line, that is, you finally have a MongoDB server running on port 27017 and a web interface on port 28017, which is the default port.

Like other MongoDB commands, in order to see the parameters of the mongod process, you can simply run following command:

mongod --help 

Here, you can see the result of running the mongod command with the --help parameter. This is shown in the following screenshot:

Note

To read more about the command and figure out what each parameter does, you can visit the MongoDB documentation page at http://docs.mongodb.org/manual/reference/program/mongod/.

Utilizing mongo

After starting the database server, we need to interact with it to issue a command, run commands, run queries, or get reports. In the MongoDB structure, the mongo file is responsible for this task. This is an interactive JavaScript shell that you can utilize from your command-line environment.

Using this executable file, database administrators and also developers can manage the database server or get available databases and collections.

The following screenshot is the result of running the mongo command:

Fortunately, the mongo component provides internal help so that developers can use it to get more information for each command.

The following screenshot shows you the result of running the help command:

Note

To acquire more information about MongoDB and all its parameters, you can visit the MongoDB documentation page at http://docs.mongodb.org/manual/reference/program/mongo/.

Learning about mongos

The mongos instance is responsible for routing read/write commands to shards in a sharded cluster. In fact, all processes and applications will connect to this instance and run queries, then the mongos instance will route commands to available shards. The applications will not interact with shards directly. To run the mongos instance, we can provide sharded cluster configurations using a config file or command-line parameters.

In further chapters, we will discuss this process in detail.

Note

To gain more knowledge about the mongos command and its parameters, you can visit the MongoDB documentation page at http://docs.mongodb.org/manual/reference/program/mongos/.

 

Import and export tools


The tools in this group are used to export or restore files from the database. Mainly, components in this category are classified into two different groups. This includes components that work with binary data or interact with common data formats such as JSON, CSV, and so on.

Using import and export tools

In order to import or export binary data from and into the database, developers can use tools in this group. We use tools and utilities in this category to create or restore backups for our MongoDB server. In further chapters, we will discuss the backup strategies in detail.

The following are the tools that are available to perform these kinds of tasks:

  • mongodump

  • mongorestore

  • bsondump

  • mongooplog

Understanding mongodump

This process comes in handy when developers or system administrators want to create a dump file from a database in the binary format. This utility and other related tools are useful for MongoDB backup strategies.

The mongodump process is able to retrieve data from either mongod or mongos processes.

Note

For more information on mongodump, visit the MongoDB documentation page at http://docs.mongodb.org/manual/reference/program/mongodump/.

Utilizing mongorestore

This process is used to restore and write the binary files that are generated by the mongodump process into the database server. To restore data, mongorestore can establish a new database or use an existing database. Just like the mongodump instance, mongorestore can connect to a mongos instance or it can connect directly to the mongod process.

Note

To read more about mongostore, you can visit the MongoDB documentation page at http://docs.mongodb.org/manual/reference/program/mongorestore/.

Learning about bsondump

The bsondump process is used to convert the BSON format data to common data formats such as JSON. Essentially, bsondump comes in handy when developers want to convert dump files that are generated by mongodump to human-readable formats.

A very simple usage of this command is shown in the following command line:

bsondump data.bson>data.json

Understanding mongooplog

The mongooplog is a utility that duplicates oplog from one server to another server, for instance, in order to perform a migration task. In order to perform the migration operation, mongooplog accepts two parameters, the from and to server addresses.

Note

What is oplog?

The oplog or operation log is a capped collection (a fixed-sized collection) of data that keeps the record of all data altering operations. In further chapters, we will explain this feature in detail.

The following is a simple usage of this command:

mongooplog --from server1 --host server2

The preceding command line will connect to the MongoDB instance of server1 and copy the entire oplog to server2 instance.

Using data tools

In this group, we have two utilities that help us generate or import data in human-readable formats such as JSON or CSV into or from the MongoDB instance. These are mentioned in the following lists:

  • mongoimport

  • mongoexport

Understanding mongoexport

In order to export data in JSON or CSV formats from the MongoDB instance, developers can use this utility.

The following is a simple usage of this command:

mongoexport --db mydb --collection posts --out export.json

The preceding command line will connect to a local instance of MongoDB, retrieve all records from the posts collection of the mydb database in the JSON format, and write all outputs to the export.json file.

Utilizing mongoimport

The mongoimport utility can help you import the produced export files in JSON, CSV or TSV formats into the MongoDB instance. The export files can be generated from either mongoexport or from other third-party export tools.

The following example is a basic usage of this command:

mongoimport --db mydb --collection posts --file export.json

The preceding command line will import the export.json entries into the posts collection of the mydb database. The same instruction can be used for other data formats using the --type option.

 

Diagnostic tools


One of the important tools of a database system is diagnostic tools. Fortunately, MongoDB has built-in diagnostic tools that enable developers to diagnose the server or get a brief report from the system.

We have the following utilities placed in this group:

  • mongostat

  • mongotop

  • mongosniff

  • mongoperf

In the next sections, you can read a brief description of each utility.

Learning about mongostat

This tool produces a brief summary of relevant statistics of the currently running MongoDB instances, either the mongod or mongos instance.

The following screenshot illustrates the output of this tool:

The preceding screenshot shows you the number of queries, update, insert, and delete operations from the database every second.

The following bullet list gives you a brief description for each column:

  • insert: This refers to the number of insert operations per second.

  • query: This refers to the number of queries per second.

  • update: This refers to the number of update operations per second.

  • delete: This refers to the number of delete operations per second.

  • getmore: This refers to the number of getmore operations (that is, the it command in mongo shell) per second.

  • command: This refers to the number of executed commands since the last mongostat call.

  • flushes: This refers to the number of fsync operations at the time of the last mongostat execution. The fsync operation is a system call that flushes all dirty in-memory pages to the disk.

  • mapped: This refers to the total amount of data mapped in megabytes.

  • vsize: This refers to the amount of virtual memory in megabytes used by the process at the time of the last mongostat execution.

  • res: This refers to the amount of resident memory in megabytes used by the process at the time of the last mongostat execution.

  • locked: This refers to the percentage of time in a global write lock.

  • idx miss: This refers to the percentage of index access attempts that required a page fault.

  • qr: This refers to the number of clients in the queue that are waiting for read operations.

  • qw: This refers to the number of clients in the queue that are waiting for write operations.

  • ar: This refers to the number of clients that execute read operations.

  • aw: This refers to the number of clients that execute write operations.

  • netIn: This refers to the traffic received by the MongoDB instance in bytes.

  • netOut: This refers to the traffic sent by the MongoDB instance in bytes.

  • conn: This refers to the current total option connections.

The refresh interval can be changed using the following command:

mongostat [options] [sleep time] 

Utilizing mongotop

The mongotop utility provides you with a mechanism to get information about time spent on read/write operations. This command is similar to Unix's top command.

The following screenshot shows you a simple usage of mongotop:

Understanding mongosniff

The mongosniff is a tool that is used to fetch live MongoDB collection statistics. While inserting or querying data from the MongoDB instances, you can run the mongosniff command and connect it to your MongoDB instance to see what the database does.

Note

Please note that in order to use this utility, you should install the libpcap library first. To install the libpcap library, please visit its official website at http://www.tcpdump.org/#documentation.

A simple usage of the mongosniff tool is as follows:

sudo mongosniff  --source NET lo0

The preceding command line will listen to the loopback interface (localhost). This interface is lo0 in Mac OS systems and lo for other operating systems, usually. You can get the list for your network interfaces using the ifconfig command. If you're using Windows as the operating system, you can get the list of network interfaces using the following command:

ipconfig /all

Utilizing mongoperf

The mongoperf tool represents the disk I/O performance. It checks the I/O in a specified interval and illustrates it. This utility can be used independent of MongoDB.

File storage (GridFS) tools

With the help of GridFS, MongoDB can be used as a filesystem. The processes in this section are used to manage and control the GridFS feature.

There is one process in this category, which is as follows:

  • mongofiles

Understanding mongofiles

This utility enables developers to retrieve files that are stored in the database in the GridFS collection. The mongofiles utility come in handy when developers need to interact with files stored in the database from the command-line environment.

The usage of this command looks like the following:

mongofiles <options> <commands> <filename>

The following example is a simple usage of this utility:

mongofiles -d mydb list

The preceding command line will retrieve all files in the GridFS collection from the mydb database.

Note

For more information on mongofiles, please visit the MongoDB documentation page at http://docs.mongodb.org/manual/reference/program/mongofiles/.

 

Summary


In this chapter, we went through basic MongoDB topics such as major MongoDB processes, how they work, and why they exist.

We learned that MongoDB consists of some main components such as core, import and export, GridFS, and diagnostic tools. Then, we discussed the basic processes that make MongoDB work, that is, mongod and mongos. Also, we learned that developers or system administrators can manage MongoDB using the mongo process. This is an interactive JavaScript shell, which enables developers to run and execute commands, queries, and administration procedures.

Next, we talked about import and export tools that give developers the ability to export and import objects from and into the database, which is used for backup and restoration procedures. In addition, you can find a brief description of the GridFS components and diagnostic tools that are required to work with the filesystem. We also learned how to find database statistics and issues with diagnostic tools.

In the next chapter, we will learn about the causes of failure in MongoDB and find remedies and solutions to overcome these problems.

About the Author

  • Afshin Mehrabani

    Afshin Mehrabani is an open source programmer. He is studying to be a computer software engineer. He started programming and web development when he was 12 years old, as well as starting with PHP. Later, he joined the Iran Technical and Vocational Training Organization. He secured the first place and received a gold medal in a competition which was conducted across the entire country in the area of web development. He became a member of the Iran National Foundation of Elite after producing a variety of new programming ideas.

    He was a software engineer at the Tehran Stock Exchange and is now the head of the web development team in the Yara Company. He cofounded the Usablica team in early 2012 to develop and produce usable applications. He is the author of IntroJs, WideArea, flood.js and some other open source projects.

    He has contributed to Socket.IO, Engine.IO, and some other open source projects. He is also interested in creating and contributing to open source applications, writing programming articles, and challenging himself with new programming technologies.

    He has written different articles about JavaScript, Node.js, HTML5, and MongoDB that have been published on different academic websites. Afshin has 5 years of experience in PHP, Python, C#, JavaScript, HTML5, and Node.js in many financial and stock trading projects.

    Browse publications by this author

Latest Reviews

(2 reviews total)
Bad
Good
Book Title
Unlock this book and the full library for only $5/m
Access now