Learning Azure DocumentDB: Create outstanding enterprise solutions around DocumentDB using the latest technologies and programming tools with Azure

As stated before, NoSQL databases are gaining popularity and are slowly replacing traditional relational databases. The main characteristics of a NoSQL database are listed next:

Having no schema (or predefined structure like tables and columns) allows us to store everything. This also includes attachments, user-defined functions, stored procedures, triggers, and more. The only restriction is that the information has to be in valid JSON.

The SQL language that can be used to query and manipulate DocumentDB is based on JavaScript. Having JavaScript at the core means that we do not need to learn new techniques or languages, and our current knowledge of JavaScript can be applied immediately. Using JavaScript is a natural way of working with JSON. JSON parsers are perfectly capable of converting query results into variables, manipulating them, and writing them back to the database. Besides working as a client with JavaScript, the internals are also based on JavaScript. The following entities are written in JavaScript as well:

We will provide extensive examples of SPs, user-defined functions and triggers later in this book.

In traditional relational databases, the DBA or developer needs to choose the (clustered) indexes. Choosing the right indexing strategy is vital for the performance and consistency of the database.

In DocumentDB, we do not need to choose the index ourselves. In fact, all information inside a document is indexed. This means that we can query on any attribute that is available inside the document. We can choose different indexing policies, but for most applications the default indexing policy will be the best choice between performance and storage efficiency. We can reduce storage space by excluding certain paths within the document used for indexing.

The indexing process inside DocumentDB treats the documents as trees. There needs to be a top node that is the entry point for all the fields inside the document. Imagine a document containing information about a person in the following JSON representation:

{
  "firstname": "John",
  "lastname": "Doe",
  "dob", "01-01-1960",
  "hobbies":
  [
    { "type":"sports", "description":"soccer"},
    { "type":"reading", "preferences":
      [
        { "type":"scifi"},
        { "type":"thriller"}
      ]
    }
  ]
}

This JSON snippet describes a person, John Doe, who was born on January 1, 1960, and has two hobbies, sports and reading. His reading hobby focuses on the sci-fi and thriller genres.

A JSON document can be depicted like this:

The blue squares are nodes that are implicitly added by the system and do not influence our data model. The figure shows that documents are internally represented as trees. As you can see, the nodes that describe a hobby do not necessary have to be the same in schema. Go ahead and try to build this model in a traditional relational database system!

Microsoft offers DocumentDB as part of their online offerings on the Microsoft Azure platform. Their as-a-service approach enables developers to start using new technologies immediately.

The performance of our DocumentDB system is influenced by a performance level. Performance levels are set on a collection and not a database. This enables fine-tuning of your environment, giving the appropriate performance boost to the right resources. Setting the performance level influences the number of so-called request units. A request unit is a measure for the resources (CPU, memory) needed to perform a certain operation.

There are three performance levels:

We need to choose the performance level carefully, since it comes with a price impact. We will discuss the pricing of DocumentDB later in this chapter.

DocumentDB also supports transactions providing Atomicity, Consistency, Isolation, Durability (ACID) guarantees. Atomicity enables all operations to be executed as a single piece of work, all being committed at once or not at all. Consistency implies that all data is in the right state across transactions. Isolation makes sure that transactions do not interfere with each other, and durability ensures that all changes that are committed to the database will always be available.

Since JavaScript is executing under snapshot isolation belonging to the collection, SPs and triggers are executed within the same scope, enabling ACID for all operations inside SPs and triggers. If an error occurs in the JavaScript logic, the transaction is automatically rolled back.

Now that we have seen a little of DocumentDB, how can we decide whether DocumentDB is applicable for our own problem scenario? In which scenarios is it a good fit and are there any trade-offs?

The main entry point is obviously a DocumentDB account. You need to have an account before you can start working with it. An account can contain databases and, as part of a preview feature, an amount of blob storage for attachments.

All resources are accessible through a logical URI, for example, the database with the name persons can be addressed through the logical URI /dbs/persons and the document describing the person John Doe, which has an ID of 12345, can be addressed by the local URI /dbs/persons/<collectionid>/docs/12345.

A database is a container where documents are stored inside collections. Part of the database also forms a user's container. The user's container has a set of permissions, and the permissions to access collections, UDFs, triggers and SP are set on a database level. We can grant access to users in a fine-grained manner for accessing collections and documents.

A database is scaled elastically and does not need any interference from the account owner. It can scale from megabytes to petabytes. The data is stored on an SSD disk, providing fast access to your documents.

Databases are spanned implicitly across different underlying machines in order to provide the level of scaling we need.

A collection can be described as a container for all the documents, but it is also a unit of scale. Adding collections will result in some SSD storage to be allocated for that particular collection. As we saw before, setting the performance level is done on a collection level. Inside your database, you can have multiple collections, each having their own performance level (S1, S2, or S3). For example, we could have a UserProfile collection containing documents with specific profile information like addresses, images, UI preferences, and so on. This collection is queried once a user logs in to your system and the profile information is retrieved from the UserProfile collection. Imagine we have another collection containing all the products that can be ordered. This collection will be accessed more frequently, hopefully, and therefore we can set an S3 level on this collection to provide the best performance for our potential buyers.

Collections grow and shrink implicitly when documents are added or removed. There is no need to allocate space or do other preconfiguration steps.

Table storage is a non-SQL tabular based storage mechanism enabling you to store rows and columns inside a table. A table is not fixed, meaning that different rows can have different columns. Azure Table storage is a perfect fit for storing large amounts of data, although it is non-relational. There are no mechanisms like foreign keys, triggers, or user-defined functions.

MongoDB is also a document database (NoSQL), which means that it is schema-free, enables high performance and high availability, and has the ability to scale. MongoDB is open source, and is built around documents and collections. The documents are compiled of sets of key-value pairs, while collections also contain documents. Compared to DocumentDB, MongoDB uses BSON instead of JSON.

The following table provides a high-level comparison on some key features:

DocumentDB does not offer referential integrity by design. There is no concept of foreign keys. Integrity can be enforced by using triggers and SPs.

The role of the Database Administrator is still needed to manage DocumentDB. We still need someone to overlook our databases and collections. Some common tasks a DBA for a document might perform are as follows:

When you set up a DocumentDB account, you will be billed immediately. An empty account with no databases and hence no collections will be charged for a single S1 collection, at around $25 per month. The reason that you are charged while not having any collections or documents is that Microsoft needs to reserve a DNS and authorization scope for the account.

Collections are billed by the hour. Having a collection for only 10 minutes will still incur charges for a whole hour. An amount of 10 GB is included for all tiers.

The following table defines the standard characteristics per performance level:

Request units are calculated based on the amount of resources that are needed for the operation performed. When more CPU, IO, and memory is needed for a certain operation, more request units are calculated. The number of request units needed for each operation is returned in the response's header (x-ms-request-charge). By reading this value, you can keep track of the usage. If you exceed the number of request units, additional operations will be throttled.

To have fine-grained control over the performance of your collections, you could do the housekeeping of consumed Requests Units (RUs) yourself and check if you often exceed the maximum number of RUs. If so, upgrading the performance level for your collection might be a good idea.

By default, a collection is provisioned with 10 GB of storage. Documents consume storage space, but indexes also fill up the space of a collection. If you need more storage space, you need to create a different collection.

Microsoft offers the ability to contact Azure support from the Azure blade portal (portal.azure.com). If you need specific adjustments that you cannot manage from the portal or that are not supported by default, contact Azure support.

Click on the New Support button and follow the wizard that shows up. Make sure you choose the Quotas request type and enter your request details. The following table shows the limits that can be stretched by Azure support:

To create a DocumentDB account, you need to go to the Microsoft Azure portal. If you don't have a Microsoft Azure account yet, you can get a trial version at https://azure.microsoft.com/en-us/pricing/free-trial/.

After logging in to the Azure portal, go ahead and create your first DocumentDB account. For now, you only need to come up with a name.

After clicking on the Create button, your DocumentDB account will be provisioned. This process might take some time to finish.

After provisioning, your account is ready for use. Select the account you have just created and you will get an overview.

On the overview blade of your account, you will see a lot of information. For now, the most important information is located in the settings blade on the right-hand side. From this blade, you can retrieve keys and a connection string. We need this information if we want to start building the console application. Select the DocumentDB option, copy the URI, and copy the primary key.

In order to be able to create collections, we need a database first. Creating a database is straightforward as it only needs a name as input. Click the Add Database button and enter a meaningful name. After selecting OK, your database is provisioned. On the left blade you can scroll down and locate your new database.

As we have seen before, a collection is created inside a database. Selecting your database gives you the ability to add a collection. When the Add Collection option is selected, you need to pick the right performance level (or tier). For this demo, the S1 tier is more than sufficient.

Now that we have our DocumentDB account, a database, and a collection, we can start building our first application.

This sample is built using Visual Studio 2015. If you do not have Visual Studio 2015, you can download the free version Visual Studio 2015 Community from https://www.visualstudio.com/en-us/products/visual-studio-express-vs.aspx.

Saving a document

Now that we have set up a solution, created a project, and enabled the right .NET library to manage DocumentDB, we are going to write some C# code.

Note

Keep in mind that although the code samples in this book are mostly in C#, you can also use the programming language of your choice. There are SDKs available for multiple platforms (Java, Python, Node.js, and JavaScript). If yours is not supported, you could always use the REST API.

• Add the following using statements to the top of the program.cs file:

  using Microsoft.Azure.Documents;
  using Microsoft.Azure.Documents.Client;
  using Microsoft.Azure.Documents.Linq;
  using Newtonsoft.Json;

• We need the URI and primary key that we retrieved in the previous paragraph.

After writing a few lines of C# code, we have the code snippet ready. It performs the following tasks:

• Makes a connection to the DocumentDB account

• Finds the database that is created in the portal

• Creates a collection named testdevicehub

• Saves a document to this collection

The code is as follows:

private static async Task CreateDocument()
  {
    //attach to DocumentDB using the URI and Key from the Azure portal
    DocumentClient client = new DocumentClient(new Uri(docDBUri), key);
    //query for the right database inside the DocDB account
    Database database = client.CreateDatabaseQuery().Where(d => d.Id == "devicehub").AsEnumerable().FirstOrDefault();
    //find the right collection where we want to add the document
    DocumentCollection collection = client.CreateDocumentCollectionQuery((String)database.SelfLink).
        ToList().Where(cl => cl.Id.Equals("testdevicehub")).FirstOrDefault();
    //create a simple document in the collection by providing the DocumentsLink and the object to be serialized
    //and stored
    await client.CreateDocumentAsync(
      collection.DocumentsLink,
      new PersonInformation
      {
        FirstName = "Riccardo",
        LastName = "Becker",
        DateOfBirth = new DateTime(1974, 12, 21)
      }
    );
  }

Replace the values of docDBUri and the key with your information and run the console application. You have just created your first document.

Now, go to the Azure portal again and open the query documents screen. You need to select the designated collection to enable this option. Running the base query returns the document that we have just created:

select * from c

Here's the screenshot:

As you can see, the document contains more than just the fields from the class PersonInformation. Here is a brief explanation of these fields:

• id: This is the unique identifier for the document. In the application we have just created, the ID is automatically generated and is represented by a GUID.

• _rid: This resource ID is an internally used property.

• _ts: This is a property generated by the system, and it contains a timestamp.

• _self: This is generated by the system, and it contains a unique URI pointing to this resource document.

• _etag: This is a system-generated property containing an ETag that can be used for optimistic concurrency scenarios (if somebody updates the same document in the meantime, the ETag will differ and your update will fail).

• _attachments: This is generated by the system, and it contains the path for the attachments resource belonging to this document.