Chapter 5. Importing Data into Neo4j

Database management systems are meaningless tools without their data. It really goes without saying. So how do we get data into a graph database management system like Neo4j? There really is no one specific answer, and in this chapter, we will try to give you a couple of pointers towards potential solutions.

It is important to point out that this has a very immediate and direct link to the previous chapter. After all, importing data without a model is pointless—it would never serve a real-world purpose, as it would never be able to answer the data queries of the user. As we previously indicated, but it does not hurt pointing it out again, graph models are tremendously important, and you need to think these through before you consider importing the data. We will, however, assume that you have taken this advice to heart and get on with the question of how to import data into Neo4j.

We will cover the following topics:

• Alternative approaches to importing data...

Alternative approaches to importing data into Neo4j

The first thing everyone should understand is that in a connected world, importing data is, per definition, more difficult to do. It is a true knot that is terribly difficult to untie for many different reasons, but that does not mean that we cannot untie it!

Logically, the problem of importing connected data is technically more difficult than with unconnected data structures. Importing unconnected data (for example, the nodes of your graph model) is always easy/easier. Just dump it all in there. However, you then come to importing the connections and relationships, and you find that there's no such thing as an external entity (also known as the database schema) that will ensure the consistency and connectedness of the import. You have to do this yourself, and explicitly by importing the relationships between the following:

• A start node that you have to find

• An end node that you have to look up

This process is just inherently more complicated...

Importing small(ish) datasets

In this section, we will give you a few examples of how you can import small(ish) datasets into Neo4j. Small-ish means comfortably importing anything from a few hundred nodes to a few hundred thousand nodes and relationships and importing larger datasets with considerable patience.

We will present three approaches in this section:

• Importing using spreadsheets

• Importing using Neo4j-shell-tools

• Importing using Load CSV

All of these approaches have specific pros and cons, and it depends on your specific situation to choose your most appropriate option.

Importing data using spreadsheets

Many people work with spreadsheets and are comfortable manipulating their data in this environment. This is why for smaller datasets, this is often a very suitable way of importing your data. The process is very simple:

Spreadsheet import process

This approach works in most of the common spreadsheet solutions out there (Microsoft Excel, Open Office Calc, Google Sheets, Apple Numbers) as it...

Scaling the import

Many users of Neo4j need to import larger datasets into Neo4j, at least for their initial startup use cases. Doing so can be difficult using any of the previous techniques and takes a long time. Although there are a number of things that you can tweak (for example, the batch sizes in Neo4j-shell-tools), there is a limit to the transactional write performance that you will get from running the Neo4j server. This limit is mostly I/O driven because of the transactional qualities of the Neo4j database management system; it basically needs to go down to disk at every commit and can take some time.

This is why Neo Technology and its community have developed an alternative way of creating Neo4j data stores without having the Neo4j server running. This allows the import process to be executed in an all or nothing fashion, without doing intermediate commits on the underlying component transactions. The import will either succeed or fail in its entirety. This nontransactional approach...

Questions and answers

Q1. Neo4j comes with a single, universal data import toolset that will allow you to import any of your datasets, small or large.

1. True

2. False

Answer: False. Importing data into a graph database will require you to think about your import use case and then choose the right tool for the job. Luckily, Neo4j comes with a number of different options that you can choose from.

Q2. Which data formats can you import into Neo4j?

1. Spreadsheets

2. CSV or TSV files

3. GEOFF or GraphML files

4. All of the above

Answer: All of the above. Different tools will allow you to import different formats more or less easily, but all are possible without a doubt.

Q3. The Neo4j batch importer is much faster than any of the other import technologies presented because:

1. It was written in C++ instead of Java

2. It does not write to a running transactional instance of Neo4j and can therefore run un-transactionally

3. It has fewer bugs, which allows it to run faster

4. It uses fancy caching technologies to make it write faster

Answer...

Summary

This concludes our overview of different data import technologies for the Neo4j database management system. We have presented you with an approach to choose the right import technologies, and then, in some detail, we have explored how at least some of these technologies can be used on a real dataset. There are, of course, other import techniques that we did not discuss. More specifically, a programmatic import using custom software could be useful for you. You should visit some of the Neo4j blogs such as http://maxdemarzi.com or http://jexp.de/blog/ if you are interested in this.

This chapter should give you enough confidence to now start working with your own data domain in the context of Neo4j.

In the next few chapters, we will start applying what we have learned by looking at different use case scenarios that Neo4j is frequently used for and discussing them in some detail.