Chapter 7: Text Analytics and Natural Language Processing Using Graphs

Nowadays, a vast amount of information is available in the form of text in terms of natural written language. The very same book you are reading right now is one such example. The news you read every morning, the tweets or the Facebook posts you sent/read earlier, the reports you write for a school assignment, the emails we write continuously – these are all examples of information we exchange via written documents and text. It is undoubtedly the most common way of indirect interaction, as opposed to direct interaction such as talking or gesticulating. It is, therefore, crucial to be able to leverage such kinds of information and extract insights from documents and texts.

The vast amount of information present nowadays in this form has determined the great development and recent advances in the field of natural language processing (NLP).

In this chapter, we will show you how to process natural language...

Technical requirements

We will be using Python 3.8 for all our exercises. The following is a list of Python libraries that you must install for this chapter using pip. To do this, run, for example, pip install networkx==2.4 on the command line and so on:

networkx==2.4 
scikit-learn==0.24.0
stellargraph==1.2.1
spacy==3.0.3
pandas==1.1.3
numpy==1.19.2
node2vec==0.3.3
Keras==2.0.2
tensorflow==2.4.1
communities==2.2.0
gensim==3.8.3
matplotlib==3.3.4
nltk==3.5
fasttext==0.9.2

All the code files relevant to this chapter are available at https://github.com/PacktPublishing/Graph-Machine-Learning/tree/main/Chapter07.

Providing a quick overview of a dataset

To show you how to process a corpus of documents with the aim of extracting relevant information, we will be using a dataset derived from a well-known benchmark in the field of NLP: the so-called Reuters-21578. The original dataset includes a set of 21,578 news articles that were published in the financial Reuters newswire in 1987, which were assembled and indexed in categories. The original dataset has a very skewed distribution, with some categories appearing only in the training set or in the test set. For this reason, we will use a modified version, known as ApteMod, also referred to as Reuters-21578 Distribution 1.0, that has a smaller skew distribution and consistent labels between the training and test datasets.

Even though these articles are a bit outdated, the dataset has been used in a plethora of papers on NLP and still represents a dataset that's often used for benchmarking algorithms.

Indeed, Reuters-21578 contains enough...

Understanding the main concepts and tools used in NLP

When processing documents, the first analytical step is certainly to infer the document language. Most analytical engines that are used in NLP tasks are, in fact, trained on documents in a specific language and should only be used for such a language. Some attempts to build cross-language models (see, for instance, multi-lingual embeddings such as https://fasttext.cc/docs/en/aligned-vectors.html and https://github.com/google-research/bert/blob/master/multilingual.md) have recently gained increasing popularity, although they still represent a small portion of NLP models. Therefore, it is very common to first infer the language so that you can use the correct downstream analytical NLP pipeline.

You can use different methods to infer the language. One very simple yet effective approach relies on looking for the most common words of a language (the so-called stopwords, such as the, and, be, to, of, and so on) and building a score...

Creating graphs from a corpus of documents

In this section, we will use the information we extracted in the previous section using the different text engines to build networks that relate the different information. In particular, we will focus on two kinds of graphs:

• Knowledge-based graphs, where we will use the semantic meaning of sentences to infer relationships between the different entities.
• Bipartite graphs, where we will be connecting the documents to the entities that appear in the text. We will then project the bipartite graph into a homogeneous graph, which will be made up of either document or entity nodes only.

Knowledge graphs

Knowledge graphs are very interesting as they not only relate entities but also provide a direction and a meaning to the relationship. For instance, let's take a look at the following relationship:

I (->) buy (->) a book

This is substantially...

Building a document topic classifier

To show you how to leverage a graph structure, we will focus on using the topological information and the connections between the entities provided by the bipartite entity-document graph to train multi-label classifiers. This will help us predict the document topics. To do this, we will analyze two different approaches:

• A shallow machine-learning approach, where we will use the embeddings we extracted from the bipartite network to train traditional classifiers, such as a RandomForest classifier.
• A more integrated and differentiable approach based on using a graphical neural network that's been applied to heterogeneous graphs (such as the bipartite graph).

Let's consider the first 10 topics, which we have enough documentation on to train and evaluate our models:

from collections import Counter
topics = Counter(
    [label 
     for document_labels in corpus["label"...

Summary

In this chapter, you learned how to process unstructured information and how to represent such information by using graphs. Starting from a well-known benchmark dataset, the Reuters-21578 dataset, we applied standard NLP engines to tag and structure textual information. Then, we used these high-level features to create different types of networks: knowledge-based networks, bipartite networks, and projections for a subset of nodes, as well as a network relating the dataset topics. These different graphs have also allowed us to use the tools we presented in previous chapters to extract insights from the network representation.

We used local and global properties to show you how these quantities can represent and describe structurally different types of networks. We then used unsupervised techniques to identify semantic communities and cluster documents that belong to similar subjects/topics. Finally, we used the labeled information provided in a dataset to train supervised...