NLU enables the processing of unstructured text and extracts meaning and critical pieces of information that are actionable. Enabling a computer to understand sentences of text is a very hard challenge. One aspect of NLU is understanding the meaning of sentences. Sentiment analysis of a sentence becomes possible after understanding the sentence. Another useful application is the classification of sentences to a topic. This topic classification can also help in the disambiguation of entities. Consider the following sentence: "A CNN helps improve the accuracy of object recognition." Without understanding that this sentence is about machine learning, an incorrect inference may be made about the entity CNN. It may be interpreted as the news organization as opposed to a deep learning architecture used in computer vision. An example of a sentiment analysis model is built using a specific RNN architecture called BiLSTMs later in this chapter.

Another aspect of NLU is to extract information or commands from free-form text. This text can be sourced from converting speech, as spoken to Amazon's Echo device, for example, into text. Rapid advances in speech recognition now allow considering speech as equivalent to text. Extracting commands from the text, like an object and an action to perform, allows control of devices through voice commands. Consider the example sentence "Lower the volume." Here, the object is "volume" and the action is "lower." After extraction from text, these actions can be matched to a list of available actions and executed. This capability enables advanced human-computer interaction (HCI), allowing control of home appliances through voice commands. NER is used for detecting key tokens in sentences.

This technique is incredibly useful in building form filling or slot filling chatbots. NER also forms the basis of other NLU techniques that perform tasks such as relation extraction. Consider the sentence "Sundar Pichai is the CEO of Google." In this sentence, what is the relationship between the entities "Sundar Pichai" and "Google"? The right answer is CEO. This is an example of relation extraction, and NER was used to identify the entities in the sentence. The focus of the next chapter is on NER using a specific architecture that has been quite effective in this space.

A common building block of both sentiment analysis and NER models is Bi-directional RNN models. The next section describes BiLSTMs, which is Bi-directional RNN using LSTM units, prior to building a sentiment analysis model with it.

LSTMs are one of the styles of recurrent neural networks, or RNNs. RNNs are built to handle sequences and learn the structure of them. An RNN does that by using the output generated after processing the previous item in the sequence along with the current item to generate the next output.

Mathematically, this can be expressed like so:

This equation says that to compute the output at time t, the output at t-1 is used as an input along with the input data x_t at the same time step. Along with this, a set of parameters or learned weights, represented by , are also used in computing the output. The objective of training an RNN is to learn these weights This particular formulation of an RNN is unique. In previous examples, we have not used the output of a batch to determine the output of a future batch. While we focus on applications of RNNs on language where a sentence is modeled as a sequence of words appearing one after the other, RNNs can be applied to build general time-series models.

RNN building blocks

The previous section outlined the basic mathematical intuition of a recursive function that is a simplification of the RNN building block. Figure 2.1 represents a few time steps and also adds details to show different weights used for computation for a basic RNN building block or cell.

Figure 2.1: RNN unraveled

The basic cell is shown on the left. The input vector at a specific time or sequence step t is multiplied by a weight vector, represented in the diagram as U, to generate an activation in the middle part. The key part of this architecture is the loop in this activation part. The output of a previous step is multiplied by a weight vector, denoted by V in the figure, and added to the activation. This activation can be multiplied by another weight vector, represented by W, to produce the output of that step shown at the top. In terms of sequence or time steps, this network can be unrolled. This unrolling is virtual. However, it is represented on the right side of the figure. Mathematically, activation at time step t can be represented by:

Output at the same step can be computed like so:

Structurally, the network is very simple as it is a single unit. To exploit and learn the structure of inputs passing through, weight vectors U, V, and W are shared across time steps. The network does not have layers as seen in fully connected or convolutional networks. However, as it is unrolled over time steps, it can be thought of as having as many layers as steps in the input sequences. There are additional criteria that would need to be satisfied to make a Deep RNN. More on that later in this section. These networks are trained using backpropagation and stochastic gradient descent techniques. The key thing to note here is that backpropagation is happening through the sequence or time steps before backpropogating through layers.

Having this structure enables processing sequences of arbitrary lengths. However, as the length of sequences increases, there are a couple of challenges that emerge:

• Vanishing and exploding gradients: As the lengths of these sequences increase, the gradients going back will become smaller and smaller. This will cause the network to train slowly or not learn at all. This effect will be more pronounced as sequence lengths increase. In the previous chapter, we built a network of a handful of layers. Here, a sentence of 10 words would equate to a network of 10 layers. A 1-minute audio sample of 10 ms would generate 6,000 steps! Conversely, gradients can also explode if the output is increasing. The simplest way to manage vanishing gradients is through the use of ReLUs. For managing exploding gradients, a technique called gradient clipping is used. This technique artificially clips gradients if their magnitude exceeds a threshold. This prevents gradients from becoming too large or exploding.
• Inability to manage long-term dependencies: Let's say that the third word in an eleven-word sentence is highly informative. Here is a toy example: "I think soccer is the most popular game across the world." As the processing reaches the end of the sentence, the contribution of the words prior earlier in the sequence will become smaller and smaller due to repeated multiplication with the vector V as shown above.
• Two specific RNN cell designs mitigate these problems: Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU). These are described next. However, note that TensorFlow provides implementations of both types of cells out of the box. So, building RNNs with these cell types is almost trivial.

Long short-term memory (LSTM) networks

LSTM networks were proposed in 1997 and improved upon and popularized by many researchers. They are widely used today for a variety of tasks and produce amazing results.

LSTM has four main parts:

• Cell value or memory of the network, also referred to as the cell, which stores accumulated knowledge
• Input gate, which controls how much of the input is used in computing the new cell value
• Output gate, which determines how much of the cell value is used in the output
• Forget gate, which determines how much of the current cell value is used for updating the cell value

These are shown in the figure below:

Figure 2.2: LSTM cell (Source: Madsen, "Visualizing memorization in RNNs," Distill, 2019)

After a quick introduction to GRUs, we will pick up on LSTMs, talk about BiLSTMs, and build a sentiment classification model.

Gated recurrent units (GRUs)

GRUs are another popular, and more recent, type of RNN unit. They were invented in 2014. They are simpler than LSTMs:

Figure 2.3: Gated recurrent unit (GRU) architecture

Compared to the LSTM, it has fewer gates. Input and forget gates are combined into a single update gate. Some of the internal cell state and hidden state is merged together as well. This reduction in complexity makes it easier to train. It has shown great results in the speech and sound domains. However, in neural machine translation tasks, LSTMs have shown superior performance. In this chapter, we will focus on using LSTMs. Before we discuss BiLSTMs, let's take a sentiment classification problem and solve it with LSTMs. Then, we will try and improve the model with BiLSTMs.

Sentiment classification with LSTMs

Sentiment classification is an oft-cited use case of NLP. Models that predict the movement of stock prices by using sentiment analysis features from tweets have shown promising results. Tweet sentiment is also used to determine customers' perceptions of brands. Another use case is processing user reviews for movies, or products on e-commerce or other websites. To see LSTMs in action, let's use a dataset of movie reviews from IMDb. This dataset was published at the ACL 2011 conference in a paper titled Learning Word Vectors for Sentiment Analysis. This dataset has 25,000 review samples in the training set and another 25,000 in the test set.

A local notebook will be used for the code for this example. Chapter 10, Installation and Setup Instructions for Code, provides detailed instructions on how to set up the development environment. In short, you will need Python 3.7.5 and the following libraries to start:

• pandas 1.0.1
• NumPy 1.18.1
• TensorFlow 2.4 and the tensorflow_datasets 3.2.1 package
• Jupyter notebook

We will follow the overall process outlined in Chapter 1, Essentials of NLP. We start by loading the data we need.

Loading the data

In the previous chapter, we downloaded the data and loaded it with the pandas library. This approach loaded the entire dataset into memory. However, sometimes data can be quite large, or spread into multiple files. In such cases, it may be too large for loading and need lots of pre-processing. Making text data ready to be used in a model requires normalization and vectorization at the very least. Often, this needs to be done outside of the TensorFlow graph using Python functions. This may cause issues in the reproducibility of code. Further, it creates issues for data pipelines in production where there is a higher chance of breakage as different dependent stages are being executed separately.

TensorFlow provides a solution for the loading, transformation, and batching of data through the use of the tf.data package. In addition, a number of datasets are provided for download through the tensorflow_datasets package. We will use a combination of these to download the IMDb data, and perform the tokenization, encoding, and vectorization steps before training an LSTM model.

The first step is to install the appropriate packages and download the datasets:

!pip install tensorflow_datasets
import tensorflow as tf
import tensorflow_datasets as tfds
import numpy as np

The tfds package comes with a number of datasets in different domains such as images, audio, video, text, summarization, and so on. To see the datasets available:

", ".join(tfds.list_builders())

'abstract_reasoning, aeslc, aflw2k3d, amazon_us_reviews, arc, bair_robot_pushing_small, beans, big_patent, bigearthnet, billsum, binarized_mnist, binary_alpha_digits, c4, caltech101, caltech_birds2010, caltech_birds2011, cars196, cassava, cats_vs_dogs, celeb_a, celeb_a_hq, cfq, chexpert, cifar10, cifar100, cifar10_1, cifar10_corrupted, citrus_leaves, cityscapes, civil_comments, clevr, cmaterdb, cnn_dailymail, coco, coil100, colorectal_histology, colorectal_histology_large, cos_e, curated_breast_imaging_ddsm, cycle_gan, deep_weeds, definite_pronoun_resolution, diabetic_retinopathy_detection, div2k, dmlab, downsampled_imagenet, dsprites, dtd, duke_ultrasound, dummy_dataset_shared_generator, dummy_mnist, emnist, eraser_multi_rc, esnli, eurosat, fashion_mnist, flic, flores, food101, gap, gigaword, glue, groove, higgs, horses_or_humans, i_naturalist2017, image_label_folder, imagenet2012, imagenet2012_corrupted, imagenet_resized, imagenette, imagewang, imdb_reviews, iris, kitti, kmnist, lfw, librispeech, librispeech_lm, libritts, lm1b, lost_and_found, lsun, malaria, math_dataset, mnist, mnist_corrupted, movie_rationales, moving_mnist, multi_news, multi_nli, multi_nli_mismatch, natural_questions, newsroom, nsynth, omniglot, open_images_v4, opinosis, oxford_flowers102, oxford_iiit_pet, para_crawl, patch_camelyon, pet_finder, places365_small, plant_leaves, plant_village, plantae_k, qa4mre, quickdraw_bitmap, reddit_tifu, resisc45, rock_paper_scissors, rock_you, scan, scene_parse150, scicite, scientific_papers, shapes3d, smallnorb, snli, so2sat, speech_commands, squad, stanford_dogs, stanford_online_products, starcraft_video, sun397, super_glue, svhn_cropped, ted_hrlr_translate, ted_multi_translate, tf_flowers, the300w_lp, tiny_shakespeare, titanic, trivia_qa, uc_merced, ucf101, vgg_face2, visual_domain_decathlon, voc, wider_face, wikihow, wikipedia, wmt14_translate, wmt15_translate, wmt16_translate, wmt17_translate, wmt18_translate, wmt19_translate, wmt_t2t_translate, wmt_translate, xnli, xsum, yelp_polarity_reviews'

That is a list of 155 datasets. Details of the datasets can be obtained on the catalog page at https://www.tensorflow.org/datasets/catalog/overview.

IMDb data is provided in three splits – training, test, and unsupervised. The training and testing splits have 25,000 rows each, with two columns. The first column is the text of the review, and the second is the label. "0" represents a review with negative sentiment while "1" represents a review with positive sentiment. The following code loads the training and testing data splits:

imdb_train, ds_info = tfds.load(name="imdb_reviews", split="train", 
                               with_info=True, as_supervised=True)
imdb_test = tfds.load(name="imdb_reviews", split="test", 
                      as_supervised=True)

Note that this command may take a little bit of time to execute as data is downloaded. ds_info contains information about the dataset. This is returned when the with_info parameter is supplied. Let's see the information contained in ds_info:

print(ds_info)

tfds.core.DatasetInfo(
    name='imdb_reviews',
    version=1.0.0,
    description='Large Movie Review Dataset.
This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well.',
    homepage='http://ai.stanford.edu/~amaas/data/sentiment/',
    features=FeaturesDict({
        'label': ClassLabel(shape=(), dtype=tf.int64, num_classes=2),
        'text': Text(shape=(), dtype=tf.string),
    }),
    total_num_examples=100000,
    splits={
        'test': 25000,
        'train': 25000,
        'unsupervised': 50000,
    },
    supervised_keys=('text', 'label'),
    citation="""@InProceedings{maas-EtAl:2011:ACL-HLT2011,
      author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},
      title     = {Learning Word Vectors for Sentiment Analysis},
      booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
      month     = {June},
      year      = {2011},
      address   = {Portland, Oregon, USA},
      publisher = {Association for Computational Linguistics},
      pages     = {142--150},
      url       = {http://www.aclweb.org/anthology/P11-1015}
    }""",
    redistribution_info=,
)

We can see that two keys, text and label, are available in the supervised mode. Using the as_supervised parameter is key to loading the dataset as a tuple of values. If this parameter is not specified, data is loaded and made available as dictionary keys. In cases where the data has multiple inputs, that may be preferable. To get a sense of the data that has been loaded:

for example, label in imdb_train.take(1):
    print(example, '\n', label)

tf.Tensor(b"This was an absolutely terrible movie. Don't be lured in by Christopher Walken or Michael Ironside. Both are great actors, but this must simply be their worst role in history. Even their great acting could not redeem this movie's ridiculous storyline. This movie is an early nineties US propaganda piece. The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions. Maria Conchita Alonso appeared phony, and her pseudo-love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning. I am disappointed that there are movies like this, ruining actor's like Christopher Walken's good name. I could barely sit through it.", shape=(), dtype=string)
tf.Tensor(0, shape=(), dtype=int64)

The above review is an example of a negative review. The next step is tokenization and vectorization of the reviews.

Normalization and vectorization

In Chapter 1, Essentials of NLP, we discussed a number of different normalization methods. Here, we are only going to tokenize the text into words and construct a vocabulary, and then encode the words using this vocabulary. This is a simplified approach. There can be a number of different approaches that can be used for building additional features. Using techniques discussed in the first chapter, such as POS tagging, a number of features can be built, but that is left as an exercise for the reader. In this example, our aim is to use the same set of features on an RNN with LSTMs followed by using the same set of features on an improved model with BiLSTMs.

A vocabulary of the tokens occurring in the data needs to be constructed prior to vectorization. Tokenization breaks up the words in the text into individual tokens. The set of all the tokens forms the vocabulary.

Normalization of the text, such as converting to lowercase, etc., is performed along with this tokenization step. tfds comes with a set of feature builders for text in the tfds.features.text package. First, a set of all the words in the training data needs to be created:

tokenizer = tfds.features.text.Tokenizer()
vocabulary_set = set()
MAX_TOKENS = 0
for example, label in imdb_train:
  some_tokens = tokenizer.tokenize(example.numpy())
  if MAX_TOKENS < len(some_tokens):
        MAX_TOKENS = len(some_tokens)
  vocabulary_set.update(some_tokens)

By iterating through the training examples, each review is tokenized and the words in the review are added to a set. These are added to a set to get unique words. Note that tokens or words have not been converted to lowercase. This means that the size of the vocabulary is going to be slightly larger. Using this vocabulary, an encoder can be created. TokenTextEncoder is one of three out-of-the-box encoders that are provided in tfds. Note how the list of tokens is converted into a set to ensure only unique tokens are retained in the vocabulary. The tokenizer used for generating the vocabulary is passed in, so that every successive call to encode a string can use the same tokenization scheme. This encoder expects that the tokenizer object provides a tokenize() and a join() method. If you want to use StanfordNLP or some other tokenizer as discussed in the previous chapter, all you need to do is to wrap the StanfordNLP interface in a custom object and implement methods to split the text into tokens and join the tokens back into a string:

imdb_encoder = tfds.features.text.TokenTextEncoder(vocabulary_set, 
                                              tokenizer=tokenizer)
vocab_size = imdb_encoder.vocab_size
print(vocab_size, MAX_TOKENS)

93931 2525

The vocabulary has 93,931 tokens. The longest review has 2,525 tokens. That is one wordy review! Reviews are going to have different lengths. LSTMs expect sequences of equal length. Padding and truncating operations make reviews of equal length. Before we do that, let's test whether the encoder works correctly:

for example, label in imdb_train.take(1):
    print(example)
    encoded = imdb_encoder.encode(example.numpy())
    print(imdb_encoder.decode(encoded))

tf.Tensor(b"This was an absolutely terrible movie. Don't be lured in by Christopher Walken or Michael Ironside. Both are great actors, but this must simply be their worst role in history. Even their great acting could not redeem this movie's ridiculous storyline. This movie is an early nineties US propaganda piece. The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions. Maria Conchita Alonso appeared phony, and her pseudo-love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning. I am disappointed that there are movies like this, ruining actor's like Christopher Walken's good name. I could barely sit through it.", shape=(), dtype=string)
This was an absolutely terrible movie Don t be lured in by Christopher Walken or Michael Ironside Both are great actors but this must simply be their worst role in history Even their great acting could not redeem this movie s ridiculous storyline This movie is an early nineties US propaganda piece The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions Maria Conchita Alonso appeared phony and her pseudo love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning I am disappointed that there are movies like this ruining actor s like Christopher Walken s good name I could barely sit through it

Note that punctuation is removed from these reviews when they are reconstructed from the encoded representations.

imdb_encoder.save_to_file("reviews_vocab")

enc = tfds.features.text.TokenTextEncoder.load_from_file("reviews_vocab")
enc.decode(enc.encode("Good case. Excellent value."))

'Good case Excellent value'

Tokenization and encoding were done for a small set of rows at a time. TensorFlow provides mechanisms to perform these actions in bulk over large datasets, which can be shuffled and loaded in batches. This allows very large datasets to be loaded without running out of memory during training. To enable this, a function needs to be defined that performs a transformation on a row of data. Note that multiple transformations can be chained one after the other. It is also possible to use a Python function in defining these transformations. For processing the review above, the following steps need to be performed:

• Tokenization: Reviews need to be tokenized into words.
• Encoding: These words need to be mapped to integers using the vocabulary.
• Padding: Reviews can have variable lengths, but LSTMs expect vectors of the same length. So, a constant length is chosen. Reviews shorter than this length are padded with a specific vocabulary index, usually 0 in TensorFlow. Reviews longer than this length are truncated. Fortunately, TensorFlow provides such a function out of the box.

The following functions perform this:

from tensorflow.keras.preprocessing import sequence
def encode_pad_transform(sample):
    encoded = imdb_encoder.encode(sample.numpy())
    pad = sequence.pad_sequences([encoded], padding='post', 
                                 maxlen=150)
    return np.array(pad[0], dtype=np.int64)  
def encode_tf_fn(sample, label):
    encoded = tf.py_function(encode_pad_transform, 
                                       inp=[sample], 
                                       Tout=(tf.int64))
    encoded.set_shape([None])
    label.set_shape([])
    return encoded, label

encode_tf_fn is called by the dataset API with one example at a time. This means a tuple of the review and its label. This function in turn calls another function, encode_pad_transform, which is wrapped in the tf.py_function call that performs the actual transformation. In this function, tokenization is performed first, followed by encoding, and finally padding and truncating. A maximum length of 150 tokens or words is chosen for padding/truncating sequences. Any Python logic can be used in this second function. For example, the StanfordNLP package could be used to perform POS tagging of the words, or stopwords could be removed as shown in the previous chapter. Here, we try to keep things simple for this example.

Transforming the data is quite trivial. Let's try the code on a small sample of the data:

subset = imdb_train.take(10)
tst = subset.map(encode_tf_fn)
for review, label in tst.take(1):
    print(review, label)
    print(imdb_encoder.decode(review))

tf.Tensor(
[40205  9679 51728 91747 21013  7623  6550 40338 18966 36012 64846 80722
 81643 29176 14002 73549 52960 40359 49248 62585 75017 67425 18181  2673
 44509 18966 87701 56336 29928 64846 41917 49779 87701 62585 58974 82970
  1902  2754 18181  7623  2615  7927 67321 40205  7623 43621 51728 91375
 41135 71762 29392 58948 76770 15030 74878 86231 49390 69836 18353 84093
 76562 47559 49390 48352 87701 62200 13462 80285 76037 75121  1766 59655
  6569 13077 40768 86201 28257 76220 87157 29176  9679 65053 67425 93397
 74878 67053 61304 64846 93397  7623 18560  9679 50741 44024 79648  7470
 28203 13192 47453  6386 18560 79892 49248  7158 91321 18181 88633 13929
  2615 91321 81643 29176  2615 65285 63778 13192 82970 28143 14618 44449
 39028     0     0     0     0     0     0     0     0     0     0     0
     0     0     0     0     0     0     0     0     0     0     0     0
     0     0     0     0     0     0], shape=(150,), dtype=int64) tf.Tensor(0, shape=(), dtype=int64)
This was an absolutely terrible movie Don t be lured in by Christopher Walken or Michael Ironside Both are great actors but this must simply be their worst role in history Even their great acting could not redeem this movie s ridiculous storyline This movie is an early nineties US propaganda piece The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions Maria Conchita Alonso appeared phony and her pseudo love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning I am disappointed that there are movies like this ruining actor s like Christopher Walken s good name I could barely sit through it

Note the "0" at the end of the encoded tensor in the first part of the output. That is a consequence of padding to 150 words.

Running this map over the entire dataset can be done like so:

encoded_train = imdb_train.map(encode_tf_fn)
encoded_test = imdb_test.map(encode_tf_fn)

This should execute really fast. When the training loop executes, the mapping will be executed at that time. Other commands that are available and useful in the tf.data.DataSet class, of which imdb_train and imdb_test are instances, are filter(), shuffle(), and batch(). filter() can remove certain types of data from the dataset. It can be used to filter out reviews above or below a certain length, or separate out positive and negative examples to construct a more balanced dataset. The second method shuffles the data between training epochs. The last one batches data for training. Note that different datasets will result if these methods are applied in a different sequence.

Performance optimization with tf.data:

Figure 2.4: Illustrative example of the time taken by sequential execution of the map function (Source: Better Performance with the tf.data API at tensorflow.org/guide/data_performance )

As can be seen in the figure above, a number of operations contribute to the overall training time in an epoch. This example chart above shows the case where files need to be opened, as shown in the topmost row, data needs to be read in the row below, a map transformation needs to be executed on the data being read, and then training can happen. Since these steps are happening in sequence, it can make the overall training time longer. Instead, the mapping step can happen in parallel. This will result in shorter execution times overall. CPU power is used to prefetch, batch, and transform the data, while the GPU is used for training computation and operations such as gradient calculation and updating weights. This can be enabled by making a small change in the call to the map function above:

encoded_train = imdb_train.map(encode_tf_fn,
       num_parallel_calls=tf.data.experimental.AUTOTUNE)
encoded_test = imdb_test.map(encode_tf_fn,
       num_parallel_calls=tf.data.experimental.AUTOTUNE)

Passing the additional parameter enables TensorFlow to use multiple subprocesses to execute the transformation on.

This can result in a speedup as shown below:

Figure 2.5: Illustrative example of a reduction in training time due to parallelization of map (Source: Better Performance with the tf.data API at tensorflow.org/guide/data_performance )

While we have normalized and encoded the text of the reviews, we have not converted it into word vectors or embeddings. This step is performed along with the model training in the next step. So, we are ready to start building a basic RNN model using LSTM now.

LSTM model with embeddings

TensorFlow and Keras make it trivial to instantiate an LSTM-based model. In fact, adding a layer of LSTMs is one line of code. The simplest form is shown below:

tf.keras.layers.LSTM(rnn_units)

Here, the rnn_units parameter determines how many LSTMs are strung together in one layer. There are a number of other parameters that can be configured, but the defaults are fairly reasonable on them. The TensorFlow documentation details these options and possible values with examples quite well. However, the review text tokens cannot be fed as is into the LSTM layer. They need to be vectorized using an embedding scheme. There are a couple of different approaches that can be used. The first approach is to learn these embeddings as the model trains. This is the approach we're going to use, as it is the simplest approach. In cases where the text data you may have is unique to a domain, like medical transcriptions, this is also probably the best approach. This approach, however, requires significant amounts of data for training for the embeddings to learn the right relationships with the words. The second approach is to use pre-trained embeddings, like Word2vec or GloVe, as shown in the previous chapter, and use them to vectorize the text. This approach has really worked well in general-purpose text models and can even be adapted to work very well in specific domains. Working with transfer learning is the focus of Chapter 4, Transfer Learning with BERT, though.

Coming back to learning embeddings, TensorFlow provides an embedding layer that can be added before the LSTM layer. Again, this layer has several options that are well documented. To complete the binary classification model, all that remains is a final dense layer with one unit for classification. A utility function that can build models with some configurable parameters can be configured like so:

def build_model_lstm(vocab_size, embedding_dim, rnn_units, batch_size):
  model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, 
                              mask_zero=True,
                              batch_input_shape=[batch_size, None]),
    tf.keras.layers.LSTM(rnn_units),
    tf.keras.layers.Dense(1, activation='sigmoid')
  ])
  return model

This function exposes a number of configurable parameters to allow trying out different architectures. In addition to these parameters, batch size is another important parameter. These can be configured as follows:

vocab_size = imdb_encoder.vocab_size 
# The embedding dimension
embedding_dim = 64
# Number of RNN units
rnn_units = 64
# batch size
BATCH_SIZE=100

With the exception of the vocabulary size, all other parameters can be changed around to see the impact on model performance. With these configurations set, the model can be constructed:

model = build_model_lstm(
  vocab_size = vocab_size,
  embedding_dim=embedding_dim,
  rnn_units=rnn_units,
  batch_size=BATCH_SIZE)
model.summary()

Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_3 (Embedding)      (100, None, 64)           6011584   
_________________________________________________________________
lstm_3 (LSTM)                (100, 64)                 33024     
_________________________________________________________________
dense_5 (Dense)              (100, 1)                  65        
=================================================================
Total params: 6,044,673
Trainable params: 6,044,673
Non-trainable params: 0
_________________________________________________________________

Such a small model has over 6 million trainable parameters. It is easy to check the size of the embedding layer. The total number of tokens in the vocabulary was 93,931. Each token is represented by a 64-dimensional embedding, which provides 93,931 X 64 = 6,011,584 million parameters.

This model is now ready to be compiled with the specification of the loss function, optimizer, and evaluation metrics. In this case, since there are only two labels, binary cross-entropy is used as the loss. The Adam optimizer is a very good choice with great defaults. Since we are doing binary classification, accuracy, precision, and recall are the metrics we would like to track during training. Then, the dataset needs to be batched and training can be started:

model.compile(loss='binary_crossentropy', 
             optimizer='adam', 
             metrics=['accuracy', 'Precision', 'Recall'])
encoded_train_batched = encoded_train.batch(BATCH_SIZE)
model.fit(encoded_train_batched, epochs=10)

Epoch 1/10
250/250 [==============================] - 23s 93ms/step - loss: 0.4311 - accuracy: 0.7920 - Precision: 0.7677 - Recall: 0.8376
Epoch 2/10
250/250 [==============================] - 21s 83ms/step - loss: 0.1768 - accuracy: 0.9353 - Precision: 0.9355 - Recall: 0.9351
…
Epoch 10/10
250/250 [==============================] - 21s 85ms/step - loss: 0.0066 - accuracy: 0.9986 - Precision: 0.9986 - Recall: 0.9985

That is a very good result! Let's compare it to the test set:

model.evaluate(encoded_test.batch(BATCH_SIZE))

    250/Unknown - 20s 80ms/step - loss: 0.8682 - accuracy: 0.8063 - Precision: 0.7488 - Recall: 0.9219

The difference between the performance on the training and test set implies there is overfitting happening in the model. One way to manage overfitting is to introduce a dropout layer after the LSTM layer. This is left as an exercise to you.

Now, let's see how BiLSTMs would perform on this task.

BiLSTM model

Building BiLSTMs is easy in TensorFlow. All that is required is a one-line change in the model definition. In the build_model_lstm() function, the line that adds the LSTM layer needs to be modified. The new function would look like this, with the modified line highlighted:

def build_model_bilstm(vocab_size, embedding_dim, rnn_units, batch_size):
  model = tf.keras.Sequential([
    tf.keras.layers.Embedding(vocab_size, embedding_dim, 
                              mask_zero=True,
                              batch_input_shape=[batch_size, None]),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(rnn_units)),
    tf.keras.layers.Dense(1, activation='sigmoid')
  ])
  return model

But first, let's understand what a BiLSTM is:

Figure 2.6: LSTMs versus BiLSTMs

In a regular LSTM network, tokens or words are fed in one direction. As an example, take the review "This movie was really good." Each token starting from the left is fed into the LSTM unit, marked as a hidden unit, one at a time. The diagram above shows a version unrolled in time. What this means is that each successive word is considered as occurring at a time increment from the previous word. Each step produces an output that may or may not be useful. That is dependent on the problem at hand. In the IMDb sentiment prediction case, only the final output is important as it is fed to the dense layer to make a decision on whether the review was positive or negative.

Due to this time unrolling, it may appear as if there are multiple hidden units. However, it is the same LSTM unit, as shown in Figure 2.2 earlier in the chapter. The output of the unit is fed back into the same unit at the next time step. In the case of BiLSTM, there is a pair of hidden units. One set operates on the tokens from left to right, while the other set operates on the tokens from right to left. In other words, a forward LSTM model can only learn from tokens from the past time steps. A BiLSTM model can learn from tokens from the past and the future.

This method allows the capturing of more dependencies between words and the structure of the sentence and improves the accuracy of the model. Suppose the task is to predict the next word in this sentence fragment:

I jumped into the …

There are many possible completions to this sentence. Further, suppose that you had access to the words after the sentence. Think about these three possibilities:

1. I jumped into the …. with only a small blade
2. I jumped into the … and swam to the other shore
3. I jumped into the … from the 10m diving board

Battle or fight would be likely words in the first example, river for the second, and swimming pool for the last one. In each case, the beginning of the sentence was exactly the same but the words from the end helped disambiguate which word should fill in the blank. This illustrates the difference between LSTMs and BiLSTMs. An LSTM can only learn from the past tokens, while the BiLSTM can learn from both past and future tokens.

This new BiLSTM model has a little over 12M parameters.

bilstm = build_model_bilstm(
  vocab_size = vocab_size,
  embedding_dim=embedding_dim,
  rnn_units=rnn_units,
  batch_size=BATCH_SIZE)
bilstm.summary()

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (50, None, 128)           12023168  
_________________________________________________________________
dropout (Dropout)            (50, None, 128)           0         
_________________________________________________________________
bidirectional (Bidirectional (50, None, 128)           98816     
_________________________________________________________________
dropout_1 (Dropout)          (50, None, 128)           0         
_________________________________________________________________
bidirectional_1 (Bidirection (50, 128)                 98816     
_________________________________________________________________
dropout_2 (Dropout)          (50, 128)                 0         
_________________________________________________________________
dense_1 (Dense)              (50, 1)                   129       
=================================================================
Total params: 12,220,929
Trainable params: 12,220,929
Non-trainable params: 0
_________________________________________________________________

If you run the model shown above with no other changes, you will see a boost in the accuracy and precision of the model:

bilstm.fit(encoded_train_batched, epochs=5)

Epoch 1/5
500/500 [==============================] - 80s 160ms/step - loss: 0.3731 - accuracy: 0.8270 - Precision: 0.8186 - Recall: 0.8401
…
Epoch 5/5
500/500 [==============================] - 70s 139ms/step - loss: 0.0316 - accuracy: 0.9888 - Precision: 0.9886 - Recall: 0.9889
bilstm.evaluate(encoded_test.batch(BATCH_SIZE))
500/Unknown - 20s 40ms/step - loss: 0.7280 - accuracy: 0.8389 - Precision: 0.8650 - Recall: 0.8032

Note that the model is severely overfitting. It is important to add some form of regularization to the model. Out of the box, with no feature engineering or use of the unsupervised data for learning better embeddings, the accuracy of the model is above 83.5%. The current state-of-the-art results on this data, published in August 2019, have an accuracy of 97.42%. Some ideas that can be tried to improve this model include stacking layers of LSTMs or BiLSTMs, with some dropout for regularization, using the unsupervised split of the dataset along with training and testing review text data to learn better embeddings and using those in the final network, adding more features such as word shapes, and POS tags, among others. We will pick up this example again in Chapter 4, Transfer Learning with BERT, when we discuss language models such as BERT. Maybe this example will be an inspiration for you to try your own model and publish a paper with your state-of-the-art results!

Note that BiLSTMs, while powerful, may not be suitable for all applications. Using a BiLSTM architecture assumes that the entire text or sequence is available at the same time. This assumption may not be true in some cases.

In the case of the speech recognition of commands in a chatbot, only the sounds spoken so far by the users are available. It is not known what words a user is going to utter in the future. In real-time time-series analytics, only data from the past is available. In such applications, BiLSTMs cannot be used. Also, note that RNNs really shine with very large amounts of data training over several epochs. The IMDb dataset with 25,000 training examples is on the smaller side for RNNs to show their power. You may find you achieve similar or better results using TF-IDF and logistic regression with some feature engineering.

This is a foundational chapter in our journey through advanced NLP problems. Many advanced models use building blocks such as BiRNNs. First, we used the TensorFlow Datasets package to load data. Our work of building a vocabulary, tokenizer, and encoder for vectorization was simplified through the use of this library. After understanding LSTMs and BiLSTMs, we built models to do sentiment analysis. Our work showed promise but was far away from the state-of-the-art results, which will be addressed in future chapters. However, we are now armed with the fundamental building blocks that will enable us to tackle more challenging problems.

Armed with this knowledge of LSTMs, we are ready to build our first NER model using BiLSTMs in the next chapter. Once this model is built, we will try to improve it using CRFs and Viterbi decoding.