Hybrid Deep Learning Methods

In this chapter, we will talk about some of the hybrid deep learning techniques that combine the data-level (Chapter 7, Data-Level Deep Learning Methods) and algorithm-level (Chapter 8, Algorithm-Level Deep Learning Techniques) methods in some ways. This chapter contains some recent and more advanced techniques that can be challenging to implement, so it is recommended to have a good understanding of the previous chapters.

We will begin with an introduction to graph machine learning, clarifying how graph models exploit relationships within data to boost performance, especially for minority classes. Through a side-by-side comparison of a Graph Convolutional Network (GCN), XGBoost, and MLP models, using an imbalanced social network dataset, we will highlight the superior performance of the GCN.

We will continue to explore strategies to tackle class imbalance in deep learning, examining techniques that manipulate data distribution and prioritize challenging...

Technical requirements

Similar to prior chapters, we will continue to utilize common libraries such as numpy, pandas, sklearn, and torch. For graph machine learning, we will use the torch_geometric library as well. The code and notebooks for this chapter are available on GitHub at https://github.com/PacktPublishing/Machine-Learning-for-Imbalanced-Data/tree/master/chapter09. You can open the GitHub notebook using Google Colab by clicking on the Open in Colab icon at the top of the chapter’s notebook or by launching it from https://colab.research.google.com, using the GitHub URL of the notebook.

Using graph machine learning for imbalanced data

In this section, we will see when graphs can be useful tools in machine learning, when to use graph ML models in general, and how they can be helpful on certain kinds of imbalanced datasets. We’ll also be exploring how graph ML models can outperform classical models such as XGBoost on certain imbalanced datasets.

Graphs are incredibly versatile data structures that can represent complex relationships and structures, from social networks and web pages (think of links as edges) to molecules in chemistry (consider atoms as nodes and the bonds between them as edges) and various other domains. Graph models allow us to represent the relationships in data, which can be helpful to make predictions and gain insights, even for problems where the relationships are not explicitly defined.

Understanding graphs

Graphs are the foundation of graph ML, so it’s important to understand them first. In the context of computer science...

Hard example mining

Hard example mining is a technique in deep learning that forces the model to pay more attention to these difficult examples, and to prevent overfitting to the majority of the samples that are easy to predict. To do this, hard example mining identifies and selects the most challenging samples in the dataset and then backpropagates the loss incurred only by those challenging samples. Hard example mining is often used in computer vision tasks such as object detection. Hard examples can be of two kinds:

• Hard positive examples are the correctly labeled examples with low prediction scores
• Hard negative examples are incorrectly labeled examples with high prediction scores, which are obvious mistakes made by the model

The term “mining” refers to the process of finding such examples that are “hard.” The idea of hard negative mining is not really new and is quite similar to the idea of boosting, on which the popular algorithms...

Minority class incremental rectification

Minority class incremental rectification is a deep learning technique that boosts the representation of minority classes in imbalanced datasets using a Class Rectification Loss (CRL). This strategy dynamically adjusts to class imbalance, enhancing model performance by incorporating hard example mining and other methods.

This technique is based on the paper by Dong et al. [5][6]. Here are the main steps of the technique:

1. Class identification in each batch:
   • Binary classification: We consider a class as a minority if it makes up less than 50% of the batch. The rest is the majority class.
   • Multi-class classification: We define all minority classes as those that collectively account for no more than 50% of the batch. The remaining classes are treated as majority classes.
2. Compute the class rectification loss:
   • Locate challenging samples:
     • Find hard positives: We identify samples from the minority class that our model incorrectly assesses with...

Summary

In this chapter, we were introduced to graph ML and saw how it can be useful for certain imbalanced datasets. We trained and compared the performance of the GCN model with baselines of XGBoost and MLP on the Facebook page-page dataset. For certain datasets (including tabular ones), where we are able to leverage the rich and interconnected structure of the graph data, the graph ML models can beat even XGBoost models. As we continue to encounter increasingly complex and interconnected data, the importance and relevance of graph ML models will only continue to grow. Understanding and utilizing these algorithms can be invaluable in your arsenal.

We then went over a hard mining technique, where the “hard” examples with the lowest loss values are first identified. Then, the loss for only k such examples is backpropagated in order to force a model to focus on the minority class examples, which the model has the most trouble learning about. Finally, we deep-dived into...

Questions

1. Apply triplet loss to the imbalanced MNIST dataset, and see whether the model’s performance is better than using the cross-entropy loss function.
2. Apply minority class incremental rectification technique to the imbalanced datasets – CIFAR10-LT and CIFAR100-LT. For a reference implementation of this technique on the MNIST-LT dataset, you can refer to the accompanying GitHub notebook.

References

1. Fraud Detection: Using Relational Graph Learning to Detect Collusion (2021): https://www.uber.com/blog/fraud-detection/.
2. Graph for fraud detection (2022): https://engineering.grab.com/graph-for-fraud-detection.
3. A. Shrivastava, A. Gupta, and R. Girshick, “Training Region-Based Object Detectors with Online Hard Example Mining,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, Jun. 2016, pp. 761–769: doi: 10.1109/CVPR.2016.89.
4. Marius Schmidt-Mengin, Théodore Soulier, Mariem Hamzaoui, Arya Yazdan-Panah, Benedetta Bod-ini, et al. “Online hard example mining vs. fixed oversampling strategy for segmentation of new multiple sclerosis lesions from longitudinal FLAIR MRI”. Frontiers in Neuroscience, 2022, 16, pp.100405. 10.3389/fnins.2022.1004050. hal-03836922.
5. Q. Dong, S. Gong, and X. Zhu, “Class Rectification Hard Mining for Imbalanced Deep Learning,” in 2017...