More images are even better

We saw how using graphs for the distribution of each feature we can get very interesting insights into the data. To make easier our observations, we grouped each feature presented on train and test data as well as, for train data only, on Survived / Not Survived. We then experimented with feature engineering to get useful, more relevant features. While observing variables separately can help us to get an initial image of data distribution, by grouping values and looking to more than one feature at a time can reveal correlations and more insights in how different features are interacting. In the following will use various graphics to explore more such correlations of features while we also explore the visualizations options. We keep for now our initial option for using a combination of matplotlib and seaborn graphical libraries.Figure 3.15. shows the number of passengers / Age interval, grouped by Passenger Class. We can see from this image that in 3^rd class...

What is in a name?

We follow now with the analysis including the Name into the data we are processing to extract meaningful information. From our initial visual inspection, we understood that all names have a similar structure. It starts with a Family Name, followed by comma, then it is a Title (short version, followed by a point), a Given Name and, for those that by marriage acquired a new name, the old or maiden name. Let’s process the data to extract these information. The code is given in the lines below.

def parse_names(row):
    try:
        text = row["Name"]
        split_text = text.split(",")
        family_name = split_text[0]
        next_text = split_text[1]
        split_text = next_text.split(".")
        title = split_text[0] + "."
        next_text = split_text[1]
        if "(" in next_text:
            split_text = next_text.split("(")
            given_name = split_text[0]
            maiden_name ...

Aggregated view of various features

We explored the categorical, numerical data as well as the text data. We learned how to extract various features from text data, and we build aggregated features from some of the numerical ones. Let’s now build two more features by grouping the Title and the Family Size. We will create two new features:

• Titles – by clustering together titles that are similar (like Miss. with Mlle. or Mrs. and Mme.) or rare (like Dona., Don., Capt., Jonkheer., Rev., the Countess.) and keeping the most frequent ones: Mr., Mrs., Master. And Miss.
• Family Type – create three clusters from the Family Size values, Single for Family Size of 1, Small (for families to up to 4 members) and Large (for families with more than 4 members)

Then, we represent on a single graph several simple or derived features that we learned have an important predictive value (see Figure 3.26).

Figure 3.25. Passengers survival rates for different features (original or derived...

A short intro into model building

As a result of our data analysis, we were able to identify some of the features with predictive value. We can now build a model that is using this knowledge. We start with a model that will use just two out of many features we investigated. This is called a baseline model and it is used as a starting point for the incremental refinement of the solution.

from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.ensemble import RandomForestClassifier
# convert categorical data in numerical
for dataset in [train_df, test_df]:
    dataset['Sex'] = dataset['Sex'].map( {'female': 1, 'male': 0} ).astype(int)
# train-validation split (20% validation)
VALID_SIZE = 0.2
train, valid = train_test_split(train_df, test_size=VALID_SIZE, random_state=42, shuffle=True)
# define predictors and target feature (labels)
predictors = ["Sex", "Pclass"]
target = 'Survived&apos...

Summary

In this chapter we started our travel around the data world on board of Titanic. We started with a preliminary statistical analysis of each feature, we continued with univariate analysis, feature engineering to create derived or aggregated features. We extracted multiple features from text, and we also created complex graphs, to visualize multiple features at the same time and reveal their predictive value. We also learned how to assign a uniform visual identity for our analysis by using a custom colormap across the Notebook. For some of the features, most notably those derived from Names, we perform a deep-dive exploration, to learn about the fate of large families on Titanic or name distribution according to the embarking port. Some of the analysis and visualization tools are easy reusable and in one of the next chapters we will see how to extract them to be as utility scripts in other Notebooks as well.

References

1. Every Pub in England, Kaggle Datasets: https://www.kaggle.com/datasets/rtatman/every-pub-in-england
2. Every Pub in England – Data Exploration, Kaggle Notebook: https://github.com/PacktPublishing/Developing-Kaggle-Notebooks/blob/develop/Chapter-04/every-pub-in-england-data-exploration.ipynb
3. Starbucks Locations Worldwide, Kaggle Datasets: https://www.kaggle.com/datasets/starbucks/store-locations
4. Open Postcode Geo, Kaggle Datasets: https://www.kaggle.com/datasets/danwinchester/open-postcode-geo
5. GADM Data for UK, Kaggle Datasets: https://www.kaggle.com/datasets/gpreda/gadm-data-for-uk
6. Starbucks Location Worldwide – Data Exploration, Kaggle Notebook: https://github.com/PacktPublishing/Developing-Kaggle-Notebooks/blob/develop/Chapter-04/starbucks-location-worldwide-data-exploration.ipynb
7. Polygon overlay in Leaflet Map: https://stackoverflow.com/questions/59303421/polygon-overlay-in-leaflet-map
8. Geopandas area: https...