Data Mining, Analysis, and Visualization

So far, we have learned about some of the core Python libraries and techniques regarding HTTP/HTTPS communication, reading content, browser automation, and more from a data extraction perspective.

Data is the new oil (we all agree about this), but solely obtaining or collecting data does not provide any significant value. Collected data is stored in files (JSON, CSV, and XML), databases, and more. Stored data needs to be identified, searched, arranged, cleaned, transformed, explored, or modeled using algorithms and can sometimes be used by many services and applications before there’s any profit from the information from it.

Various technologies and concepts are involved in identifying and collecting data and processing it in order to extract some value. Data analysis implements and executes logic and algorithms using data-related applications and tools to generate valuable information. Visualization, on the other hand, displays...

Technical requirements

A web browser (Google Chrome or Mozilla Firefox) will be required, and we will be using JupyterLab for Python code.

Please refer to the Setting things up and Creating a virtual environment sections of Chapter 2 to continue with setting up and using the environment we have created. Refer to the following links to install and upgrade the required libraries:

• pandas : https://pandas.pydata.org/docs/getting_started/install.html
• ydata_profiling : https://ydata-profiling.ydata.ai/docs/master/pages/getting_started/installation.html
• plotly : https://plotly.com/
• wordcloud : https://pypi.org/project/wordcloud/

The Python libraries that are required for this chapter are as follows:

• csv
• json
• sqlite3

The code files for this chapter are available online in the book’s GitHub repository: https://github.com/PacktPublishing/Hands-On-Web-Scraping-with-Python-Second-Edition/tree/main/Chapter10.

Introduction to data mining

The term “mining” normally means the extraction or the process of extracting something. Data mining is the process of extracting data or discovering information from data. Data mining is a growing and ever-developing concept that discovers hidden, unexpected, and other various forms of information from datasets or databases, which helps in KD and decision-making.

In terms of data, mining is used as a form of analysis to discover patterns, hidden facts, and more. When knowledge is discovered using mining techniques, this is known as knowledge discovery in databases (KDD or knowledge discovery and data mining). There are plenty of terms used to describe data mining, such as KDD, information harvesting, pattern discovery from databases, and many more; although the final results are the same, these terms differ in the steps and processing architecture.

Important note

KDD is an almost cyclical process that has data mining as one of its...

Handling collected data

The availability of data is the main concern before attempting to process it for information and pattern detection. Handling collected data normally refers to gathering data in files and databases in some format and using it effectively and efficiently.

There are many tools and applications that handle data. Choosing the right tool or way of storing and using data shows your professionalism as a developer.

In the following sections, you will be learning about concepts related to handling files and dealing with types of files (JSON and CSV) that are in huge demand in the market and are associated with a large number of IT-driven applications.

Basic file handling

File handling is the core or basic technique for storing and reading data from files. This technique of handling and managing data is used a lot in various programming languages. File handling does not require additional software or tools unless some application extensions are used. Formatting...

Data analysis and visualization

Python programming is popular because of its easy usage and the availability of libraries for scientific computing, text computation, data analysis, machine learning, and much more. Data analysis is a systematic process. Unknown facts, hidden patterns, summary data, and a lot of other information can be obtained using data analysis. Data analysis is also treated as a subset of data science, and it has been booming with the use of Python and its features.

In this section, we will be analyzing some datasets, exploring some of the important features of pandas, and visualizing the results using plotly.

Analyzing data generally involves a few basic steps:

1. Identify: Identify the source of data or the origin of data, such as a website, PDF file, or image.
2. Collect: Collect the identified data using scraping or other techniques. Storing data is also important here.
3. Clean: Preprocess and clean the collected data. Clean data is easier to process...

Summary

Generating and gathering information using different analysis techniques and using it for decision-making is a growing field. Fields such as business intelligence (BI), AI, and ML require, and use, various data analysis techniques. Python programming provides a great infrastructure for the processes of data collection, data processing, information abstraction, and knowledge discovery. Libraries such as pandas, NumPy, csv, json, and plotly are the core Python libraries of the overall systematic process.

A practical introduction to the concepts related to data mining, data analysis, EDA, and data visualization was the main agenda of this chapter.

In the next chapter, we will be learning about machine learning and web scraping.

Further reading

• Data Mining:
  • https://www.ibm.com/topics/data-mining
  • https://www.springboard.com/blog/data-science/data-mining-python-tutorial/
  • https://www.investopedia.com/terms/d/datamining.asp
• Data Analysis:
  • https://monkeylearn.com/data-analysis/
  • https://www.datapine.com/blog/data-analysis-methods-and-techniques/
  • https://www.investopedia.com/terms/d/data-analytics.asp
• Exploratory Data Analysis:
  • https://ydata-profiling.ydata.ai/docs/master/index.html
  • https://www.digitalocean.com/community/tutorials/exploratory-data-analysis-python
  • https://www.kaggle.com/code/imoore/intro-to-exploratory-data-analysis-eda-in-python
• Visualization:
  • https://plotly.com/python/
  • https://www.ibm.com/topics/data-visualization
  • https://matplotlib.org/stable/tutorials/introductory/pyplot.html
  • https://gilberttanner.com/blog/introduction-to-data-visualization-inpython/