We are living in a digital world where our daily activities, government operations, and business operations generate huge amounts of data. The world’s data is growing at 20% each year, which means that every four years we are doubling the data. We are sitting on a huge mountain of data. Our day-to-day activities such as social media, entertainment, online learning, online business, and government activities add more data. Most organizations worldwide want to utilize and generate value from this data to make them healthier, more productive, more efficient, and more sustainable places. Mathematicians, statisticians, and computer engineers are developing statistical, data mining, machine learning, natural language, and image processing applications for generating insights and faster decision-making.

Data analysis is an interdisciplinary field that comprises statistical and computing fundamentals for powerful decision-making for individual...

In the last decade, there has been a huge growth in the data analysis field. Lots of efforts are being made to establish the standard methodologies for data analysis and data analysis-based application development. In this section, we will discuss various process methodologies such as KDD, SEMMA, CRISP-DM, and the standard process. These methodologies have few overlapping or similar steps with different objectives.

Knowledge discovery from data (KDD)

Knowledge Discovery from Data is what KDD stands for. Data mining is also known by the term KDD. The practice of discovering and utilizing patterns for knowledge discovery is known as data mining. Finding hidden patterns in the data sources that are provided is the primary objective of the KDD process. There are seven main phases to the KDD process:

• Data cleaning: In this first stage, we handle the noisy data, missing values, duplicates, and outliers in the given dataset.
• Data integration: Then, data migration...

Data analysis is the process of exploring and investigating data to discover hidden facts and patterns that can serve business and policy-making decisions. It is a sub-field of data science that explores data to find insights. Data analysis tools and techniques are widely engaged in various business, government, and social domains by data analysts, business analysts, statisticians, social scientists, data scientists, researchers, and consultants. The main purpose of data analysis is to optimize business operations, profits, and effectiveness. The process of data analysis is to query and collect the relevant data from various sources, explore and understand the data, visualize the findings, prepare dashboards and reports, and present it in the form of a story to the senior management for decision-making.

In contrast, data science is a multidisciplinary field that combines domain expertise, computer science, and statistics. Data...

We can easily download the installer file for installing Python 3 from the official website of Python (https://www.python.org/downloads/) for Windows, Linux, Mac, and other 32-bit or 64-bit systems. By Double-clicking on this installer, we can easily install it on your computer. Additionally, the Python installation offers the "IDLE" Integrated Development Environment (IDE), which is useful for development. In the upcoming sections of this chapter, we will go deep into Python installation of each of the operating systems.

Python installation and setup on Windows

This book is based on the latest version of Python 3. All the coding examples used in this book are written in Python 3 version, so before starting the hands-on coding, we have to install Python 3. Python is an open-source, easy-to-learn, object-oriented, and distributed language. It is also licensed for commercial use. There are many implementations of Python, including commercial implementations...

In this section, we will see all the IDE and software that we need to use in this book. A Python program can be easily written on any editor and executed on any system that already has Python installed. W A program can be written in Notepad, TextEdit, or any other editor, and it can be run via the terminal or command prompt. We can also use various DEs such as Spyder, VS Code, Jupyter Notebook, PyCharm, and Atom. In this book, we will primarily use the Anaconda IDE for all the data analysis tasks. Anaconda is a freely available open-source, enterprise-ready, and provides a solid foundation for Python-based data analysis. Also, it offers several Python libraries for data analysis, including NumPy, SciPy, Pandas, Scikit-learn, and so on. Anaconda can easily be downloaded and installed, as follows:

1. Installer can be downloaded from the official Anaconda website. https://www.anaconda.com/download.
2. Download the installer as per your operating system.
3. Execute...

In this introductory chapter, we covered data analysis frameworks or process model, such as KDD, SEMMA, CRISP-DM, and standard process for data analysis. The job responsibilities and skill sets of data scientists, data engineers, ML engineers, data analysts, and NLP engineers were then covered. Next, we installed the packages that we will use throughout this book: NumPy, SciPy, Pandas, Matplotlib, IPython, Jupyter Notebook, Anaconda, Jupyter Lab, PyCharm, and VS Code. Installing Anaconda or Jupyter Lab, which comes with NumPy, Pandas, SciPy, and Scikit-learn integrated in, is a better option than installing all those modules. Next, we successfully implemented a vector addition application and discovered that NumPy performs better than the other libraries. We looked through the available documentation and learning resources on the nternet. We also talked about Pycharm, VS Code, Databricks, Jupyter Notebook, Jupyter Lab, and their features.

In the next chapter, Chapter 2, NumPy...