Well, first we will classify people into four groups based on skill: data architect, data analyst, data engineer, and data scientist.
Usually, data skills are separated into two broad categories:
- Engineering skills: All the DevOps (yes, DevOps is the new black): setting up servers and clusters, operating systems, write/optimize/distribute queries, network protocol knowledge, programming, and all the stuff related to computer science
- Analytical skills: All mathematical knowledge: statistics, multivariable analysis, matrix algebra, data mining, machine learning, and so on.
Data analysts and data scientists have different skills but usually have the same mission in the enterprise.
Data engineers and data architects have the same skills but usually different work profiles.
Large enterprises collect and generate a lot of data from different sources:
- Internal sources: Owned systems, for example, CRM, HRM, application servers, web server logs, databases, and so on.
- External sources: For example, social network platforms (WhatsApp, Twitter, Facebook, Instagram), stock market feeds, GPS clients, and so on.
Data architects:
- Understand all these data sources and develop a plan for collecting, integrating, centralizing, and maintaining all the data
- Know the relationship between data and current operations, and understand the effects that any process change has on the data used in the organization
- Have an end-to-end vision of the processes, and see how a logical design maps a physical design, and how the data flows through every stage in the organization
- Design data models (for example, relational databases)
- Develop strategies for all data lifecycles: acquisition, storage, recovery, cleaning, and so on
A data engineer is a hardcore engineer who knows the internals of the data engines (for example, database software).
Data engineers:
- Can install all the infrastructure (database systems, file systems)
- Write complex queries (SQL and NoSQL)
- Scale horizontally to multiple machines and clusters
- Ensure backups and design and execute disaster recovery plans
- Usually have low-level expertise in different data engines and database software
Their primary tasks are the compilation and analysis of numerical information.
Data analysts:
- Have computer science and business knowledge
- Have analytical insights into all the organization's data
- Know which information makes sense to the enterprise
- Translate all this into decent reports so the non-technical people can understand and make decisions
- Do not usually work with statistics
- Are present (but specialized) in mid-sized organizations for example, sales analysts, marketing analysts, quality analysts, and so on
- Can figure out new strategies and report to the decision makers
This is a modern phenomenon and is usually associated with modern data. Their mission is the same as that of a data analyst but, when the frequency, velocity, or volume of data crosses a certain level, this position has specific and sophisticated skills to get those insights out.
Data scientists:
- Have overlapping skills, including but not limited to: Database system engineering (DB engines, SQL, NoSQL), big data systems handling (Hadoop, Spark), computer language knowledge (R, Python, Scala), mathematics (statistics, multivariable analysis, matrix algebra), data mining, machine learning, and so on
- Explore and examine data from multiple heterogeneous data sources (unlike data analysts)
- Can sift through all incoming data to discover a previously hidden insight
- Can make inductions, deductions, and abductions of data to solve a business problem or find a business pattern (usually data analysts just make inductions of from data)
- The best don't just address known business problems, they find patterns to solve new problems and add value to the organization
As you can deduce, this book is mainly focused on the data architect and data engineer profiles. But if you're an enthusiastic data scientist looking for more wisdom, we hope to be useful to you, too.
