The past, present, and future of data engineering

The data engineering practice has been there since the early internet era in the 1990s. Going back to Chapter 1, Fundamentals of Data Engineering, in the Start with knowing the roles of a data engineer section, in the past, data engineers were mostly ETL developers using specific tools. Most of these tools were proprietary tools and located on-premises. The term data engineer itself wasn’t commonplace; the more common terms used to be data modelers, database admin, and ETL developer (ETL references the proprietary ETL tool’s name). Each of the ETL tools had the necessary expertise and best practices surrounding them.

Now, in the present, data engineering has evolved into a more mature and singular role. This means that the practice is receiving a lot more common principles, concepts, and best practices. This is due to two reasons – the rapid improvement in the technologies supporting the practice and the fact...