Spark architecture
The Apache Spark architecture is complex, to say the least, and requires in-depth knowledge. However, you only need some background knowledge to be reasonably productive with Spark. So, first, let’s go through the basics of Apache Spark.
Introduction to Apache Spark
Spark is a popular parallel data processing framework built from the lessons learned after the Apache Hadoop project. Spark is written in Scala, a JVM language, but supports other languages, including Python, R, and Java, to name a few. Spark can be used as the central processing component in any data platform, but others may be a better fit for your problem. The key thing to understand is that Spark is separated from your storage layer, which allows you to connect Spark to any storage technology you need. Similar tools include Flink, AWS Glue, and Snowflake (Snowflake uses a decoupled storage and compute layer pattern behind the scenes).
