Apache Spark

Apache Spark

1. Architecture

   • Driver Process
     • Jobs
       • Stages
         • Tasks
           • Narrow Transformations
             • Stage Pipelining
           • Wide Transformations
             • Shuffle Persistence
         • Shuffle Operations
       • Logical Execution Plans
       • Physical Execution Plans
       • DAGs
   • Executor Process
   • Spark Session

2. Deployment Mode

   • Local Mode
   • Cluster Mode
     • Master Node
     • Worker Node
   • Client Mode

3. HDFS

   • Architecture
     • Namenode
     • Datanode
       • Read Operation
       • Write Operation
     • Client

Images and files in the map