spark Architecture

spark Architecture

Important background processes ---

Spark Context

Yarn Resource Manager, Application Master & launching of executors (containers).

CoarseGrainedExecutorBackend & Netty-based RPC.

SparkListeners.

Execution of a job (Logical plan, Physical plan).

Spark-WebUI.

There are 3 core components of Hadoop:

Hadoop Distributed File System (HDFS) — A distributed storage file system that provides high-throughput access to application data.

Yet Another Resource Negotiator (YARN) — Framework for job scheduling and cluster resource management.

MapReduce — System for parallel processing of large data sets

Contenido del apunte (4)