Notes
spark Architecture
Content of the notes (4)
Page 1
Page 2
Important background processes ---
Spark Context
Yarn Resource Manager, Application Master & launching of executors (containers).
CoarseGrainedExecutorBackend & Netty-based RPC.
SparkListeners.
Execution of a job (Logical plan, Physical plan).
Spark-WebUI.
Page 3
Hadoop Architecture
There are 3 core components of Hadoop:
Hadoop Distributed File System (HDFS) — A distributed storage file system that provides high-throughput access to application data.
Yet Another Resource Negotiator (YARN) — Framework for job scheduling and cluster resource management.
MapReduce — System for parallel processing of large data sets
Page 4
Information
- Description
- Note on spark Architecture, created by Abhi Sethi on 24/01/2022.
- Created by
- Abhi Sethi · Jan 2022
- Updated
- Jan 24, 2022
- Views
- 0
- Rating