Flashcards
Apache Spark Concepts
Flashcards do conjunto (29)
- 1Apache Spark or MapReduceUnified computing engine and a set of libraries for parallel data processing on computer clusters. It is the standard tool to handle big data
- 2Driver ProcessProcess responsible for maintaining information about the Spark Application; responding to a user’s program or input; and analyzing, distributing, and scheduling work across the executors
- 3Executor ProcessProcess responsible for executing code assigned to it by the driver; and reporting the state of the computation back to the driver node
- 4Spark SessionPart of the driver process, it provides user code with the primary interface for Spark functionality
- 5JobLaunched by the driver process, it is a sequence of stages that represent an application action
- 6StageA group of tasks that may execute together to compute the same operation on multiple RDD partitions or machines. Stages are separated by shuffle operations that repartition data
- 7TaskA computation that runs in a single executor operating transformations over blocks of data. Several instances of the same tasks may run in parallel using distinct executors
- 8Logical Execution PlanStructured tree that represents the expected output after applying a series of transformations, which are independent of the cluster’s characteristics
- 9Physical Execution PlanCompiled from logical execution plans, it defines the actual job stages and their component tasks. It may also account for the cluster's characteristics
- 10Directed Acyclic Graph (DAG)A directed graph generated through the logical and physical plans. It consists of vertices and edges, where vertices define tasks or reshuffle operations and edges reflect execution precedence
- 11Resilient Distributed Dataset (RDD)A read-only collection of data that can be partitioned across a subset of Spark cluster machines and form the main working component
- 12Narrow TransformationMaps each input partition to only one output partition. Examples include flatMap, map, filter, ...
- 13Wide TransformationAlso called a shuffle, since the data needs to be reshuffled across executors. It will have input partitions contributing to many output partitions. Examples include reduceByKey, ...
- 14Stage PipeliningAn automatic operation that pipelines transformations and allows in-memory data processing with no intermediate disk writes, if multiple filters are specified on dataframes. This means narrow transformations can fit in the same stage
- 15Shuffle PersistenceOptimization operation that occurs when dealing with more than one wide transformation. To start, the source stage of a shuffle operation writes shuffle files to local disks (at the level of each executor). In the sink stage, that performs the grouping and reduction, data is fetched from these shuffle files in groups of keys. The point is to avoid recomputation of the the source stage if there is a failure during the sink stage, and to be able to schedule it flexibly
- 16Local ModeDeployment mode for Spark applications, which makes them run on a single machine. Parallelism is only made possible by multithreading over the machine’s CPU cores (and for some operations also GPUs). It is typically employed during application development for small datasets. It is not appropriate for production or for large datasets
- 17Cluster ModeDeployment mode where a Spark application driver and its executors all run inside a cluster in association to workers. Thus, all driver/executor interactions take place within the cluster
- 18Spark ClusterSet of computers that work together so that they can be viewed as a single system. For Spark, it is composed of master and worker nodes
- 19Master NodeMediates access to workers in the cluster
- 20Worker NodeHosts driver or executor processes of Spark applications
- 21Client ModeDeployment mode where the application driver runs in a machine outside the cluster. The driver’s machine is often called a “gateway machine” or “edge node”. It may be more convenient or flexible mode in a number of situations, mainly if the driver program (but not executors) requires resources not accessible within the cluster or for security reasons (user code may not be trustworthy)
- 22HDFS (Hadoop Distributed File System)File system designed to store very large files across distributed machines in a large cluster with streaming data access patterns. It is commonly used for Apache Spark, Hadoop MapReduce, and other frameworks in the Hadoop ecosystem
- 23DistributedData is distributed across several machines to allow for fault tolerance and parallel processing
- 24Streaming Data Access PatternsFiles are typically written once or in append mode, and read many times subsequently
- 25HDFS ClusterCluster composed of namenodes, which manage the file system and its meta-data, and datanodes, which provide actual storage
- 26HDFS ClientIt accesses namenodes, to get information about HDFS files, and datanodes, to read and write data
- 27HDFS FileProvides the abstraction of a single file, but is in fact divided into blocks, each with an equal size of typically 64 or 128 MB
- 28HDFS BlockThe elementary unit for read/write operations by client applications. Each one is stored and replicated independently in different machines. They are also stored in datanodes by the host file system as regular files
- 29Advantages of HDFS FilesSupport for large files since an HDFS file can be larger than any single disk in the network. Fault tolerance and high availability since if a block becomes unavailable from a datanode, a replica can be read from another datanode in a way that is transparent to the client. Data locality in integration with MapReduce since Hadoop MapReduce takes advantage of the block separation to move computation near to where data is located, rather than the other way round since that would be quite more costly in terms of network communication and operation time (moving computation is cheaper than moving data)
Informações
- Descrição
- Mestrado Segurança Informática FlashCards sobre Apache Spark Concepts, criado por Inês Macedo em 19-06-2021.
- Criado por
- Inês Macedo
- Visualizações
- 0
- Avaliação