Karteikarten
Big Data Technologies Concepts
Karten in diesem Stapel (15)
- 1Apache BeamOpen source unified programming model to define and execute data processing pipelines, including batch and stream processing
- 2PipelineComponent of Apache Beam that encapsulates the workflow of your entire data processing tasks from start to finish. It reads input data, transforms it and writes the output data. All Beam driver programs must create a Pipeline, while specifying the execution options that tell the Pipeline where and how to run
- 3PCollectionComponent of Apache Beam that represents a distributed data set that a Beam pipeline operates on. It may be bounded (finite) as in batch use cases, or unbounded (infinite) as in streaming use cases. Every PCollection has a timestamp associated, however if it is not necessary it is set to “negative infinity”
- 4PTransformComponent of Apache Beam that represents a data processing operation, or a step, in your pipeline. Every PTransform takes one or more PCollection objects as input, performs a processing function that you provide on the elements of that PCollection, and produces zero or more output PCollection objects
- 5ParDoComponent of Apache Beam used for generic parallel processing. It is similar to the “Map” phase of a Map/Shuffle/Reduce-style algorithm. A ParDo transform considers each element in the input PCollection, performs some processing function (your user code) on that element, and emits zero, one, or multiple elements to an output PCollection
- 6DoFnComponent of Apache Beam that applies your logic in each element in the input PCollection and lets you populate the elements of an output PCollection to be included in your pipeline. It is wrapped in a ParDo PTransform
- 7WatermarkEstimates how complete a PCollection is. The contents of a PCollection are complete when a watermark advances to “infinity”. This way we know that an unbounded PCollection is finite (has ended)
- 8WindowIt defines the number of elements that will be processed in the pipeline at once. When elements are read from external sources they arrive in the global window, and when they are written to the outside world, they are placed back into the global window. It has a maximum timestamp, so all data related to an expired window may be discarded at any time
- 9Windowing StrategySpecify essential information for grouping and triggering operations. Operating on a one-by-one element basis may be very inefficient, depending on the operation. For example, GroupByKey is governed by a windowing strategy
- 10CoderSpecifies the binary format of the elements of a PCollection. The format can be just bytes or some encoding system (for example, graphical accents)
- 11Beam RunnerIt generally refers to the software that takes a beam pipeline and runs it. It usually includes some customized operators for your data processing engine and it sometimes refers to the full stack. It has a single method run(pipeline), which should be asynchronous and result in a PipelineResult, which is a job descriptor
- 12Beam ExecutionRunners can execute a pipeline in different ways. Serialization and communication between machines is one of the most expensive operations. Avoiding serialization may require re-processing elements after failures or may limit the distribution of output to other machines
- 13DaskFlexible library for parallel computing in python. It is implemented on top of multiprocessing and multithreading. It is composed of two parts, a dynamic task scheduling optimized for interactive computational workloads, and big data collections, such as parallel arrays, dataframes and lists
- 14Dask AdvantagesFamiliar - provides parallelized numpy array and pandas dataframe objects. Flexible - provides a task scheduling interface. Native - enables distributed computing in pure python with access to the PyData stack. Fast - low overhead, low latency, and minimal serialization. Scales up - runs on clusters with 1000s of cores. Scales down - can trivially run in a laptop using a single process. Responsive - designed for interactive computing and providing rapid feedback
- 15Dask DataFrameLarge parallel DataFrame composed of many smaller Pandas DataFrames, split along the index. It is usually used when Pandas fails due to data size or computation speed
Informationen
- Beschreibung
- Mestrado Segurança Informática Karteikarten am Big Data Technologies Concepts, erstellt von Inês Macedo am 20/06/2021.
- Erstellt von
- Inês Macedo
- Aufrufe
- 1
- Bewertung