Flashcards
Data Engineering Lecture 7: Stream Processing
Flashcards do conjunto (53)
- 1Stream Processing
- 2Often, we do not know when data will be available. In such cases:- We cannot store them.
- We cannot access them repeatedly.
- We do not know how fast they will arrive.
- Even insert them in a database does not make sense.
In sum, we cannot fit them in a disk, but we would like to analyze them to make some decisions. - 3The main differences compared to database management and batch processing are:- The input is not controlled by the system.
- The input timing/rate is often unknown but controlled by users. - 4The process could hold sliding windows of records (i.e., micro-batch) instead of single records.
- 5Stream data processing use cases:- Monitoring and alerting: notification on events / patterns
- Real-time reporting: aggregate statistics for dashboards
- Real-time ETL (i.e., Extract, Transform, Load) and event-driven data updates
- Real-time decision making: fraud detection
- Data stream mining: summary statics with limited memory - 6General stream processing model
- 7Streaming models exhibit real-time latency requirements:- Real-Time: guaranteed task completion by a given deadline (few milliseconds; autonomous car)
- Near Real-Time: few milliseconds to seconds (Alexa) - 8General stream processing model aspects- A stream of records, often referred to as “elements” or,” tuples”. In theory, it could be everything!
- Streaming models exhibit real-time latency requirements.
- In practice, even longer latency could be accepted. - 9Architecture – Native Streaming (Operator Model)
- 10Asynchronous operationsOperation you run after collecting enough events in memory. This implies we have quasi real time, as we are still collecting information in a memory.
- 11If you have a lot of memoryYou need to wait so it can be slow.
- 12If you have little memoryYou may not have enough data to run an operation.
- 13Operator and Queue SharingIf you have multiple operations, these will be reduced.
- 14Operator sharingManage complex operations
- 15Queue sharingAvoid duplication in output queues
- 16Given a set of continuous deployed queriesCommon subexpression elimination is applied to optimize code
- 17Further operation takes longer time (e.g., 20 ms) than a previous operation (e.g., 3 ms).- Back Pressure
- Load Shedding
- Distributed Stream Processing - 18Back PressureSlow down sources to avoid data loss (e.g., blocking queues)
- 19Load Shedding - Random-samplingWhen there is some guarantee that random samples will give good enough accuracy (i.e., approximation is close)
- 20Relevance-based load sheddingAlgorithm knows relevant events and which to take
- 21Summary-based load sheddingProvide summary of que, use this as input for the next operation
- 22Given an SLASelect the queries and shedding that minimize the error and satisfy the constraints
- 23Load Shedding - There is a tradeoff to balanceUse a threshold that would be good for the specific purpose
- 24Distributed Stream Processing- We send input to different nodes, first that answers wins
- Data flow partitioning
- Key range partitioning - 25Key range partitioningDistribute the data stream
- 26Data flow partitioningDistributed the query
- 27Random Sampling
- 28Problem Random SamplingRecords often are not the units of the statistical analysis. (e.g., a general tweet is not the unit of analysis to find the trending hashtags of a specific user)
- 29Event TimeReal time when the event/data item was created – I speak to Google home.
- 30Ingestion TimeSystem time when the data item was received – Google knows it must do something.
- 31Processing TimeSystem time when the data item is processed – operation is executed
- 32In a real-world scenario, the processing time is delayed, and data/order can be processed unordered. Unordered data for a Tesla for instance, would be bad!Therefore, heuristics (e.g., watermarks/unique tag) or more complex triggers (providing results later) are used.
- 33Durability and consistency guarantees- At most once
- At least once
- Exactly once - 34When to use which guarantee?It is about complexity and performance: if you want something precise, go for ‘exactly once’, if you want something fast, use ‘at most once’.
- 35At Most Once- ”Send and forget”, ensure data is never counted twice
- Might cause data loss on failures
- Data is stored nowhere - 36At Least Once- ”Store and forward”, item stays until the acknowledgements from receiver i.e., replay a stream from a checkpoint on failures
- Might create incorrect state because data is processed multiple times - 37Exactly Once- “Store and forward” with guarantees regarding state updates and sent messages
- Often via dedicated transaction mechanisms - 38Window SemanticsSome operations (e.g., joins) can be computed over windows of time or elements to avoid them being undefined over unbounded streams. Example has two streams (orange and green). The join is just an example. It is also possible to calculate the average, sum, etc.
- 39Tumbling Window- Every data item is part of only one window.
- Using tumbling windows, a join is computed as usual.
- E.g., how many interactions did I have with my Google home in the last 5
seconds?
- E.g., calculating the trending hashtags on twitter in the last 10 minutes, and every 10 minutes you move your window.
- E.g., amazon: in this warehouse, what is the most sold item? (It is very similar to batch) - 40Tumbling Window Model
- 41Sliding Window- Time- or tuple-based sliding windows.
- New data is inserted, and old data expires.
- Considers more data from the past.
- E.g., we have a window of 20 items, which starts with zero items. As soon at
it has 20 and another is added, the old one is removed.
- E.g., what was the trending hashtag of the last 10.000.000 tweets? Every time a new tweet is posted, we forget about the old one.
- E.g., car speed: average speed in the last hour? You compute speed every second and the hour is moving with you. You forget the oldest info and input the newest. - 42Sliding Window Model
- 43Large scale steam processing- Shuffle Grouping
- Field Grouping
Apache Spark will apply these concepts according to your specific need. - 44Shuffle Grouping- Tuples are randomly distributed across workers.
- Good load balance.
- Problem: I want to calculate the average speed of my car. I now distribute this across 100 different workers around the world. This would not be good for me. Solution is Field grouping. - 45Field Grouping- Tuples are partitioned by grouping attributes.
- Guarantees order within keys, but the load is not balanced if skewed.
- Example: 100 tesla cars in a region, where each car sends its data to only one node. In this case, some nodes will be used more (frequent drivers) and some less (cars in a garage). - 46Discretized steam (batch) computationCombined with distributed batch analytics, enables fault tolerance (when a node fails, we want to have a prevention mechanism)
- 47Discretized steam (batch) computation - The input tuples are batched based on the ingestion timeStateless, deterministic tasks are run periodically as distributed jobs.
- 48Tasks Discretized Stream ComputationAll tasks return RDDs (i.e., sequences of immutable partitioned datasets)
- 49Limitations: Discretized Stream ComputationHigher latency than usual
- 50Apache Spark - Discretized Stream ComputationApache Spark is based on micro-batch computation with exactly once guarantee. This characteristic is provided by back-pressure (faster operation slows down) and watermarks (unique tag). It means that when you get at output from Spark, it is very likely that this output was corresponds to the first item that you sent to Spark.
▪ All the items are processed in order!
▪ Drawback: latency will be much higher than usual - 51Batch Computation
- 52Spark is a good trade offBetween the full tolerance (for not losing data), the accuracy of the results (exactly once guarantee), and latency (you will not sacrifice a lot)
- 53New version Apache SparkContinuous streaming is also supported
Informações
- Descrição
- Engineering FlashCards sobre Data Engineering Lecture 7: Stream Processing, criado por Tom van Eijk em 20-12-2021.
- Criado por
- Tom van Eijk
- Visualizações
- 3
- Avaliação