O GoConqr está de visual novo! Conte para nós o que você achou.

Registre-se grátis

Flashcards

Data Engineering Lecture 6: Batch Processing

Tom van Eijk Engineering Inglês 45 flashcards

Data Engineering Lecture 6: Batch Processing

Flashcards do conjunto (45)

  1. 1Batch processing has many benefits- It can shift the time of job processing to when the computing resources are less busy (Load balancing).
    - It reduces manual intervention and supervision.
    - By keeping an overall high rate of utilization, it amortizes the costs.
    - It allows the system to use different priorities for batch and interactive work.
    - It allows you to do big data analysis on multiple machines when stored on HDFS
  2. 2Batch Processing ModelBatch Processing Model
  3. 3HDFSFile system where all data is present. We would just like to analyze this.
  4. 4Large datasets entail vast amounts of data (multi-terabyte datasets) in parallel on large clusters (thousands of nodes) of commodity hardware. Thus, horizontal scaling is required. Some Examples:▪ Bioinformatics data from about 3.3 billion base pairs in the human genome to vast protein sequences.
    ▪ The internet: e.g., Facebook, Twitter, weblogs, maps.
    ▪ Financial applications to grasp financial trends.
    ▪ Astronomy: e.g., the Hubble ultra-deep telescope shows hundreds of galaxies, each with billions of stars.
    ▪ Health Care with massive amounts of patient, drug, and treatment
    data.
  5. 5Horizontal vs vertical scalingHorizontal vs vertical scaling
  6. 6Classic data mining- Reading a word from disk versus main memory is 105 slower!
    - Although reading many contiguously stored words is faster, modern disks still only reach 150 MB/s for sequential reads.
    - Therefore, there is an I/O bound in Data Mining!
    - The biggest performance bottleneck is reading from/writing to disk.
    - 100 GBs ~10 minutes to read
    - 200 TBs ~200 minutes to read ~13 days
    - Performance Hadoop is much slower than performance in Spark. It builds more on memory, which is faster compared to disk speed.
  7. 7Classical Data Mining ModelClassical Data Mining Model
  8. 8Classical data mining (e.g., Apache Lucene) focuses onEfficient disk usage, where the I/O bound is a significant limitation when processing large files.
  9. 9How to solve this issue the I/O bound issue?Distributed architecture -> modern setups often have multiple CPUs and disks per server (even our mobile phones)
  10. 10Distributed architectureDistributed architecture
  11. 11Challenges for cluster computing- Nodes fail
    - Network is a bottleneck
    - Traditional distributed programming is often ad-hoc
  12. 12Nodes fail1 in 1,000 nodes fails every day. We care a lot about this problem!
  13. 13Solve nodes failDuplicate data (HDFS – mapping & reduce actually happen in file system)
  14. 14Network is a bottleneckTypically, 1-10 Gb/s throughput
  15. 15Solve network as a bottleneckBring computation to nodes rather than nodes to data. (Sort and Shuffle)
  16. 16Solve traditional distributed programming as often ad-hoc (specific)We should aim for programs that can be easily distributed (Simply define the map and reduce functions)
  17. 17Solve the previous three limitationsThese limitations are addressed by MapReduce and practical application in Hadoop/Spark!
  18. 18What is MapReduce:1. A programming model
    ▪ Input chunks => map tasks | group_by keys | reduce tasks => output
    | is the Linux pipe symbol: the output of a process is used as input of the next one.
    ▪ e.g., Word Counter. We want to count unique words.
    tokenize(document) | sort | uniq -c
    2. A system that distributes MapReduce-style programs across a distributed file system.
    e.g., Apache Hadoop with HDFS or Apache Spark
  19. 19MapReduce programming modelMapReduce programming model
  20. 20MapReduce from the programmer perspectiveGiven a set of key-value pairs, you must specify two methods.
    - Map (k, v) → * takes a key-value pair and outputs a set of key-value pairs.
    ▪ e.g., the key is the filename, and the value is a single line in the file.
    ▪ A Map call is executed for each (k, v) pair.
    - Reduce (k’, *) → * reduces together all values v’ with same key k’ in v’ order.
    ▪ A Reduce call is executed for each unique key k’.
  21. 21You provide the Map and Reduce functions and the input files. The program workflow:1. Read and prepare the input as a set of key-value pairs.
    2. Run the Map code on the kv pairs to generate a new set of k'v’ pairs.
    3. Shuffle the k'v’ pairs to the reduce functions.
    4. Reduce all k'v'-pairs grouped by key into new k'v''-pairs.
    5. Produce the final output.
  22. 22The program workflow:The program workflow:
  23. 23All phases are distributed with many tasks doing the work. An example for word counting with MapReduceAll phases are distributed with many tasks doing the work. An example for word counting with MapReduce
  24. 24Distributed filesystemsConsider two different files (i.e., C and D) and four servers. Some data is duplicated. This gives us improved reliability if one node wat to fail. More duplication means less space for other data.Distributed filesystems
  25. 25Components of distributed filesystem- Data Nodes => workers
    - Name Node => master/primary node
    - Client library for file access
  26. 26Data Nodes => workers▪ In data nodes, each file is split into contiguous chunks.
    ▪ Typically, each chunk is 16 - 64 MB.
    ▪ Each chunk is replicated to avoid data losses.
    ▪ Try to keep data replicas in different racks.
  27. 27Name node => master/primary node▪ Takes care of the coordination: e.g., it knows where the files are stored
  28. 28Client library for file access▪ Communicates to the master node to find chunk servers.
    ▪ Connects directly to chunk servers to access data.
  29. 29MapReduce environmentApache Hadoop
  30. 30Apache Hadoop and Spark distributes MapReduce model programs across a distributed file systemDashed lines are hidden to the user
  31. 31Apache Hadoop methodApache Hadoop method
  32. 32Data flow definitionWe want to reduce data movement as this is inefficient
  33. 33Data flow context- Input and final output are stored on a distributed file system (FS):
    - the scheduler tries to put the map tasks “close” to the physical storage location of the input data.
    - Intermediate results are stored in the local FS of the Map and Reduce workers.
    - The output is often input to another MapReduce task
  34. 34Apache Hadoop and Spark handles▪ partitioning
    ▪ scheduling of the map and reduce function executions
    ▪ group by key
    ▪ restart from node failures
    ▪ inter-machine communication.
  35. 35Master node tasks- A task can have three statuses: idle, in progress, completed. The master node takes care of the coordination.
    - Idle tasks get scheduled as workers become available.
    - When a map task is completed, its node returns the location and size of the intermediate files to the master. The master node pushes this info to reducers.
    - Master node pings workers periodically to detect failures.
  36. 36Dealing with Failures- Map worker failure
    - Reduce worker failure
    - Master failure
    - Map tasks are scheduled near the physical storage of chunks to improve performance. Intermediate results are stored locally. Finally, the number of reducers should be higher than the number of nodes.
  37. 37Map worker failure▪ Tasks completed or in progress by the map worker are reset to idle (i.e., they need to be scheduled).
    ▪ Reduce workers are notified when a task is rescheduled on another worker.
  38. 38Reduce worker failure▪ Only tasks in progress by the reduce worker are reset to idle (i.e., they need to be scheduled).
  39. 39Master failureMapReduce task is aborted, and the client is notified
  40. 40How many Map and Reduce jobs are needed?- M map tasks, R reduce tasks.
    - Rule of a thumb:
    ▪ Make M much larger than the number of nodes in the cluster.
    ▪ One chunk of the distributed FS per map is common.
    ▪ Use dynamic load balancing to speed up recovery from worker failures.
    - Usually, R is smaller than M
    ▪ Because the output is spread across R files.
  41. 41Beyond MapReduce – three things to improve, building on the topics discussed before.1. Backup tasks
    2. Combiner function
    3. Partition function
  42. 42Backup tasks- Problem
    ▪ Slow workers significantly increase the job completion time:
    - Solution
    ▪ Near the end phase, spawn backup copies of tasks: whichever one finishes first “wins”.
    - Effect
    ▪ Dramatically shortens job completion time.
  43. 43Combiner function- Often a Map task will produce many pairs of the form (k,v1), (k,v2), … for the same key. (e.g., popular words in the word count example)
    - We can save network time by adding an extra layer and pre-aggregating values in the mapper:
    ▪ Combine (k, list(v1)) → v2
    ▪ Combine is usually hosted in the same node of the reduce function.
    - Combiners work only if the reduce function is commutative and associative.
    - Back to our word counting example. Combiner mix the values of all keys of a single mapper (i.e., a single machine). Much less data needs to be copied and shuffled!
  44. 44Combiner function in MapReduceCombiner function in MapReduce
  45. 45Partition function- Continuous splits of the input are assigned to the same worker.
    The reduce function should ensure that records with the same intermediate key end up in the same worker.
    - MapReduce uses a default partition function: hash(key) mod R.
    - Sometimes it is helpful to override the hash function:
    hash (hostname (URL)) mod R ensures URLs from a host end up in the same output file.

Informações

Descrição
Engineering FlashCards sobre Data Engineering Lecture 6: Batch Processing, criado por Tom van Eijk em 21-12-2021.
Criado por
Tom van Eijk
Visualizações
0
Avaliação

Fale Conosco

Sobre o que é sua mensagem?

O que você acha do novo visual?

Você gostou?