Hadoop and Spark Integration

Hadoop and Spark Integration

1. Is based on

   • Hadoop
     • Open-source tool that allows users to store and process large amounts of data across multiple computers efficiently in a distributed system
       • 1.Hadoop Architecture and Components
         • what is
           • Allows large amounts of data to be stored, organized, and processed across multiple connected computers.
             • Consist of
               • HDFS (Hadoop Distributed File System)
                 • What is
                   • Component that stores large amounts of data across multiple computers in a cluster.
                     • Types
                       • • Stores large datasets across multiple computers • Divides data into blocks • Replicates data to improve availability and fault tolerance.
               • MapReduce
                 • What is
                   • programming model that processes large amounts of data across multiple computers in parallel.
                     • Types
                       • • A programming model • Divides work into map and reduce tasks • large datasets can be processed in parallel.
               • YARN (Yet Another Resource Negotiator)
                 • What is
                   • Manages resources like memory and processing power to run tasks efficiently.
                     • Types
                       • • Manages cluster resources • Schedule and coordinates applications • MapReduce and other processing frameworks such as Spark
       • 2.Hadoop Evolution and Google File System
         • What is
           • open-source framework designed to store and process large amounts of data across multiple computers.
             • Evolution from
               • 2003
                 • 2004
                   • 2006
                     • 2012
                       • 2013–2014
                         • Apache Spark gains prominence as a fast processing engine and becomes an Apache top-level project in 2014.
                       • YARN is introduced with Hadoop 2.0 to improve cluster resource management
                     • Emerges as an open-source project inspired by distributed storage and processing ideas
                   • Google publishes the MapReduce paper, describing a model for parallel data processing
                 • Google publishes the Google File System (GFS) paper, describing distributed storage concepts
                   • HDFS was inspired by Google’s GFS but developed as a separate open-source system
       • 3. Advantages and Limitations of Hadoop
         • Is characterized by
           • Advantages
             • Hadoop offers scalability, data protection, low costs, and support for large amounts of data and different formats.
           • Limitations
             • Hadoop can be complex to manage, slower for some tasks, and less suitable for small datasets or real-time processing.
       • 4. Hadoop, Machine Learning, and Cloud Computing
         • Is integrated into
           • Machine Learning
             • Hadoop
               • Stores large amounts of data and supports distributed processing.
             • Cloud Computing
               • Provides scalable infrastructure to manage Hadoop and Spark clusters.
             • Spark MLlib and TensorFlow, depending on the architecture
               • They support data analysis and machine learning using distributed and cloud systems.