BigData and Hadoop

BigData and Hadoop

1. Characteristics, advantages and environment of Hadoop

   • Distributed Storage
     • Data is stored in blocks across multiple machines
   • Parallel Processing
     • Tasks are executed simultaneously, reducing computation time
   • Fault Tolerance
     • Automatic data replication ensures recovery from node failures
   • Scalability
     • Clusters can grow by adding more nodes without major reconfiguration
   • Flexibility
     • Handles structured, semi-structured, and unstructured data
   • Open Source
     • Maintained by the Apache Software Foundation, widely adopted
   • Java-based Environment
     • Built primarily using Java, allowing portability
   • Main elements
     • HDFS (Hadoop Distributed File System)
       • Stores data across multiple nodes
     • YARN (Yet Another Resource Negotiator)
       • Manages cluster resources
     • MapReduce
       • Processing model for parallel computation
     • Cluster environment
       • Master node (NameNode, ResourceManager) + worker nodes (DataNodes, NodeManagers)
     • Ecosystem tools
       • Hive, Pig, HBase, Spark, Sqoop, Flume, Oozie, etc

2. Timeline of the Development of Hadoop

   • 2003: Google publishes GFS paper.
     • 2004: Google publishes MapReduce paper.
       • 2005: Hadoop starts inside the Nutch project (Doug Cutting & Mike Cafarella).
         • 2006: Hadoop becomes an Apache top-level project.
           • 2008: Hadoop 1.x (MapReduce + HDFS).
             • 2013: Hadoop 2.x introduces YARN, scalability improvements.
               • 2017–present: Hadoop 3.x with erasure coding, improved HDFS, containerization.

3. Google File System (GFS) and Hadoop

   • HDFS was inspired by Google File System (GFS) and adopts its principles of block-based, replicated storage across distributed nodes.
     • Hadoop’s storage model originates from Google’s design.
     • Large blocks (64–128 MB)
     • Ensures reliability (default 3 replicas)
     • NameNode + DataNodes
     • Optimized for batch/data analytics workloads

4. MapReduce in Hadoop

   • MapReduce is a programming model that processes large datasets in parallel by dividing tasks into a Map phase and a Reduce phase
     • Map phase: Processes (key, value) pairs and produces intermediate results
       • Shuffle & Sort: Groups and sorts keys between phases
         • Reduce phase: Aggregates intermediate results into final output
           • Parallelism: Each task runs independently across nodes
             • Use cases: Word count, log analysis, pattern detection

5. Relation of Big Data Analytics with Machine Learning and Cloud Computing

   • Big Data Analytics uses large-scale data processing platforms (like Hadoop) to generate insights, while Machine Learning extracts predictive patterns, and Cloud Computing provides scalable infrastructure to support both.
     • Big Data Analytics: Processes massive, diverse data. Uses tools like Hadoop, Spark, Hive
     • Machine Learning: Builds models to classify, predict, or cluster data. Depends on Big Data for high-accuracy patterns.
     • Cloud Computing: Provides scalable, on-demand infrastructure (AWS, Azure, GCP). Reduces hardware cost and increases flexibility.
       • Big Data feeds ML models. Cloud hosts and scales Big Data + ML workloads. Distributed systems support end-to-end analytics.