HADOOP

HADOOP

1. Characteristics and environment of Hadoop

   • Hadoop operates on a master-slave model because the NameNode coordinates and the DataNodes store the data. It allows scaling from one server to thousands of nodes using standard hardware.
   • Automatically replicates data across different nodes. If one fails, the system continues to operate without loss of information.
   • Executes computational tasks directly on nodes storing the relevant data, significantly reducing network congestion and improving processing efficiency.
   • Supports linear scaling from single-node deployments to clusters comprising thousands of nodes, maintaining performance through distributed processing.

2. Timeline of the Development of Hadoop

   • 2003–2004: Google publishes the GFS and MapReduce papers, inspiring the development of Hadoop.
   • 2005–2006: Doug Cutting and Mike Cafarella create Hadoop as part of the Nutch project, named after Cutting's toy elephant.
   • 2008: Hadoop becomes a top-level Apache project.
   • 2011: Release of Hadoop 1.0 with stable support for HDFS and MapReduce.
   • 2013: Hadoop 2.0 introduces YARN, enabling multiple workloads in addition to MapReduce.
   • 2017: Hadoop 3.0 includes erasure coding, improvements to YARN, and support for containers.
   • 2020–present: Integration with cloud ecosystems and machine learning.

3. Hadoop Advantages

   • Ability to expand by adding nodes without interrupting services.
     • Break complex tasks down into manageable parts that can be executed in parallel.
       • Integration with Spark, Kafka, HBase, Hive, etc., facilitates different types of analysis.
         • Handles structured, semi-structured, and unstructured data such as text, images, logs, etc.

4. Google File System (GFS) and Hadoop

   • GFS was Google's system for handling massive amounts of data. Hadoop was inspired by it to create HDFS, but with one major difference: GFS was exclusive to Google, while HDFS is open and works on any hardware. HDFS improved metadata management and data replication, making it more efficient and fault-tolerant. In addition, HDFS is optimized for inexpensive hardware and allows for seamless horizontal scaling.

5. MapReduce in Hadoop.

   • Processes large volumes of data in parallel.
     • Divide the data into key-value pairs.
       • Combine and summarize the results.
         • Word count (distributed counting)
           • Scalable and fault tolerant.

6. Relation of Big Data Analytics with Machine Learning and Cloud Computing.

   • Big Data Analytics and Machine Learning :The data stored in Hadoop/Spark is the basis for training Machine Learning models. With tools such as MLlib, algorithms are executed in a distributed manner, allowing user behavior to be analyzed for personalized recommendations.
     • Big Data Analytics and Cloud Computing :The cloud offers elastic scalability for processing massive amounts of data with services such as AWS S3 and EMR. This reduces costs and allows global access, ideal for processing logs in real time with Kafka and Spark.
       • Triple Integration (Data + ML + Cloud) : Combines cloud data, ML models, and scalable analytics to create comprehensive solutions. Example: fraud detection with cloud data, Spark models, and dashboard results, achieving agility and cost reduction.