Hadoop Ecosystem

Hadoop Ecosystem

1. Hadoop is an open-source framework that enables storage and distributed processing of large data sets across clusters of computers using simple programming models.

2. Characteristics and Environment of Hadoop

   • Arquitectura distribuida y escalable (soporta miles de nodos).
   • Open-source project under Apache Foundation.
   • Fault-tolerant through data replication.
   • Integrates with other Big Data tools
     • Hive, Pig, Spark, Flume, Sqoop
   • Supports various programming languages
     • (Java, Python, R).
   • Designed for batch processing and large-scale data analytics.

3. Timeline of Hadoop Development

   • 1999 – Apache Software Foundation (ASF) formed as a non-profit
   • 2002 – Nutch created by Doug Cutting and Mike Cafarella.
   • 2006 – Cutting joins Yahoo, takes Nutch with him.
   • 2008 – Nutch divided and Hadoop is born. - Yahoo releases Hadoop as open-source project to ASF. – Hadoop-based start-up Cloudera incorporated.
   • 2009 – Cutting leaves Yahoo for Cloudera.
   • 2011–2015: Ecosystem expands — Hive (SQL interface), Pig (data scripting), HBase (NoSQL database), and Spark (in-memory computing).
   • 2016–present: Integration with cloud services (AWS EMR, Azure HDInsight, Google Dataproc) and Machine Learning frameworks.

4. Advantages of Hadoop

   • Scalable
     • Easily adds new nodes without downtime.
   • Cost-effective
     • Uses inexpensive commodity hardware.
   • Fault tolerance
     • Data is automatically replicated across nodes
   • Flexible
     • Processes structured, semi-structured, and unstructured data.
   • High throughput
     • Suitable for large-scale batch data processing.
   • Open source
     • Free and widely supported community.

5. Google File System (GFS) and Hadoop

   • GFS (Google File System) inspired the creation of HDFS (Hadoop Distributed File System).
   • HDFS divide los archivos en bloques (64–128 MB) y los distribuye entre los nodos del clúster.
   • A NameNode manages metadata, while DataNodes store actual data
   • Designed for write-once, read-many workloads.

6. MapReduce in Hadoop

   • Programming model for parallel data processing.
   • Map phase: Divides data into smaller sub-tasks and processes them independently.
   • Reduce phase: Aggregates results from the Map tasks to produce the final output.
   • Provides fault tolerance by re-executing failed tasks.
   • Enables massive data analysis without centralized processing.

7. Relation with Big Data Analytics, Machine Learning, and Cloud Computing

   • Hadoop is a core technology for Big Data Analytics, enabling processing of petabytes of data.
   • Integrates with Machine Learning frameworks like Apache Mahout, MLlib (Spark), and TensorFlow via Hadoop YARN.
   • Works seamlessly with cloud platforms
     • AWS
     • Google Cloud
     • AZURE
   • Used for data storage, preprocessing, and distributed computing before applying ML models.
   • Forms the backbone of data lakes and enterprise data pipelines.

Images and files in the map