Hadoop

Hadoop

1. Features and Environment

   • Framework de Código Abierto: Basado en Java y desarrollado por la Apache Software Foundation.
   • Horizontal Scalability: Allows adding more nodes to the system without major changes.
   • Fault Tolerance: Automatically replicates data across multiple nodes to ensure availability.
   • Distributed Processing: Splits tasks across several nodes to optimize processing time.
   • Compatibility: Supports various types of data (structured, semi-structured, and unstructured).

2. Timeline of Hadoop Development

   • 2003: Google File System (GFS): Publication of the paper describing its distributed file system.
   • 2004: MapReduce: Publication of the paper introducing the programming model for processing large volumes of data.
   • 2006: Birth of Hadoop: Doug Cutting and Mike Cafarella develop Hadoop, inspired by Google's papers. HDFS and MapReduce: Core components of the original Hadoop project.
   • 2011: Apache Top-Level Project: Hadoop becomes an independent project under Apache.

3. Advantages of Hadoop

   • Efficient Distributed Processing: Breaks down large tasks into multiple nodes.
   • Capacity for Large Data Volumes (Big Data): Handles petabytes and exabytes of information.
   • Scalability and High Availability: Adding more nodes improves performance without interruptions.
   • Cost Reduction: Uses common, low-cost hardware.
   • Support for Diverse Data: Can handle text files, images, videos, and unstructured data.

4. Google File System and Hadoop

   • Inspiration from GFS: The Hadoop Distributed File System (HDFS) is based on Google’s GFS.
   • Distributed Storage: Splits and replicates large volumes of data across multiple servers.
   • Reliability: If a node fails, data is replicated on other nodes to prevent loss.

5. MapReduce in Hadoop

   • MapReduce: A programming model for processing and analyzing large datasets.
   • Process Phases:
     • Map: Reads input data and converts it into key-value pairs. Distributes data across multiple nodes for parallel processing.
     • Reduce: Takes the results from the "Map" phase and combines them to produce the final output.
   • Advantage: Speeds up data processing through parallelism.

6. Relationship Between Big Data Analytics, Machine Learning, and Cloud Computing

   • Big Data Analytics
     • Utilizes Hadoop and other tools to analyze large volumes of data. Discovery of patterns, trends, and valuable insights.
   • Machine Learning
     • Application of algorithms to learn from data and make predictions. Uses data processed by Hadoop to train models.
   • Cloud Computing
     • Scalable and flexible infrastructure for Big Data and Machine Learning. Allows processing large volumes of data without investing in proprietary hardware. Examples: Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure.