Hadoop

Hadoop

1. Characteristics

   • Distributed Storage
     • Distributed File System (HDFS)
       • Store data across multiple machines
         • high availability
           • fault tolerance
             • Data is split into blocks
   • Scalability
     • clusters can scale horizontally
       • adding more nodes.
   • MapReduce Processing
     • parallel processing
       • The Map step divides a task
         • Reduce step aggregates the results.
   • Cost-Effective
     • open-source solution
       • run on commodity hardware
         • it affordable for large-scale data processing
           • Data Locality Optimization
   • Data Locality Optimization
     • computation to where the data is stored
     • minimizing network congestion
     • improving speed by processing data

2. Environment

   • Cluster-Based Architecture
     • deployed in a cluster environment
       • multiple nodes work together
   • Commodity Hardware
     • often use inexpensive
       • off-the-shelf hardware rather than high-end servers
         • reduces costs but requires redundancy
           • fault tolerance since commodity hardware
   • Distributed Computing
     • environment is distributed
       • meaning computation
         • data storage are spread across multiple nodes
           • supports high availability
             • parallel processing
   • Batch Processing
     • best suited for batch processing rather than real-time analytics
       • MapReduce framework processes data in large blocks
   • High-Volume Data
     • is optimized for environments with massive volumes of structured, semi-structured, and unstructured data
       • architecture is ideal for applications with high storage demands and complex processing requirements.
   • Security and Governance
     • distributions include Kerberos authentication
       • HDFS file permissions
         • encryption
           • integration with access management tools

3. Timeline of the development

   • 1999 apache fundation
     • 2002 - Nutch created
       • 2006 - Cutting joins yahoo takes Nutch
         • 2008 - Nutch divides and born Hadoop
           • 2008 . Yahoo release Hadoop as open sourse proyect
             • 2008 hadoop based starts up cloudera inc
               • Cutting leaves Yahoo for Cloudera
                 • 2011 - Yahoo spins off Hortonworks
                   • 2011 - MapR technologies releases hadoop distro
                     • 2013 - greenplum releases hadoop distro

4. Ecosystem

   • Core Componets
     • YARN ((Yet Another Resource Negotiator) cluster management component that handles resource allocation and task scheduling for various applications
       • HDFS distributed file storage system that splits data into blocks
         • MapReduce: A programming model for parallel processing of large datasets. It consists of two main functions:
   • Data Storage and Management
     • Sqoop isa tool for transferring structured data between Hadoop and relational databases
       • Flume: A service for collecting and moving large volumes of log data to HDFS or other storage systems, commonly used in real-time data ingestion
         • Kafka is a distributed messaging system designed to handle large volumes of streaming data. It integrates well with Hadoop for real-time processing.
   • Data Processing and Analysis
     • Hive is a data warehousing and SQL-like query language tool that makes it easier to work with data on HDFS. Hive is useful for ETL processes, reporting, and SQL-based analytics on large datasets.
       • Pig is a high-level scripting language for data transformation and analysis.
         • Spark is a fast, in-memory data processing engine for big data analytics. It integrates with Hadoop and supports batch processing, real-time analytics, and machine learning.
           • Tez is an optimized data-processing framework that offers an alternative to MapReduce.
             • Flink is a stream-processing framework that supports real-time processing and integrates with Hadoop to provide low-latency analytics on streaming data.
   • Data Access and Querying
     • Drill: A schema-free, SQL-based query engine that allows for interactive analysis of large-scale datasets.
       • Impala: A fast, SQL query engine that provides low-latency, real-time querying of data stored in HDFS and HBase.
         • Phoenix: A SQL query layer for HBase, enabling users to execute low-latency queries on top of HBase tables using SQL syntax.
   • Machine Learning and Data Science
     • Mahout: A scalable machine learning library that provides algorithms for clustering, classification, and collaborative filtering.
       • MLlib (Spark’s Machine Learning Library): A library within Apache Spark that provides machine learning algorithms and utilities.
         • H2O.ai: An open-source machine learning platform that integrates with Hadoop to provide advanced machine learning capabilities.
   • Workflow and Coordination
     • Oozie: A workflow scheduler that manages complex job dependencies and workflows in Hadoop.
       • Zookeeper: A centralized service for coordinating distributed applications and managing configuration and synchronization.
   • Data Serialization and Storage Formats
     • Avro: A data serialization system used for structured data serialization in Hadoop. Avro provides efficient and fast data interchange, especially for streaming data.
       • Parquet: A columnar storage format that enables efficient data storage and retrieval. Parquet is widely used in the Hadoop ecosystem due to its high performance, especially for analytical workloads.
         • ORC (Optimized Row Columnar): A storage format designed to improve data reading, writing, and processing performance in Hadoop, particularly with Hive.
   • Security and Governance
     • Ranger: A framework for managing security and data access policies across Hadoop. Ranger provides role-based access control (RBAC) and audits for compliance.
       • Knox: A security gateway that provides perimeter security for Hadoop clusters, allowing controlled access to Hadoop services from external clients.
         • Atlas: A data governance and metadata management tool that provides data lineage, classification, and auditing for Hadoop clusters.
   • Monitoring and Management
     • Ambari: is aweb-based tool for provisioning, managing, and monitoring Hadoop clusters. Ambari simplifies cluster configuration, deployment, and troubleshooting.
       • Ganglia: A distributed monitoring system for large clusters, often used to track metrics and health of nodes within a Hadoop environment.
         • Nagios: An open-source monitoring tool that can be used with Hadoop for alerting and monitoring of nodes, services, and applications.

5. Google File System and Hadoop.

6. MapReduce Workflow

   • Splitting
     • Input data (typically stored in HDFS) is split into smaller blocks. Each block is then processed independently by a mapper.
   • Mapping (Map Phase)
     • In the map phase, the input data is processed by mapper functions, which transform the data into key-value pairs. Each map task operates on a portion of the data in parallel, improving processing speed and efficiency.
   • Shuffling and Sorting:
     • After mapping, all key-value pairs with the same key are grouped together. Hadoop sorts these key-value pairs, making it easier for the reducer to process related data in the next stage.
   • Reducing (Reduce Phase)
     • In the reduce phase, each group of sorted key-value pairs is processed by a reducer function, which aggregates or combines the values associated with each key. The results are typically stored back into HDFS.

7. Big Data Analytics, Machine Learning, and Cloud Computing form a symbiotic relationship

   • Big Data provides the vast and diverse datasets necessary for ML algorithms,
     • Machine Learning derives insights from this data, and predictions
       • Cloud Computing supplies the resources needed to store, process, and analyze this data efficiently, allowing for scalable and flexible solutions. Together, they empower organizations to gain deeper insights and make data-driven decisions faster and more effectively than ever before.

Imágenes y archivos del mapa