Hadoop

Hadoop

1. Definition

   • Java-based open source framework
     • It is used within BigData for:
       • Treatment
         • large amounts of data
           • structured
           • unstructured
       • Parallel processing
       • Distributed Storage
     • Facilitates vertical scaling

2. Ecosystem

   • Main components
     • Hadoop Distributed File System (HDFS)
       • Feature
         • Local storage is located on each node
         • is distributed among different points connected to a main node
         • Use NameNode to manage
           • The file system namespace
           • Access to files and multiple DataNodes for managing data storage
       • Advantages
         • network latency is eliminated
         • High-performance access to application data
     • Yet Another Resource Negotiator (YARN)
       • Feature
         • Manage
           • Cluster computing resources
           • Hadoop applications.
     • MapReduce
       • parallel processing of large data sets
         • Uses a key/value format.
         • Uses nodes
           • Use instructions to send information
           • Each one processes each subset in parallel with other processing tasks
         • It is divided into stages
           • Stage Map
             • Input data is split into shards
               • and assigned to nodes in the cluster.
               • Each node runs a map() function
               • that transforms the data into intermediate key-value pairs.
           • Shuffle y Sort
             • Intermediate key-value pairs are
               • grouped
                 • to the corresponding nodes according to their keys.
                   • Values ​​associated with the same key are processed together in the next stage.
               • sorted
               • distributed
           • Stage Reduces
             • The nodes execute the reduce() function on the grouped data.
             • The values ​​associated with each key are combined to produce final results
     • Hadoop Common
       • includes libraries and utilities used and shared by other Hadoop modules.
   • Complementary technologies
     • Data ingestion
       • Flume
         • ingest information in data lakes.
           • Distributed and highly efficient service
             • For
               • Distribute
               • Add
               • collect
                 • large amounts of information
       • Sqoop
         • transfer a large volume of data between
           • Hadoop
           • relational database management systems.
       • Kafka
         • facilitates data ingestion into Apache Hadoop
         • Divide the information
     • Data query
       • Hive
         • It has a SQL interface to Hadoop, as if it were a database
       • Hbase
         • distributed and scalable NoSQL database
         • for querying structured data in HDFS
       • Hue
         • Makes writing SQL queries easier
         • connects catalogs to the cluster databases
       • Impala
         • SQL database that runs on Hadoop.
         • Used in Business Intelligence for its low latency
     • Spark
       • data processing engine
       • useful in data analysis for Machine Learning.
     • Oozie
       • Allows
         • the execution
           • Over time of the tasks to be executed within the Hadoop ecosystem
         • planning
     • Pig
       • Framework that uses its own programming language
         • Compiler that generates MapReduce commands
     • Zookeper
       • It is used for the coordination of distributed applications

3. History

   • Late 1900s and late 2000s
     • search engines were created
   • In 2006 Apache Hadoop emerged
     • focused primarily on batch ETL processes for on-premise.
   • 2004
     • The Nutchuts developers set about writing an open source implementation
       • Provided the solution to process the large data sets.
   • 2007
     • Yahoo started using Hadoop on a 1000 node cluster.
   • 2008
     • Hadoop confirmed its success by becoming the top-level project at Apache.
       • it became the fastest system on the planet by sorting through a full terabyte of data.
       • beat supercomputers
   • 2012
     • The second (alpha) release of the Hadoop-2.x series with a more stable version of YARN
       • Hadoop broke records by processing a petabyte of data in less than 17 hours.
       • I know it allowed the execution of applications beyond MapReduce.
   • 2013-2016
     • Tools associated with Hadoop have expanded its ecosystem
       • enabling more advanced and real-time analysis.
   • 2023
     • The data cloud is consolidated as an innovation
       • It focuses on batch processing, but in streaming.

4. advantages

   • Scalability
     • uses a distributed computing model
       • allows
         • fast data processing
         • Rapid scaling when adding computing nodes.
   • Low cost
     • open source framework
       • Can run on commodity hardware
       • has a broad ecosystem of tools
   • Flexibility
     • There is no need to preprocess the data before storing it
   • resilience
     • enables
       • fault tolerance
       • system resilience
         • if one of the hardware nodes fails, tasks are redirected to other nodes
   • High availability
     • Designed to
       • ensure data availability
         • even in the event of node failures.
       • and keep the system running

5. Google File System

   • What is it?
     • Google created a distributed file system
     • intended to provide fast access to data
     • it uses massive groups of common servers.
   • Definition
     • The file system provides a programming interface
   • Features
     • that makes it easy to build applications based on GFS
     • Each Google file system is made up of three main parts:
       • sharded servers
       • clients
       • master servers.
     • It is designed exclusively for Google's internal tools and platforms.
     • It allows for more advanced configurations for replication based on specific needs.
     • It is optimized for specific workloads

6. Relation of

   • Big Data Analytics
     • Generates processed data that serves as input for Machine Learning algorithms.
       • It is done using tools such as HDFS (storage) and MapReduce
   • Machine Learning
     • Learn patterns from data to make predictions
     • It relies on large data sets generated by Big Data Analytics.
     • It can be integrated with frameworks such as
       • Mahout
         • for distributed processing.
       • Spark MLlib
   • Cloud Computing
     • Provides scalable and flexible infrastructure to handle Big Data and run Machine Learning models.
     • Facilitates massive data storage and processing
     • Offers on-demand tools to train and deploy ML models.
   • Features
     • These disciplines complement each other to address challenges of
       • processing
         • big data volumes
       • analyzing
       • extracting