Hadoop

Hadoop

Characteristics of Hadoop.

Open-Source System: Licensed under Apache.

Distributed Processing: Data is split across cluster nodes.

Horizontal Scalability: Expand capacity by adding inexpensive hardware.

Fault Tolerance: Automatic data replication.

High Availability: Redundant services.

Cluster-Based: Operates on commodity hardware.

environment of Hadoop.

Data centers, cloud platforms (AWS, Azure, GCP), hybrid deployments.

Timeline of Hadoop Development

2003: Google publishes Google File System (GFS).

2004: Google releases MapReduce paper.

2005: Doug Cutting and Mike Cafarella develop Hadoop for Nutch.

2006: Hadoop becomes an independent Apache project.

2008: Hadoop became top-level project as apache.

Hadoop defeated supercomputers

2011: Commercial support grows (Hortonworks, Cloudera, MapR).

Apache released first stable version 1.0

2012: Hadoop 2.0 version wich contains YARN was released.

2017: Hadoop 3.0 was released.

2020: Hadoop 3.1.3 was released.

2021: AWS and Google cloud introduce serverless options for their big data managed services

2024: Hadoop 3.4 is the latest version of hadoop.

Present: Hadoop coexists with Spark, Flink, and cloud services.

Advantages of Hadoop

High Scalability: Handles thousands of nodes.

Cost-Effective: Uses standard hardware.

Processes Huge Volumes: Terabytes to petabytes.

Flexible Data Handling: Structured, semi-structured, and unstructured data.

Rich Ecosystem: Includes Hive, Pig, HBase, Spark.

Cloud Integration: Supports elastic infrastructure.

Google File System and Hadoop

GFS

Inspiration for Hadoop

Distributed file system for handling large files on commodity servers.

Concepts: large blocks, replication, NameNode, DataNodes.

HDFS (Hadoop Distributed File System) mirrors GFS principles.

MapReduce in Hadoop follows Google’s processing model.

MapReduce in Hadoop

Programming Model:

Distributed Execution: Parallel task processing across nodes.

Map: Processes input into key-value pairs.

Reduce: Aggregates intermediate results.

Efficient for Batch Processing: Ideal for massive data sets on HDFS.

Relation of Big Data Analytics with Machine Learning and Cloud Computing

Big Data Analytics:

Machine Learning:

Cloud Computing:

Extracts patterns and insights from large datasets.

Tools: Hadoop, Spark.

Models patterns from Big Data.

Big Data fuels ML model training.

Provides scalable, flexible infrastructure.

Services: AWS EMR, Azure HDInsight, Google Dataproc.