Hadoop

Hadoop Development Timeline

Advantages

Distributed File Systems (DFS)

Relationship of Big Data analytics with machine learning and cloud computing.

the Hadoop ecosystem

Open Source

Big data management

Support for multiple languages

Integration with other tools

license-free

Authentication, authorization, encryption

Features

Security

Flexible

Ideal for data lakes and analysis tests

HDFS

YARN

MapReduce

HBase

Hive

Spark

Pig

Oozie

Sqoop

Zookeeper

Flume

Mahout

Presto

In charge of

Store data in distributed blocks.

Manages

Resources and task planning

Process

Parallel data

NoSQL database over HDFS

SQL on Hadoop

In-memory processing

Scripting language

Workflow planning

Integration with relational databases

Distributed coordination

Streaming data ingestion

Machine Learning

Low latency distributed SQL queries

Low cost

Structured, semi-structured and unstructured data

Fault Tolerant

Data replication in HDFS

(2002-2005) - Apache Nutch Project

Initial Implementation

(February 2006) - Birth of Hadoop

Development and Adoption

(April 2006) - Hadoop 0.1.0

Release of the first official version of Apache Hadoop

(2008) - Hadoop at the ASF

Hadoop becomes an Apache Software Foundation project

(2011) - Hadoop 1.0

Release of Hadoop 1.0 (stable version)

(2012) - Hadoop 2.0.0-alpha with YARN

Version 2.0.0-alpha introduces YARN, separating resource management from MapReduce

(2013) - 2.2.0 with YARN

Release Version 2.2.0 (stable)

(2017) - Hadoop 3.0.0 released.

Improved cloud integration

(2017- 2024) - Hadoop 3.x releases.

Progressive improvements (v.3.1 to 3.4), latest version: Hadoop 3.4.0 in February 2024.

Allow

Big data

access to files in multiple locations

Through

a network

Google File System (GFS)

Hadoop Distributed File System (HDFS)

Open-source implementation

Used in

Files divided in 64MB chunks

Master + Chunkservers+ Clients

NameNode + DataNodes

fault tolerance

Through

3-copy replication

Developed

for

mass storage

Designed

for

large data volumes

Features

by Google

Features

Uses

Big Data Analytics

Machine Learning

Distributed Storage

Machine learning

Cloud

provides data to train models of

used as a support for the

Facilitates storage, processing and deployment for

extracts patterns/predictions