HADOOP

HADOOP

Hadoop is an open-source framework used to store and process large datasets in a distributed computing environment. According to (SAS Institute, s.f), Hadoop is an open-source software platform, meaning its code is publicly available for use, modification, and distribution by anyone. This platform can store and process large amounts of data using clusters of common hardware, which consist of multiple interconnected machines working together.

Characteristics

MapReduce in Hadoop

Google File System and Hadoop.

Disadavantages

Hadoop advantages

HADOOP Timeline: Some of the statements were taken from (SAS Institute, s.f).

environment of Hadoop

Open source: Source code available for

Modification

Aplication

Distribution

Scalability: Hadoop captures large volumes of data and processes them:

Data distribution

Nodes

Clústering

fault tolerance: designed to handle hardware failures

Data is replicated across multiple nodes

Flexibiity: It can process any type of data

Structured

Unstructured

Semi-structured

Efficiency: Process arge data sets

MapReduce

Clústeres de Hardware Común: uses inexpensive and standard hardware

Ram memory and others.

Standard server

Hard drives

Tools

MapReduce

HDFS

YARN

Projects

Hive: Data storage system.

HBase: Distributed and scalable NoSQL database

Spark: In-memory data processing engine

Pig: Platform for data analysis.

1999

2002

2004

2006

2008

2009

Apache software foundation (ASF) formed as a non-profit,(SAS Institute, s.f).

The web search engine is created, it crawls and indexes web pages on a large scale (Apache Nutch).

Google shares documents about the Google File System (GFS) and MapReduce.

The founders part ways, and Cutting takes control of Nutch and joins Yahoo.

Nutch splits, and thus Hadoop is born.

Doug Cutting leaves Yahoo to join Cloudera to continue Hadoop development

Yahoo launches Hadoop as an open-source project for AFS.

2011

Yahoo spins off Hortonworks as commercial Hadoop distro.

2013

MapR technologies releases Hadoop distro

Greenplum releases Hadoop distro, Pivotal HD

2012

Introduction of YAR

2020

Introduction to Hadoop Ozone.

Some of the advantages are already mentioned so they will not be defined.

Scalability

Cost-effectiveness

Fault Tolerance

Flexibility

Limited multi-user writing

Complexity of use

Security concerns

Inefficiency with small files

Googe File System

Hadoop

Developed by Google

Data processing for Google search engine and other applications

Deveoped by Apache Software Foundation

Large-scale data processing for various applications

Includes a single Namenode and multiple DataNodes

It consists of a master server and multiple shard servers, which store data in fixed-size chunks.

Data is collected on multiple shard servers

Data is collected on multiple DataNodes

HDFS was inspired by GFS

Distributed File Systems

Both provide reliable access to data using commodity hardware.

Programming model

Efficiency

Data processing with distributed and parallel algorithm

Allows for parallel processing, making it faster and more scalable.

Two main functions

Map (filters and sorts data) and Reduce (aggregates results)

BIg Data Analytics- Machine Lesarning- Cloud Computing

Big Data Analytics

Machine Learning

Cloud Computing

involves inspecting diverse data sets to discover hidden patterns.

allows you to learn from data through algorithms that identify hidden patterns and make predictions based on large data sets

allows you to store, process, and analyze large volumes of data and provides the resources needed for machine learning.

Relation

Scalability

Data Storage and Processing

Real-Time Analytics

Cost-Efficiency

Colaboration and integration