Hadoop

Hadoop

Main Features

Hadoop environment

Advantages of hadoop

Chronology of Hadoop Development

Fault Tolerance

Distributed Storage and Processing

Flexibility

Scalability

Cluster

HDFS

Hadoop Ecosystem

MapReduce

If a node fails, the system automatically redirects tasks to other available nodes without interrupting the process

Uses the MapReduce programming model to process distributed data in parallel across the cluster.

Designed to handle large amounts of data across multiple nodes in a cluster

It supports a wide variety of data types, AND can integrate with different technologies and tools for data analysis.

Composed of multiple nodes running Hadoop software, including storage nodes (DataNodes) and processing nodes (NodeManagers).

Distributed file system designed to store large data sets in a distributed manner on the Hadoop cluster.

Set of complementary tools that integrate for various functions, such as Apache Hive for SQL queries, Apache Spark for in-memory processing, Apache HBase for NoSQL databases, among others.

Programming framework for distributed data processing in the Hadoop cluster.

2005: Doug Cutting and Mike Cafarella develop Hadoop as an open source project, inspired by Google whitepapers

2008: Hadoop version 0.18.0 is released, including the Hadoop Distributed File System (HDFS) and the MapReduce programming model.

2012: Hadoop 1.0.0 is released as the first stable version of the project, marking an important milestone in its development.

2013: Hadoop 2.0.0 is released, which includes significant improvements, such as the introduction of YARN (Yet Another Resource Negotiator), a framework that allows data processing applications to run not only with MapReduce, but also with other models. programming.

2014: Hadoop becomes an integral part of many large-scale data storage and analysis systems around the world, used by companies such as Facebook, Twitter, LinkedIn, and many others.

Hadoop offers a robust, cost-effective, and highly scalable solution for storing and processing large volumes of data, making it a popular choice for businesses facing Big Data challenges.

Relationship of Big Data Analytics with Machine Learning and Cloud Computing.

Google file system and Hadoop

Big Data Analytics

Machine Learning

Cloud Computing

It allows computers to learn and adapt on their own, without needing to be explicitly programmed for every situation. Instead, machines analyze data, discover patterns, and use those patterns to make decisions or predictions. This makes them perfect for tackling complex or constantly evolving data sets.

Data analytics is the practice of examining large amounts of information to uncover patterns, trends, and connections. This helps businesses make better decisions. It involves using advanced techniques like statistics, data mining, and predictive analysis to get valuable insights from both organized and unorganized data

Cloud computing is an on-demand delivery model for IT resources like storage, processing power, and applications. These resources are provided over the internet and can be scaled up or down based on your needs. This eliminates the need for companies to maintain their own physical servers and allows them to access powerful computing resources at a lower cost and with more flexibility.

DESIGN

GFS, WAS DEVELOPED BY Google, HDFS, Developed as part of the Apache Software Foundation's Hadoop project.

APPROACH

GFS, handles large files, in the range of several gigabytes or even terabytes, while HDFS was designed to handle large volumes of data, divided into fixed size blocks, in the range of megabytes or even gigabytes.

WRITE

GFS mainly supports append write operations, it means that files are mainly write-only or read-only, but are not modified after the initial write, on the other hand HDFS supports write and random read operations, allowing file modification after the initial writing.

Synchronous and Asynchronous Write

GFS prioritizes data consistency and uses synchronous writing, ensuring that data is written to multiple nodes synchronously before committing the write to the client, while HDFS prioritizes performance and uses asynchronous writing, i.e. that data is written to disk without waiting for synchronous confirmation across multiple nodes.

LATENCY

FAULT TOLERANCE

Due to its focus on consistency, GFS may have higher latency compared to HDFS for certain read and write operations.

HDFS is designed to be highly fault tolerant, with built-in fault detection and data replication mechanisms to ensure data availability and integrity.