Hadoop

Hadoop

1. Features

   • Storage Capacity: Hadoop can use a cluster with many nodes to offer massive data storage and processing power at a price suitable for businesses.
     • Fast Data Access: Hadoop provides the ability to quickly access data after processing. Hadoop processes data sequentially rather than randomly, thus greatly reducing the input and output load.
       • Fault Tolerance and Scalability: Hadoop offers high reliability because, due to its design and implementation, it detects errors and returns the execution using different nodes. It also supports high scalability, allowing multiple servers to be added to the cluster and used for data storage and processing.
         • Simple Programming Model (MapReduce): It uses the MapReduce paradigm to divide tasks into two phases: mapping and reducing, facilitating parallel processing.

2. Environment

   • HDFS (Hadoop Distributed File System): A distributed file system that stores data in replicated blocks across multiple nodes to ensure availability and fault tolerance.
     • MapReduce: A programming model that divides tasks into two phases: Map (processing) and Reduce (aggregation), facilitating parallel processing.
       • YARN (Yet Another Resource Negotiator): A resource manager that coordinates task execution across the cluster, allowing multiple applications to efficiently share resources.
         • Hadoop Common: A set of utilities and libraries required for the other modules to function properly.

3. TIMELINE

4. ADVANTAGES

   • Affordable cost: Runs on commodity hardware, significantly reducing infrastructure costs.
     • Data flexibility: Supports structured, semi-structured, and unstructured data, making it ideal for diverse sources such as social media, sensors, logs, etc.

5. GFS

   • Google File System (GFS) and Hadoop Distributed File System (HDFS) are two distributed file systems designed to handle large volumes of data, but they have key differences in their purpose, architecture, and use. A clear comparison is presented:

6. MapReduce in Hadoop

   • This is a programming model for distributed jobs. It processes data in parallel, then performs a mapping (map) and a reduction (reduce) process. The main idea is for developers to write the code, scheduling the tasks that MapReduce must perform so that Hadoop can execute it using HDFS for rapid data access.
     • How does a MapReduce program, also called a Job, work? Initially, the program execution starts when the client sends the configuration to the JobTracker. This Hadoop service is responsible for distributing MapReduce tasks to specific nodes in the cluster. This configuration specifies the Map, Shuttle (combine), and Reduce functions, as well as data I/O.

7. Relation of Big Data Analytics with Machine Learning and Cloud Computing.

   • Big data provides vast amounts of information that serve as the basis for training machine learning models, which extract patterns, make predictions, and automate decisions. These processes require a powerful and scalable infrastructure, which is precisely what cloud computing offers: flexible storage, distributed processing, and remote access to advanced tools. Together, these technologies make it possible to transform massive data into useful knowledge, driving innovation in sectors such as healthcare, finance, education, and more.

Imagens e arquivos do mapa