Hadoop

Hadoop

1. Characteristics and environment

   • Mass Storage: Hadoop offers massive storage for any type of data.
     • Parallel processing: Hadoop uses parallel processing to divide workloads into smaller loads that can run at the same time
       • Scalability: Hadoop allows you to scale quickly by adding compute nodes.
         • Flexibility: Hadoop allows data to be stored flexibly, without the need to preprocess it before storing it.
           • Fault Tolerance: Hadoop enables fault tolerance and system resilience.
             • Integration with other services: Hadoop can integrate with other cloud services, such as Amazon S3, Amazon Kinesis, Amazon Redshift, and Amazon DynamoDB.
               • Ecosystem: The Apache Hadoop ecosystem includes HDFS, MapReduce, YARN, and other complementary technologies.
                 • Origin: Hadoop was created in the 2000s as part of search engine indexing.
                   • Name: The project was called Hadoop in honor of the creator's son's toy elephant.

2. Timeline of the development

   • 1999: Apache Software foundation ASF formed as a non - profit
     • 2002: Nutch created by doug cutting and mike cafarella
       • 2006: cutting joins yahoo, takes nutch with him
         • 2008: Nutch divided and hadoop is born
           • 2008: Yahoo releases hadoop as open-source project to ASF
             • 2008: hadoop-based start-up cloudera incorporated
               • 2009: cutting leaves yahoo for cloudera
                 • 2011: yahoo spins off hortonworks as commercial hadoop distri
                   • 2011: MapR technologies releases hadoop distri
                     • 2013: Greemplum releases hadoop distra pivotal Hd

3. advantages

   • Storage: Hadoop can store and process data of any type, regardless of its volume, format, structuring, or source.
     • Scalability: Hadoop can scale from a single server to thousands of machines, allowing projects to have no limits on storage or processing power.
       • Low cost: Hadoop is an open source project, so the software is free to use. Additionally, it uses general-purpose hardware, which reduces costs compared to other systems.
         • Fault tolerance: Hadoop replicates data across multiple nodes, so the loss of one or more nodes does not affect data integrity.
           • Flexibility: Hadoop can process structured, semi-structured and unstructured data.
             • Integration with other services: Hadoop can be integrated with other cloud services, such as Amazon S3, Amazon Kinesis, Amazon Redshift, Amazon DynamoDB, and Amazon EMR.

4. Google File System and Hadoop

   • The Google File System (GFS) file system is proprietary to Google and is not available for external use, while the Hadoop Distributed File System (HDFS) file system is an open source project:
     • Google File System (GFS): It is property of Google and is not available for external use.
       • Hadoop Distributed File System (HDFS): It is a distributed file system that is a core component of Apache Hadoop. HDFS is an open source project maintained by the Apache Software Foundation.
         • Hadoop is an open source framework used to store and process large amounts of data. It relies on distributed storage and parallel processing to handle large amounts of data.
           • Some features of Hadoop are:
             • Can run on basic hardware
               • It allows you to create clusters of multiple computers to analyze massive data sets in parallel.
                 • It supports applications that have data sets that are typically gigabytes to terabytes in size.
                   • It accommodates streaming access to data sets.
                     • It is designed to automatically detect failures and recover quickly.

5. MapReduce in Hadoop

   • Previously, in traditional systems, technologies have focused on bringing data to storage systems. However, in Hadoop processes, it is about bringing the processing closer to the place where the data is stored and thus take advantage of parallelization techniques, significantly increasing the scalability and performance of systems that work with large amounts of data.
     • Hadoop MapReduce is a data processing paradigm characterized by being divided into two distinct phases or steps: Map and Reduce. These threads associated with the task are executed in a distributed manner, on different processing nodes or slaves. To control and manage its execution, there is a Master or Job Tracker process. It is also in charge of accepting new jobs sent to the system by clients.
       • This processing system is supported by distributed data storage technologies, in whose nodes these map and reduce type operations are executed. Hadoop's distributed file system is HDFS (Hadoop Distributed File System), responsible for storing files divided into data blocks. HDFS provides the pre-splitting of data into blocks that MapReduce needs to execute. The results of the processing can be stored in the same storage system or in a database or external system.

6. Relation of Big Data Analytics with Machine Learning and Cloud Computing

   • Machine Learning is a highly interdisciplinary field of computer science that focuses mainly on building models based on learning algorithms that impact almost all scientific disciplines, from bioinformatics, information retrieval or statistics. Machine Learning algorithms can be divided into three categories: supervised, unsupervised, and reinforcement learning.
     • Machine Learning Algorithm Categories:
       • Supervised learning makes decisions based on logic provided by an algorithm that takes 'labeled' input data. Supervised learning performs classification and regression data processing tasks using algorithms such as SVM (Support Vector Machine), Naive Bayes, or computational and statistical classifiers.
         • Unsupervised learning algorithms discover patterns and behaviors in data to segment information and learn more about it. Algorithms such as k-means, or self-organizing maps (SOM) are part of unsupervised learning.
           • Reinforcement learning (RL) is inspired by behavioral psychology on the idea of ​​providing a reward or punishment for actions performed by software agents in a given environment.