Hadoop

Hadoop

1. It is an open source software framework that serves to store data and run applications on commercial hardware clusters, hadoop nos provides massive storage for any type of data, and enormous processing power and the ability to process concurrent tasks or jobs virtually unlimited.

   • This structure has the following features
     • Has a fault tolerance already that this structure is distributed on a large number of nodes, there is fault tolerance in them because, in the event of a failure, the tasks are transferred transparent to another pair.
       • Data that is not processed before be stored, they give us flexibility, since that both structured data and the unstructured ones such as texts, images and others, can be stored indefinitely, according to requires and decisions are made about the use later.
   • What is the relationship between hadoop and Google File System?
     • Hadoop Distributed File System is developed as part of Hadoop in 2005, HDFS implements ideas GFS key, it is responsible for adopting the approach of split data into large blocks and replicate them to fault tolerance, this optimizes reads and writes sequential in large volumes of data.
       • Google File System developed in 2003 by Google was designed to handle large distributed data volumes on hardware common, this provides high availability and fault tolerance through data replication, This tool introduced the concept of dividing data in large blocks distributed between multiple nodes.
         • The relationship between both concepts is that Hadoop I adopt the concept of distributed file system inspired by Google File System, but I adapt it to a open source environment, making it possible for organizations outside of Google will use it, thanks to This inspiration is that Hadoop Distributed File System became the standard in Big Data solutions, serving as a base for storage in many modern systems.
   • Hadoop timeline
     • In 2003 Google published the Google File System article.
       • In 2004, the article on MapReduce, a model of programming for distributed data processing.
         • In 2005 Hadoop was created as part of the project Apache Nutch, a web search engine.
           • By 2006 Hadoop becomes a project independent at the Apache foundation.
             • In 2007 Yahoo! adopt Hadoop and start use it on a large scale, as it also builds the largest Hadoop cluster with 1000 nodes.
               • By 2008 Apache released the official version 1.0 of Hadoop and Facebook together with LinkedIn use Hadoop to manage large volumes of data.
                 • In 2009 Hadoop became the de facto standard for Big Data processing.
                   • By 2010 Cloudera launches its first distribution Hadoop-based enterprise.
                     • In 2011 Apache launches Hadoop 2.0 with support for YARN o Yet Another Resource Negotiator, which allows you to run multiple applications in parallel on a single cluster.
                       • By 2012, ecosystem tools such as Hive, Pig, HBase and Oozie.
                         • In 2013, Hadoop was adopted in industries beyond technology, such as banking, telecommunications and retail.
                           • For 2014, Apache Spark emerges as complement to Hadoop, offering one more engine fast for in-memory processing.
                             • In 2015 Hadoop is integrated with tools modern analytics, such as TensorFlow and Apache Kafka.
                               • By 2016 Hadoop is optimized for environments in cloud, integrating with AWS, Google Cloud and Azure.
                                 • In 2018, specific distributions emerged for enterprise use cases, such as MapR, Hortonworks, and Cloudera.
                                   • For 2020 onwards, cloud technologies such as Spark or Snowflake have gained ground, Hadoop continues being the basis of many processing systems data in large companies.
   • What is Hadoop MapReduce?
     • It is a processing paradigm data characterized by being divided into two phases or differentiated steps that are Map and Reduces these subprocesses associated with the task are executed in a distributed manner, in different processing nodes or slaves.
   • What is the relationship of Big Data Analytics with Machine Learning and Cloud Computing?
     • The relationship between Big Data Analytics and Machine Learning is that while Big Data Analytics focuses on the analysis of large volumes of structured data and unstructured to extract useful information, while that Machine Learning makes use of algorithms that learn patterns from the data and make predictions or automated decisions, which causes a synergy where Machine Learning models require of large amounts of data to be effective, for what accesses this data, which is provided by Big Data.
       • Regarding the connection between Big Data Analytics and Cloud Computing, we have to generate Big Data enormous volumes of information that require storage and processing, and the cloud offers a scalable and economical solution, which causes a synergy between Cloud infrastructure Computing that provides platforms such as Amazon AWS, Google Cloud and Microsoft Azure for process and store Big Data. It also manages a distributed processing of services such as Hadoop and Spark are optimized to run on cloud environments, which speeds up analysis.
         • Between Machine Learning and Cloud Computing there is a relationship between the cloud that provides tools and resources to train, deploy and scale models of Machine Learning, as there is also a synergy on platforms integrated into services such as Google AI, AWS SageMaker and Azure ML make it easier for us to training models without the need for physical infrastructure. Fruit of this relationship also there is intensive processing such as Cloud Computing that allows us to execute algorithms complexes using powerful GPUS and TPUS.
           • The interaction between the 3 concepts is that Big Data Analytics prepares data for Machine Learning through cleaning, transformation and extraction of characteristics, Machine Learning is responsible for creating models that extract knowledge from these data and Cloud Computing is responsible for supporting both by providing infrastructure and capacity to store and process massive data and train complex models.
   • Hadoop has the following advantages
     • Adaptability: As is the possibility of integrating others systems with him.
       • Scalability: Since it allows scale the system according to data volume grows received.
         • Efficiency: When managing a scheme distributed storage nodes, where each node stores a fragment of the information, and the way in which it is structured processing of data within each node, which makes hadoop offers a system highly efficient, with great processing capacity and high speed.
           • Cost reduction: Being a free software license can be acquire at no cost.
             • Flexibility: Can be stored as much data as we want and decide how we want to use them.
               • Error management: When operating in different nodes is that if one fails, neither the structure and data are not visible committed, since their tasks are they pass to another of the nodes.