Hadoop and Big Data

Hadoop and Big Data

1. Environment of Hadoop

   • HDFS
     • File management
     • Allows to scale a single cluster to hundreds of nodes.
     • Designed for fast recovery from failures
   • YARN
     • Allocates system-wide resources
     • Constant monitoring of resources
   • MapReduce
     • Allows scalability
     • Creates tuples and reduces them to specify information
   • Hive
     • Integrated open source query and analysis system for your data warehouse
     • Data is initially stored unchanged and not in a predefined schema
   • HBase
     • Non-relational data management system
     • Row and column based database
     • It is flexible to make changes at any time.
   • Zookeeper
     • Coordination service for distributed applications,

2. Characteristics of Hadoop

   • Ability to store and process large amounts of any type of data quickly.
   • Multiple copies of all data are automatically stored.
   • You can store as much data as you want and decide how to use it later.
   • Multiple nodes are created as needed

3. Timeline Hadoop

   • 2002
     • Doug Cutting and Mike Cafarella start work on new search engine
   • 2003
     • GFS inspires HDFS development
   • 2006
     • Yahoo backs Doug Cutting to develop Hadoop
     • Hadoop becomes part of Apache Lucene project
   • 2008
     • Introduction of key concepts such as JobTracker and TaskTracker.
   • 2013
     • It includes YARN, which separates system resource management from data processing.
   • 2017
     • Hadoop is consolidating its position as the main technology for massive data processing.

4. Advantages of hadoop

   • Computing power
     • Procesa big data a gran velocidad.
     • The more compute nodes you use, the more processing power you have.
   • Fault tolerance
     • Protected against hardware failures.
     • Multiple copies of all data are stored automatically.
   • Flexibility
     • You do not have to pre-process data before storing it.
     • You can store as much data as you want
   • Low cost
     • Open source framework is free of charge
   • Scalability
     • You can easily grow your system to process more data by adding nodes.

5. MapReduce in Hadoop

   • It is a data processing paradigm characterized by being divided into two distinct phases or steps: Map and Reduce.
     • These sub-processes associated with the task are executed in a distributed manner, in different processing nodes or slaves. To control and manage their execution, there is a Master or Job Tracker process.
   • It is necessary that DDFS first distributes the files
   • The input data is divided into independent fragments that are processed by the mappers in parallel.
   • Implements automatic recovery mechanisms in case of hardware or software failures.

6. Google File System (GFS)

   • Distributed file system designed by Google, due to its capabilities to handle large volumes of data efficiently on hardware clusters
   • the creators of Hadoop, took it as the basis for building an open-source distributed storage system, which later became HDFS.

7. Relation of Big Data Analytics with Machine Learning and Cloud Computing.

   • Enables organizations to manage and analyze large volumes of data
   • extract valuable knowledge
   • Automate complex processes efficiently and scalably