Quiz
Modulo 7: Fundamental Big Data engineering
Perguntas do quiz (149)
1Data engineering
- Ais the field of developing, testing, deploying and maintaining data processing solutions via collecting, parsing, transforming, joining processing and managing data
- Bmake data avalaible for data scientists to develop models and data products
- Ctwo main activities that comprise data _____________include storage and processing of data
- Dis tasked with making data amenable to various types of data analyses, including model development (data mining and other business-process specific algorithms) and reporting)
2Big Data engineering
- Atwo main activities that comprise ____________ include storage and processing of data, wich is typically structured.
- Bwithin the realm of__________________ involves developing highly distributed, scalable, fault-tolerant data processing solutions to process large amounts of data in order to garner insights
- Ccomprises data processing in support of the Big Data analysis lifecycle.
- Dmake data available for data scientists to develop models and data products
- EThey are required to have knowledge of various data storage and processing technology alternatives for acquiring, storing and processing tecnology alternatives for acquiring, storing and processing data that is often semi-structured and unstructured in nature
3Many of challenges faced in Big Data engineering are related to managing the three primary V of big data:
Volume- Aprocessing large amounts of structured, unstructured, and semi-structured data arriving at a fast pace, including extraction of relevant data from semi-structured and unstructured datasets
- BInternet-scale datasets and associated batch an realtime data processing
- Ccollection and aggregation of data from multiple sources with disparate schemas or without any schema
4Many of challenges faced in Big Data engineering are related to managing the three primary V of big data:
Velocity- AInternet-scale datasets and associated batch an realtime data processing
- Bcollection and aggregation of data from multiple sources with disparate schemas or without any schema
- Cprocessing large amounts of structured, unstructured, and semi-structured data arriving at a fast pace, including extraction of relevant data from semi-structured and unstructured datasets
5Many of challenges faced in Big Data engineering are related to managing the three primary V of big data:
Variety- Aprocessing large amounts of structured, unstructured, and semi-structured data arriving at a fast pace, including extraction of relevant data from semi-structured and unstructured datasets
- BInternet-scale datasets and associated batch an realtime data processing
- Ccollection and aggregation of data from multiple sources with disparate schemas or without any schema
6Big Data engineering challenges:
- Acomprises data processing in support of the big Data analysis lifecycle
- Bimporting/exporting large amounts of data from to traditional storage technologies, including OLTP (CRM, ERP, SCM systems) and OLAP systems (data warehouse)
- Cfield of developing, testing deploying and maintaining data processing solutions via collecting, parsing transforming, joining, processing and managing data
- Dvalidating and cleansing data in realtime or batch mode and creating efficient data models
- Establishing an optimal data storage and processing enviroment based on the type of data and its processing requirements
- Fdeveloping efficient data processing algorithms that run over cluster of computers
- Gdeveloping Big Data pipelines and Big Data applications that may include meaningful data visualizations
7The big Data Engineer certification provides in_depth knowledge of the concepts ans characteristics of
- Astorage devices nd
- Banalytics engine
- Cprocessing engine
- Dresource manager
8data can either be stored using __________ or devices
- Adisk_based
- Brealtime
- Cbatch
- Dmemory_based
9generally data needs to be stored to a disk before it can be processed. However, this only applies to ____________ mode
- Adisk_based
- Bbatch processing
- Cmemory_based
- Drealtime
10In ________________mode data is first processed in memory and then stored to the disk
- Abatch processing mode
- Brealtime processing
- Cdisk_based
- Dmemory based
11The acquiered data is generally not in a format or a structure that can be directly processed, Therefore, as a result of data wrangling activity (data cleansing, data filtering, data preparing), it needs to be stored again
- AVerdadeiro
- BFalso
12Data should not be needs to be stored once it gets processed, as a result of analytics and for archival purposes
- AVerdadeiro
- BFalso
13storage is generally required
- AWhen data gets processed as a result of an ETL activity or uotput in generated as a result of an analytical operation
- BWhen datasets are acquired or when data gets generated inside the enterprise boundary
- CIn realtime processing mode, data is first processed in memory and then stored to the disk
- DWHen data is manipulated for making it amenable to data analysis
14__________is the process of horizontally partitioning a large dataset into a collection of smaller, more manageable datasets called ____
- ACA, theorem
- Breplication, partition
- CACID, unique
- DSharding, shard
- EBASE, atom
15the ____ are distributed across multiple nodes, where a node is a server or a machine
- Amaster-slave
- Bshard
- Cmap
16Shard
- Ais stored on a separate node and each node is responsible for only the data stored on it
- Ball_______ collectively represent the complete database
- Cshares the same schema
- Dstores muktiplecopies of a database
17Sharding
- A____________ distributes a processing load across multiple nodes to achieve horizontal scalability
- B___________ may or not be transparent to the client
- Csupports horizontal scaling as a method for increasing resource capacity. This is accomplished by adding similar or higher capacity resources alongside the existing resources
- DOnce saved, the data is replicated over to multiple slave nodes
- ESInce each node is responsible for a part of the whole dataset, read/write times are greatly improved
18Sharding: depending on the query, data may need to be fetched from both shards
- AVerdadeiro
- BFalso
19A benefit of________ is that it provides partial tolerance towards failures. In case of a node failure, only data stored on that node is affected
- Areplication
- Bsharding
- Cpartitioning
20sharding: with regards to data partitioning, query patterns need to be taken into a account so that shards thenselves do not become performance bottlenecks
- AVerdadeiro
- BFalso
21For example, queries requiring data from multiple shards will impose performance penalties. Data locality, or keeping commonly accessed data collodcated on a single shard, helps to counter such performance issues
- Asharding
- Breplication
- CAcid
22___________stores multiple copies of a dataset, know as _________, on multiple nodes. This providese for scalability, availability, and faulttolerance
- Areplicaction, replicas
- Bsharding, shard
- Cmaster, slave
23Replicaction: Methods of implementing
- Ashard
- Bmaster-slave
- Cpeer-to-peer
- Dpartition
24Nodes are arranged in a _____________ configuration where all data is inserted and updated via a _____node
- Apeer-to-peer, peer
- Bmaster-slave, master
- Csharding, shard
25once saved, the data is replicated over multiple _____nodes
- Asharding, shard
- Bpeer-to-peer, peer
- Cmaster-slave, slave
26All external write requests (insert, update, delete) occur on the master node, whereas data can be read from any slave node
- AReplication
- Bpeer-to-peer
- Cmaster-slave
27Replicaction: master-slave
- Aideal for read intensive loads rather than write intensive loads, as growing read demands can be fulfilled simply via horizontal scaling wich adds additional slave nodes
- BWrites are consistent as all writes are coordinated by the master node. This means that write performance suffers as the amount of writes increases
- CSince each node is responsible for a part of a whole dataset, read/write times are greatly improved
- Din case the master node fails, reads are still possible via any of the slave nodes
- Ea slave node can be configures as d backup node for the master node
28Replicaction: master-slave
- AIn case of master node failure, writes are not supported until a master node is reestablished. It is either resurrected from a backup of the master node, o a new master node is chosen from slave nodes.
- BFor example, queries requiring data from multiple shards will impose performance penalties. Data locality, or keeping commonly accesed data collocated on a sngle shard, helps to counter such performance issues
- CRead inconsistency can be an issue it a slave node is read prior to the update being propagated over to it from the master node
- DTo ensure read consistency, a voting system can be implemented where a read is declared consistent if the majority of the slave nodes contain the same version of the record. Implementation of such a voting system required reliable and fast communication mechanism between the slave nodes
29all nodes operate at the same level.
- AReplication: peer-to-peer
- BSharding
- CReplication: master-slave
30Replication: Each node, know as a __, is equality capable of handing reads and writes
- Ashard
- Bpeer
- Cslave
31Replication: Peer-to-peer
- Aread inconsistency can be an issue if a principal node is read prior the update being propagated over to it
- BEach write is copied to all peers
- Cprone to write inconsistencies that occur as a result of a simultaneous update of the same data across multiple ___
- DThis can be addessed by implementing pessimistic or optimistic concurrency.
32Replication: peer-to-peer: pessimistic concurrency
- Ais a proactive approach that uses locking to ensure that only one update succeds at a time. however, this affects availability as a the database remains unavailable until all locks are released
- Bis a reactive approach that does not use locking. Instead, it allows the inconsistency to happen first and restores consistency after the fact
33Replication: peer-to-peer
- AReads can be inconsistent between the time period when some of the ____ have been update while the others are being updated. However, reads eventually become consistent when the updates have been copied over to all ____
- BIdeal for read intensive loads rather than write intensive loads, as growing read demands can be fufilled simply via horizontal scaling wich adds additional ____ nodes
- CTo ensure read consistency, a voting system can be implemented where a read is declared consistent if the majority of the _ contain the same version of the record
- DImplementation of such a voting system requires a reliable and fast communication mechanism between the ____
34To improve on the limited fault tolerance offered by sharding, while additionally benefiting from the increased availability and scalability of replicaction , both sharding and replication can be combined
- AVerdadeiro
- BFalso
35Multiple shards become slaves of a single master whereas the master itself is a shard
- AMaster--slave
- BMaster-slave Replication
- CPeer-to-peer
- DPeer-to-peer Replication
36Master-slave replication
- AAlthoug more than one master is possible, a single slave-shard can only be managed by a single master-shard
- BReplicas of shard are kept on multiple slave nodes to provide scalability and fault tolerance for read operations
- CWrite consistency is mainteined by the master-shard
- Dhowever, this means that fault tolerance Ç(with regards to write operations) is affected if the master-shard becomes non-operational or a network outage occurs
- EEach shar gets replicated to multiple peers, and each peer is only responsible for subset of data rather than the complete dataset
37Each shard gets replicated to multiple peers, and each peer is only responsible for a subset of data rather than the complete dataset
- Amaster-slave / replication
- Bpeer-to-peer replication
- CReplication
38Peer-to-peer replication
- AColelctivelly, this helps to achieve increased sacalability and fault tolerance
- BHowever, this means that fault tolerance (with regards to write operations) is affected if the master-shard becomes non-operational or a network outage occurs
- CAs there is no master involved, there is no single point of failure with regards to both read and write operations
39Consistency, availability and partition tolerance theorem states that a distributed system, particularly a database, running on a cluster can only provide two of the following
- ACAP theorem
- BBASE
- CACID
40CAP theorem
- AConsistency
- Bparttition tolerance
- CAvailability
- Dvolumen
- Evelocity
41A read from any node results in the same data across multiple nodes.
- Aconsistency
- Bavailability
- Cpartition tolerance
42A read/write request will always be acknowledged in the form of a succes or a failure
- Aconsistency
- Bavailability
- Cpartition tolerance
43The database system can tolerate communication outages wich split the cluster into multiple silos and can still service read/write requests.
- Aconsistency
- Bpartition tolerance
- Cavailability
44CAP theorem
- AIf consistency (C) and availability (A) are required, available nodes need to communicate to ensure consistency (C). Therefore, partitio tolerance (P) is not possible.
- BIf consistency (C) and partition tolerance (P) are required, nodes cannot remain available (A) as the nodes will become unavailable while achieving consistency (C) wich cannot happen while supporting partition tolerance (P)
- CIf availability (A) and partition tolerance (P) are required then consistency (C) is not possible because of the data communication requirement between the nodes. So, the database can remain available (A) but with inconsistent results
45CAP Theorem
- AIn a distribute database, scalability and fault tolerance can be improved through additional nodes, although this challenges consistency (C), which can also cause availability (A) to suffer due to the latency caused by increased communication between nodes.
- BDistributed database systems cannot 100% partition tolerant (P). Although communication outages are rare and temporary, partition tolerance (P) must always be sopported, Thus, CAP is more about being either CP or AP
- COn the other hand, RDBMSs are CA as they are generally available (A) while being consistent (C) at the same time. RDBMSs generally run on a single node, thus partition tolerance (P) is not a large consideration
46This theorem is also know as Brewer´s theorem after the name of its proposer
- ABASE
- BCAP
- CACID
47ACID
- AAvailability
- BAtomicity
- CConsistency
- DIsolation
- EDurability
48All operations will always succed for fail completely. In other words there are no partial transactions
- AIsolation
- BAtomicity
- CConsistency
- DDurability
49The database will only allow valid data and the database will always be in a consistent state after an operation. Any write followed by an immediate read is guaranteed to be consistent across multiple clients
- AAtomicity
- BConsistency
- CIsolation
- DDurability
50The result of a transaction are not visible to other operations until it is complete
- AAtomicity
- BConsistency
- CIsolation
- DDurability
51The results of an operation are permanent, that is once a transaction has been commited it cannot be rolled back. This is irrespective of any system failure
- AAtomicity
- BConsistency
- CIsolation
- DDurability
52is a database desing principle based on the CAP theorem and followed by database systems that leverage distributed technology.
- ABASE
- BACID
- CCAP
53BASE
- ABasically available
- BConsistency
- CSoft state
- DEventual consistency
54database will always acknowledge the client´s request either in the form of requested data or a success/failure notification.
- ASoft state
- BBasically available
- CEventual consistency
55The database may not be in a consistent state when data is read, thus the results may change if the same data is requested again. This is because the data could be updated for consistency, even though no user has written to the data between the two reads. This property is closely related to eventual consistency
- ABasically available
- BSoft state
- CEventual consistency
56Reads by different clients, right after a write, may not return consistent reults. The database only attains consistency once the changes have been propagated to all nodes. So while the database is in the process of attainingthe state of eventual consistency, it will be in a soft state.
- Abasically avalaible
- Beventual consistency
- CSoft state
57__________ emphasizes availability over immediate consistency, in contrast to _________ wich ensures immediate consistency at the expense of availability due to record locking
- ABASE, ACID
- BACID, BASE
- CCAP, ACID
58Base: This soft approach towards consistency allows BASE compliant datatbases to serve multiple clients without any latency albeit serving inconsistent results
- AEventual consistency
- Bsoft state
- Cbasically available
59BASE However, BASE compliant databases are generally not used for transactional systems where lack of consistency can become a concern
- AVerdadeiro
- BFalso
60Data storage device : Big Data consists of datasets that cannot be stored using traditional storage solutions, mainly because of the large volume of data they contain.
- AVerdadeiro
- BFalso
61Big data storage device : Velocity is a factor that makes traditional database solutions inappropiate for Big Data storage, mainly because of their centralized desing which offers little or no scalability
- AVerdadeiro
- BFalso
62Big Data storage device: The variety characteristic of Big Data datasets requires special attention as it is estimated that 80% of Big Data datasets are unstructured data, and traditional storage solutions do not support storing semi-structured and unstructured data in a scalable and efficient manner
- AVerdadeiro
- BFalso
63The following storage device characteristics need to be considered:
- Ascalability
- Bsoft consistency
- CRedundancy & availability
- DFast access
- ELong-term storage
- FSchema-less storage
- GInexpensive Storage
64Scalability
- Abig data datasets come in huge volumes at a fast pace and are generally acquired from multiple sources. This results in large amounts of data within a short period of time
- BWith the potential of findidng new insights about the way businesses work, enterprises are retaining more and more data , both generated inside the enterprise and obtained from outside sources, as part of their data acquisition activities.
- CTraditionally, historic data that is no longer required is generally archived in offline storage. However, this makes the historic data unavailable for instant analysis
65Scalability
- AIt may not be possible to re-acquire data due to expensive acquisition costs, for example when a dataset is purchased from a data provider.
- BTraditionally, historic data thata is no longer required is generally archived in offline storage. However, this makes the historic data unavailable for instant analysis
- Cit may not be possible to re-generate data if the events that generated the data initially were one-off events, for example a smart meter reading for a certain point in time
- DIn all cases, we need a scalable storage solutions with enough capacity for current and future data capture requirements
66scalability
- AApart from storing the raw data, additional storage is required to store the data created as a result of the data wrangling activity
- BA storage device needs to make efficient use of the underlying disk resources to minimize storage waste.
- CThe data storage requirement, inthe case of joining multiple datasets, will increase in size due to the need to keeo both the original datasets and the joined dataset
- DStorage is further required in order to persist the analytic results from a data analysis activity
- Ein order to address the increasing demands for data storage, a storage device can either scale up or scale out
67Scalability: scale up (vertical scaling)
- Ais a strategy for increasing resource capacity by replacing an existing low capacity device with a higher capacitty device.
- Bdoes not cause disruption and system downtime
- CFor example to double the capacity of a storage device, data from a 500 GB disk can be copied over to 1 TB disk.
- Dwill cause disruption and system downtime
68Scalability: Scale out (horizontal scaling)
- Ais a strategy for increasing resource capacity by replacing an existing low capacity device with a higher capacitty device.
- Bis a strategy for increasing resource capacity by adding similar or higher capacity commodity devices alongside the existing device
- CFor example, to double the storage capacity, a 500GB disk can be added alongside an existing 500GB disk
- DDoes not cause disruption and system downtime
69Considering the differences between scalingup and scaling out, a storage device should be able to scale out as this is a lower cost strategy that enables increasing storage capacity without incurring system downtime and interruption.
Sharding can help create scalable storage as a shards can be stored on the nodes added via scaling out.- AVerdadeiro
- BFalso
70Redundancy & availability
- ABig Data datasets, both in raw and processed form, are a business asset and require attention with regards to storage. Multiple business functions may need to glean value out of these datasets, cometimes simultaneously
- BBig data datasets come in huge volumes at a fast pace and are generally acquired from multiple sources. This results in large amounts of data within a short period of time.
- CThis dependence on Big Data datasets across an enterprise requires a reliable storage device that is fault-tolerant and is highly available
- DA big data solution enviroment is generally composed of cluster that are built using commodity servers connected via a high bandwidth network. With more nodes and network connections, the chances of a node becoming unavailable increases either due to hardware breakdown such as disk failure, or due to network failure.
71Redundacy & availability
- AAs a result, storage redundancy is required to ensure auninterrupted acces to data in the event of a storage device failure, thereby providing high availability and fault tolerance
- BSharding can help create scalable storage as shards can be stored on the nodes added via scaling out
- Cto provide such redundancy, a storage device implements automatic sharding and replication with a configuration of either sharding and master-slave replication or sharding and peer-to-peer replication
72Big Data analytics generally involve both
- Arealtime processing
- Bbatch processing
- Cin_memory
- Din disk
73_____________requires fast read/write capabilities that are usually implemented via in-memory storage solutions
- ARealtime data analysis
- Bbatch processing
74Fast access
- ABig Data analytics generally involve both realtime as well as batch processing
- BRealtime data analysis requires fast read/write capabilities that are usually implemented via in-memory storage solutions.
- CBatch processing requires stream-based data access with high throughput, implemented via traditional disk-based storage devices or newer solid state drives that provide better performance at the expense of higher cost
- DThe basic tenet of Big Data is to extract value from large amounts of data
- EAt the same time, data arriving at high velocity needs a storage device that supports fast writes with minimal overhead. for example, schema validation at read time instead of write time
75Long-term Storage
- AThe basic tenet of Big Data is to extract value from large amounts of data
- BRealtime data analysis requires fast read/write capabilities that are usually implemented via in-memory storage solutions
- CThis requires enterprises to retain data for longer periods of time (to create larger datasets) so that future data analyses are more insightful due to having more historic data available
- DThis characteristic warrants the need for a storage device with increased storage capacity that is reliable and can be brougth online without incurring too much time delay
76Long -term storage
- ATraditionally, historic data that is no longer required is generally archived in offline storage. however, this makes the historic data unavailable for instant analysis.
- Bbig data analytics require historic data to be available online for discovering hidden patterns leading to valuable insights
- Cdata arriving at high velocity needs storage device that supports fast writes with minimal overhead, for example, schema validation at read time instead of write time
- DAs a result, historic data is kept online by adding more storage capacity via scaling out
77schema-less storage
- AAs a result, historic data is kept online by adding more storage capacity via scaling out
- Bbig data datasets come in multiple formats with limited or no schema, such as semi-structured and unstructured data.
- Cwithout any prior knowledge about the structure of data does not guarantee that the data would conform to the same structure in the future, as in the nature of unstructured data
- DThis requires the storage device to support __________ data persistence along with the added ability to make schema changes on the fly without breaking existing applications or incurring downtime
78inexpensive storage
- AWith all the characteristics required to store voluminous data, provide replication, and support long-term storage, the cost of storage devices can become a concern.
- Bwithout any prior knowledge about the structure of the data does not guarantee that the data would conform to the same structure in the future, as is the nature of unstructured data
- CA storage device needs to make efficient use of the underlying disk resources to minimize storage waste.
- DUse of proprietary storage devices generally requires replacement of existing nodes with more expensive ones in order to scale up, eventually hitting a limit
- EA storage device needs to make use of commodity hardaware that can scale out so that costs can be kept down as enterprises amass more and more data
79On-Disk storage device
- Adistributed file system
- Bdatabase
- Cfile system
- DNoSQL
80On-disk storage device: Distributed file system
- AA storage device that is implemented with a __________ provides simple, fast access data storage that is capable of storing large datasets that are non-relational in nature, such as semi-structured and unstructured data
- BTo handle large volumes of data arriving at a fast pace, relational databases generally need to scale.
- CAlthough based on straightforward file locking mechanisms for concurrency control, it provides fast read/write capability, which addresses the velocity characteristic of Big Data.
- Dare good fit when data must be accessed in streaming mode with no random reads and writes.
81On-disk storage device: Distributed File System
- Ais not ideal for datasets comprising a large number of small files as this creates excessive disk-seek activity, slowing down the overall data access
- BIs also more overhead involved in processing multiple smaller files, as dedicated processes are generally spawned by the processing engine at runtime for processing each file before the results are synchronized from across the cluster
- Cgood for handling transactional workloads involving small amounts of data with random read/write
- Dwork best with fewer but larger files accessed in a sequential manner
- EMultiple smaller files are generally combined into a single file to enable optimum storage and processing
- FDo not provide any file searching capability out of box
82On-Disk storage device: Distributed file system
- ACan be employed in clustered enviroment.
- B____ storage device is suitable when large datasets of raw data to be stored or when archiving of datasets is required
- Cprovides an inexpensive storage option for storing large amounts of data over a long period of time that remains online
- DGood for handling transactional workloads involving small amounts of data with random read/write
- EMore disk can simply be added without needing to offload the data to offline data storage, such as tape drives
- Flike any file systems, are agnostic to the data being stored and therefore support schema-less data storage
- Gprovides out of box redundancy and high availability by copying data to multiple locations via replication
83On-dis storage device: Database
- ARelational databases or relational database management systems (RDBMSs)
- BFile systems
- Cnon-relational databases or Not only SQL (NoSQL)
- DNewSQL
84RDBMS relational database management systems
- AAre good for handling transactional workloads involving small amounts of data with random read/write
- BSchema-less data model-Data can exist in its raw form
- CAre ACID, and in order to honor this compliance, they are generally restricted to a single node
- DThe redundancy and fault tolerance provided by sharding and replication in a clustered enviroment are not inherently supported.
85RDBMS relational database management systems
- ATo handle large volumes of data arriving at a fast pace relational databases generally need to scale
- BEmploy vertival scaling, not horizontal scaling, wich is a more costly and disruptive scaling strategy. This makes _______ less than ideal for long term storage of data
- CMore disks can simply be added without needing to offload the data offline data storage, such as tape drives
- DNote that some relational databases, for example IBM DB2 pureScale, Sybase ASE Cluster Edition, Oracle Real Application Clusters (RAC) and Microsoft Parallel Data warehouse PDW, are capable of being run on clusters. however, these database clusters still use shared storage that can act as a single point of failure
86RDBMS
- ANeed to be manually sharded, mostly using application logic.
- Bthis means that the client (applicaction logic) needs to know which shard to query in order to get the required data
- CReads across multiple nodes may not be consistent inmediately after a write. however, all nodes will eventually be in a consistent state
- DThis further complicates the data processing when data from multiple shards is required
87RDBMS
- Agenerally require data to adhere to a schema. As a result, storage of semi/unstructured data whose schema is not know or keeps changing is not supported
- BRefers to tecnologies used to develop net generation non_relational databases that are highlly scalable and fault-tolerant
- CThe schema conformance is validated at the time of data insert/update,wich introduces overhead and leads to latency
- DA less ideal choice for storing high velocity data that needs a highly available database storage device with fast data write capability
- EIs not useful as a storage device in a Big data solution enviroment
88On-disk storage device: Database________________Refers to tecnologies used to develop net generation non_relational databases that are highlly scalable and fault-tolerant
- ARDBMS
- BNoSQL
- CNewSQL
89On-disk storage device: NoSQL Characteristics
- ASchema-less data model: Data can exist in its raw form
- Bscale out rather than scale up: More nodes required rather than replacing the existing one with better higher performance node
- CHighly available: Built on cluster-based technologies providing fault tolerance out of box
- DLower operational costs: Built on open source platforms with no licensing costs, and can be deployed on commodity hardware.
- EEventual consistency; Reads across multiple nodes may not be consistent immediately after a write. However, all nodes will eventually be in a consistent state
- FBASE, not ACID: BASE compliance requires a database to maintain high availability in the event of network/node failure, while not requiring the database to be in a consistent state whenever an update occurs. The database can be in a soft/inconsistent state until it eventually attains consistency.
90On-Disk storage device: NoSQL Characteristics
- AAPI driven data access - Data access is generally supported via API based queries, including RESTful APIs. Whereas some
implementations may also provide SQL-like query capability - BAuto Sharding and replication -To support horizontal scaling and provide high availability, a NoSQL storage device automatically employs sharding and replication techniques where the dataset is partitioned horizontally and then copied over to multiple nodes
- CIntegrated caching - Removes the need for a third-party distributed caching layer, such as Memcached
- DDistributed query support - NoSQL storage devices maintain consistent query behavior across multiple shards
- EPolyglot persistence - The use of NoSQL device storage does not mandate retirin traditional RDBMSs. In fact, both can be used at the same time, thereby supporting polyglot persistence (an approach of persisting data using different types of storage technologies). this is good for developing systems requiring structured as well as semi/unstructured data.
- FAggregate-focussed - Unlike relational databases that are most effective with fully normalized data, NoSQL storage devices store de-normalized data aggregated data (an entity containing merged, often nested, data for an object) Thereby eliminating the need for joins and mapping between application objects and the data stored in the database. Note that graph database storage devices are not aggregate-focused
- AAPI driven data access - Data access is generally supported via API based queries, including RESTful APIs. Whereas some
91On Disk storage device: NoSQL Characteristics Volume:
- AThe storage requirement of ever increasing data volumes commands the use of databases that are highly scalable while keeping costs down for the business to remain competitive.
- BThe fast influx of data requires databases with fast access data write capability
- CNoSQL storage devices fulfill this requirement by providing scaling out capability while using inexpensive commodity servers
- DFurthermore, there may not be licensing costs involved as NoSQL databases generally follow the Open Source development model
92On Disk storage device: NoSQL Characteristics Velocity:
- ANoSQL storage devices fulfill this requirement by providing scaling out capability while using inexpensive commodity servers
- BThe fast influx of data requires databases with fast access data write capability
- CNoSQL storage devices enable fast writes by using schema-on-read rather than schema-on-write principle
- DBeing highly available, NoSQL storage devices can ensure that write latency does not occur because of node/network failure
93On- Disk Storage Device NoSQL Types Variety :
- AA storage device needs to handle different types of data formats including documents, emails, images and videos and incomplete data
- BNoSQL storage devices enable fast writes by using schema-on-read rather than schema-on-write principle
- CNoSQL storage devices can store these different forms of semi-structured and unstructured data formats
- DAt the same time, NoSQL storage devices are able to store schema-less data and incomplete data with the added ability of making schema changes as the data model of the datasets evolve. In other words, NoSQL databases support schema evolution
94On-Disk storage device: NoSQL types
- AKey-value
- BRDBMS
- Ccolumn-family
- DDocument
- EGraph
95NoSQL: Key-value
- AAct like hash tables
- BIs a list of values each value is identified by a key
- Cindexes that speed up searches are generally supported
- DThe value is opaque to the database and is essentially stored as a BLOB
- EThe value stored can be aggreted, ranging from sensor data to videos
- FValue look-up can only be performed via the keys as the database is oblivious to the details of the stored aggregate
96NoSQL: Key-value
- APartial updates are not possible. An update is either a delete or an insert operation
- BA select operation can retrieve a part of the aggregate value
- C_______ storage devices generally do not maintain any indexes, therefore writes are quite fast
- D______ Based on a simple storage model, storage devices are highly scalable
- EThe key is usually appended with the type of the value being saved for easy retrieval. for example 123_sensor1
97No-SQL: key-Value
- Asome implementations support compressing values for reducing the storage footprint. However, this introduces latency at read time, as the data needs to be decompressed firts
- Bmost key-values storage devices provide collections or bucket(like tables) into wich key-value pairs can be organized
- Ccan be encoded using either a text-based encoding scheme, such as XML, or JSON or BSON (binary)
98NOSQL: Key-Value
- AUnstructure data storage is required
- Bhigh performance read/writes are required
- Cthe value is fully identifiable via the key alone
- Dsearches need to be performed on different fields of the document
- Evalue is a standalone entity that is not dependent on other values
- Fvalues generally have a comparatively simple structure or are binary
- Gquery patterns are simple. involving insert, select and delete operations only
- Hstored values are manipulated at the application layer
99NoSQL:Key-value is inappropiate
- Aapplications require searching or filtering data using attributes of the store value
- Bthe value is fully identifiable via the key alone
- Crelationship exist between different key-value entries
- Da group of keys values need to be updated in a single transaction
- Emultiple keys require manipulation in a single operation
- Fschema consistency across different values is required
- Gupdate to individual attributes of the value is required
- Hexamples: Riak,Redis and amazon Dynamo DB
100NoSQL: Document
- AThe table is a list of values where each value is identified by a key
- BAlso store data as key-value pairs. However, unlike key-value storage devices, the stored value is a document that can have a complex nested structure, such as an invoice
- CCan be encoded using either a text-based encoding scheme, such as XML or JSON, or using a binary encoding scheme, such as BSON (binary JSON)
- DLike key-value storage devices, most document storage devices provide collections or buckets (like tables) into which key-value pairs can be organized
101NoSQL: Document differences compared with key-value
- Adocument storage devices are value-aware
- BThe stored value is self-describing: the schema can be inferred from the structured of the value
- CA select operation can reference a field inside the aggregate value
- DA select operation can retrieve a part of the aggregate value
- Epartial update are supported; a subset of the aggregate can be updated
- Findexes that speed up searches are generally supported
102NoSQL: Document
- AEach document canhave a different schema
- BThe value is opaque to the database and is essentially stored as a BLOB
- CIs possible to store different types of documents or documents of the same type that have fewer or more fields with respect to each other
- DAdditional fields can be added to a document after the initial insert, thereby manifesting flexible schema support
- EIt should be noted that document storage devices are not limited to storing data that occurs in the form of actual documents, such as an XML file, but can also be used to store any aggregate that consists of a collection of fields having a flat or a nested schema
103NoSQL:Document is appropiate when:
- AStoring semi-structured document-oriented data comprising flat or nested schema
- Bschema evolution is a requirement as the structure of the document is either unknown or is likely to change
- Capplications require a partial update of the aggregate stored as a document
- Dsearches need to be performed on different fields of document
- Eunstructured data store is required
- Fstoring domain objects, such as customers, in object form
- Gquery patterns involve insert, select, update, and delete operations
104NoSQL:Document is inappropiate when
- AMultiple documents need to be updated as part of a single transaction
- Bstoring domain objects, such as customers, in object form
- CPerforming operations that need joins between multiple documents or storing data that is normalized
- Dschema enforcement for achieving consistent query design is required as the document structure may change between successive query runs, wich will require retructuring the query
- Estored value is not self-describing
- Fbinary data needs to be stored
105NoSQL:Document examples
- ARiak, Redis and Amazon Dynamo DB
- BMongoDB, CouchDB and terrastore
106NoSQL: Column-Family
- AColumn-family storage devices store data much like a traditional RDBMS but group related columns together in a row, resulting in column-families
- BThe table is a list of values where each value is identified by a key
- CEach column can be a collection of related columns itself, referred to as a super-column
- DEach super-column can contain an arbitrary number of related columns that are generally retrieved or updates as a single unit
- EEach row consists of multiple column-families
- FEach row can have a different set of columns, thereby manifesting flexible schema support
- GEach row is identified by a row key
107NoSQL: Column-Family
- Aprovide fast data acces with random read/write capability
- BStore different column in separate physical file, wich greatly helps in speeding up queries as only the required column-families are searched
- CPartial update are not possible. An update is either a delete or insert operation
- Dprovide support for selectively compressing
- ELeaving searchable ______ uncompressed can make queries faster because the target column does not need to be decompressed for lookup
- FMost implementations support data versioning after wich the configured columns are automatically removed
108NoSQL: Column-family is appropriate when
- ARealtime random read/write capability is needed and data being stored has some defined structure
- BData represents a tabular structure, each row consists of a large number of columns and nested groups of interrelated data exist
- CSupport for schema evolution is required as column families can be added or removed without any system downtime
- Dstoring domain objects, such as customers, in object form
- ECertain fields are mostly accessed togheter, and searches need to be performed using field values
- Fefficient use of storage is required when the data consists of sparsely populated rows (no column, no space)
- Gquery patterns involve insert, select, update, and delete operations
109NoSQL: Column-Family is inappropriate
- AMultiple documents need to be updated as apart of a single transaction
- Brelational data access is required, for example joins
- CACID transactional support is required
- Dbinary data needs to be stored
- ESQL-compliant queries need to be ecxecuted
- Fquery patterns are likely to change frequently that initiate a corresponding restructuring of how column-families are arranged, for example, during proof of concept development
110Cassandra, HBase and Amazon SimpleDB
- AMongoDB, CouchDB and terrastore
- BNoSQL:Column-Family
- CRiak, Redis and Amazon Dynamo DB
111NoSQL: Graph
- APersist inter-connected entities
- BThe value is opaque to the database and is essentially stored as a BLOB
- CUnlike other NoSQL storage devices, where the emphasis is on the structure of the entities, ____________storage devices place emphasis on storing the linkages between entities
- DEntities are stored as nodes, also called vertices, while the linkages between entities are stored as edges. In RDBMS parlance, each node can be thought of a single row while the edge denotes a join
- ENodes can have more than one type of link between them through multiple edges.
- FEach node can have attribute data as key-value pairs, such as a customer node with ID, name and age attributes
112NoSQL: Graph
- AEach edge can have its own attribute data as key-value pairs, wich can be used to further filter query results
- BMultiple edges asre similar to defining interconnected nodes based on node attributes and/or edge attributes, commonly referred to as node traversal
- CQueries generally involve finding interconnected nodes based on node attributes and/or edge attributes, commonly referred to as node traversal
- DCan be a collection of related columns itself referred to as a super-column
- EEdges can be unidirectional or bidirectional, setting the node traversal direction
- FGenerally, graph storage devices provide consistency via ACID compliance
113NoSQL:Graph
- AThe value stored can be any aggregate, ranging from sensor data to videos
- BThe degree of usefulness of a graph storage device depends on the number and types of edges dened betwwen the nodes. The higher the number and more diverse the edges are the more diverse types of queries it can handle
- CAs a result, it is important to comprehensively capture the types of relations tha exist betqeen the nodes
- DGenerally allow adding new types of nodes without making changes to the database, it also enables defining additional links between nodes as new types of relationships or nodes apperar.
114NoSQL: Graph is appropriate when
- Ainterconecccected entities need to be stored
- Bquerying entities based on the type of relationship with each other rather than the attributes of the entities
- Cfinding groups of interconnected entities
- Dfinding distances between entities in terms of the node traversal distance
- Emining data with a view toward finding patterns
- Fstoring semi-structured document-oriented data comprising flat or nested schema
115NoSQL: Graph is inappropriate when
- Aupdates are required to a large number of node attributes or edge attributes, as this involves searching for nodes or edeges wich is a costly operation compared to performing node traversals
- Bbinary storage is required or else queries based on selection of node/edge attributes dominate node traversal queries
- CEntities have a large number of attributes or nested data- is the best store ligtweight entities in a graph storage dev ice while storing the rest of the attribute data in a separate non-graph NoSQL storage device
- DACID transactional support is required
116NoSQL: Graph
- ANeo4j, infinite Graph and orientDB
- BRiak, redis and Amazon Dynamo DB
- Cinclude Cassandra, HBase and Amazon SimpleDB
117NewSQL
- ANoSQL storage devices are highly scalable, available fault-tolerant, and very fast for read/writeoperations. However, they do not provide the same transaction and consistency support as exhibited by ACID compliant RDBMS
- BFollowing the BASE model, NoSQL storage devices provide eventual consistency rather than immediate, and could therefore be in an inconsistent stage while reaching the state of consistency. As a result, they cannot be used for implementimg large scale transactional systems
- C____ storage devices combine the ACID properties of RDBMS with the scalability and fault tolerance offered by NoSQL storage devices
- DEach super-column can be a collection of related columns itself referred to as a super-column
- E______ databases generally support SQL compliant syntax for data definition and data manipulation operations, and they often use a relational data model for data storage
- F_____________ can be used for developing OLTP systems with veri high volumes of transactions, for example a bank. They can also be used for realtime analytics, for example operational analytics, as some implementations are memory based
- GAs compared to a NoSQL storage device, _________ storage device provides an easier transition from a traditional RDBMS to a highle scalable database due to its support for SQL
118NewSQL examples
- ANeo4J, infinite graph and orientDB
- BRiak, Redis and Amazon Dynamo DB
- CVoltDb, FoundationDB, NuoDB and innoDB
119Distributed/Parallel Data Processing
- Abig Data datasets are generally stored utilizing distributed technologies (distributed file system or NoSQL database)
- BThe very nature of a distributed storage device requires a processing engine that can process data without needing to transfer the data from storage to a computing resource as with distributed data processing
- CSchemas may change over time in a effort to accomodate changing bussines requirements, or simply because of an application software upgrade
- DIn support of maximizing the value characteristics of Big Data, it is imperative to employ a processing model based on the divide-and-conquer principle, as with parallel data processing
120Schema-less Data Processing
- Aoften, it may be feasible to process data offline in batches, such as with overnight report generation
- Bbig Data datasets come in multiple formats (variety characteristic) and may not conform to any schema, especially unstructured data
- C____________may change over time in an effort to accomodate changing business requirements, or simply because of an application software upgrade.
- DThe lack of adherence to any particular data model requires flexible processing of Big Data datasets so that they can be processed in raw form without the need to be stored in a particular data model
121Multi-workload support
- ABig data datasets may arrive thick (volume characteristic) and/or fast (velocity characteristic)
- BOften, it may feasible to process data offline in batches, such as with overnight report generation
- CIn other cases, the results may be required in in realtime, such as with GPS signal processing workloads (transactional and batch)
- DIn order to achieve maximum value from Big Data datasets the underlying processing platform may need to support both transactional and batch workloads
- EA single processing engine may fulfill this requirement, or multiple processing engines may need to be used
- FSimilarly, more data sources with unknown schemas may need to be added in the future
122multi-workload support
- AAlthough having the ability to process data both in realtime (transactional) and batch mode is ideal for extracting maximum value out of Big Data datasets, support for both modes may not be required or may not be feasible
- Bthe processing of Big data datasets requires a highly scalable processing engine that can be linearly scaled
- CGenerally, assembling a batch processing Big Data solution enviroment is simpler and cheaper when compared to a realtime processing Big Data solution
- DConsequently, adding support for multi-workload processing should be business driven involving careful cost-benefit analysis
123Linear Scalability
- AOwing to the volume and velocity characteristics of Big Data, the processin demand can grow quite sharply with increasing volumes of data arriving at a fast pace
- Bsupporting a distributed processing enviroment with parallel processing capabilities requires a processing engine that can provide a steady throughput as the data volumes grow
- CThe processing of Big Data datasets requires a highly scalable processing engine that can be linearly scaled
- DBig Data datasets may arrive thick (volume characteristics) and/or fast (velocity characteristics)
- EIn the context of processing, linear scalability means that one receives a proportional increase in performance with the addition of more processing nodes
- FRealtime business intelligence and analytics can leverage such a linearly scalable processing enviroment to deliver quicker responses involving complex operations on an entire dataset
- G_____________ is generally achieved by employing a scaling out strategy as it provides a simple, non-disruptive, and cost effective method for increasing processing capacity
124Redundancy & fault tolerance
- AA highly distributed data processing enviroment with parallel data processing capabilities generally involves complicated architecture
- BWith a horizontally scaled processing architecture, wich by desing involves a large number of nodes and networking components, the chances of partial system failure increase
- CSimilarly, more data sources with unknown schemas may need to be added in the future
- DAs a system failure in the middle of a long running distributed task would be detrimental to achieving the analytic goals, aprocessing engine needs to provide fault tolerance so that a partial failure does not render the entire system unavailable and data processing does nort need to started from scratch
- EGenerally, fault tolerance is provided through redundancy
- FWith redundant processing resources, the system can still be available in the event of a partial system failure
- GHorizontal scaling lends itself quite useful in this case as redundancy can generally be increased by simply adding more processing resources
125Low Cost
- AAt the start of a Big Data initiative, the cost for a highly scalable distributed data processing enviroment involving a few processing nodes and networking equipment may not be that high.
- BOver time, as the data volume grows and the types and frequency of analytics being run increase, the requirement for an increased number of processing resources can translate into soaring IT costs involving both software and hardware
- CWith redundant processing resources, the system can still be available in the event of partial system failure.
- DThis could prove counterproductive to the very reason the Big Data initiative was undertaken (often to help the business deliver increased value, drive down costs, find new sources of reveneu, or establish new service offerings)
- EUse a open source software deployed over commodity hardware helps to keep the costs down
- FAnother aspects of keeping costs down is the ability of the processing engine to take advantage of cloud
- GThe on-demand and elastic nature of the cloud helps avoid any up-front capital investment, coupled with faster setup of the data processing solution enviroment
126Big data processing
- A____________ large amounts of data is not a new phenomenon, and different large scale data processing architectures exist
- B______________________ requires a distributed enviroment that is capable of processing data in parallel a characteristic supported by the cluster architecture
- Care highly scalable, supportin g horizontal sclaing with linear performance gains
127Big Data Processing: Cluster
- Ais a group of nodes connected together via a network to process tasks in parallel
- B__________ is a centrally managed network of nodes (computers) where each node is responsible for a sub-task of a large problem
- Cenable distributed data processing
- D_____comprises low-cost commodity nodes that collectively provide increased processing capacity with inherent redundancy and fault tolerance, as it consists of physically separate nodes
- EThe majority of Big Data processing occurs
- Fare higly scalable, supporting horizontal scaling with linear performance gains
- GProvide an ideal deployment enviroment for a processing engine as large datasets can be divided into smaller datasets and then processed in parallel in a distributed manner
- HCan be utilized both by a realtime processing engine and a batch proceessing engine, such as Spark and MapReduce respectively
128Big Data Processing: Batch Mode
- Adata is processed offline in batches where the response time could vary from minutes to hours
- BData is first persisted to the disk and only then processed
- CStrategic BI predictive/prescriptive analytics and ETL operations are generally _____________________
- Ddata is processed in-memory as it is captured before being persisted to the disk
- E______________involves processing a range of large datasets, either on their own or joined together, essentially addressing the volume and variety characteristics of big Data datasets
- Fthe majority of big data processing occurs in ______________________________
- G______________ is relatively simple, easy to set up, and low cost in comparison to realtime mode
129Big Data Processing: realtime mode
- A_____________________ data is processed in -memory as it is captured before being persisted to the disk
- BResponse time generally ranges from a few seconds to under a minute
- Cstrategic BI, predeictive/prescriptive analytics and ETL operations are generally batch-oriented
- D______________ adresses the velocity characteristic of Big Data dataset
- E_____________________ is alsocalled event or stream processing as the data either arrives continuously (stream) or at intervals (event)
- FThe individual event/stream datum is generally small in size, but its continuous nature results in very large datasets
- GAnother related term, interactive mode, falls within the category of realtime, interactive mode generally
- HOperational BI/analytics are generally
130Map reduce processing engine
- Ais a widely used implementation of the batch processing engine mechanism
- BIt is a highly scalable and reliable processing engine based on the principle of divide-and-conquer
- CIt provides built-in fault tolerance and redundancy
- Dit divides a bigger problem into a set of smaller problems that are easier and quicker to solve
- Eit has its roots both in distributed computing as well as in parallel computing
- FOperational BI/analytics are generally conducted in realtime mode
- G_____________ is a batch -oriented processing engine used to process large datasets using parallel processing deployed over clusters of commodity hardware
131Map Reduce
- Awith redundant processing resources, the system can still be available in the event of a partial system failure
- B_____________ does not require the input data to conform to any particular data model. Therefore, it can be used to process schema-less datasets
- CA dataset is bloken down into multipl smaller parts and operations are performec on each part independently and in parallel
- DThe results from all operations arre then summarized to arrive at the answer
- E____________ processing engine generally supports batch workloads only
132MapReduce
- A_______, the data processing algorithm is instead moved to the nodes that store the data
- BThe data processing algorithm executes in parallel on these nodes, thereby eliminating the need to first move the data
- CThis not only saves network bandwidth but also results in a large reduction in processing time for large datasets, as processing smaller chunks of data in parallel is much faster
- DReads across multiple nodes may not be consistent immediately after a write. However, all nodes will eventually be in a consistent state
133Map Reduce concepts
- Alow cost
- Bmap : map task
- Ccombine (optional) : map task
- Dpartition : map task
- Eshuffle & sort : Reduce task
- Freduce : Reduce task
- Glinear scalability
134MapReduce: Map
- AThe firts stage of map reduce, during wich the dataset file is divided into multiple smaller splts
- BEach split is parsed into its constituent records as a key-value pair
- CThe processing of Big Data datasets requires a highly scalable processing engine that can be linearly scaled
- DThe key is usuallu the ordinal position of the record and the value is the actual record. For exampl (234, sky is blue)
- EThe parsed key-value pairs for each split are then sent to a map function (mapper), with one mapper function per split. The map function executes user-defined logic
135MapReduce: Map
- AEach split generally contains multiple key-value pairs and the mapper is run once for each key-value pair in the split
- BThe mapper processes each key-value pairs as per the user-defined logic and further generates as a key-value pair as its output
- CThe output key can either be the same as theinput key or a substring value from the input value, or another serializable user-defined object
- Dsimilarly, the output value can either be the same as the input value or a substring value from the input value, or another serializable user-defined object
- Egenerally, the output of the map functions is handled directly by the reduce function.However, map task and reduce tasks are mostlyrun over different nodes
136MapReduce: Combine
- AGenerally, the output of map function is handled directly by the reduce function. However, map tasks ans reduce tasks are mostly run over different nodes
- BRequires moving data between mappers and reducers that can consume a lot of valuable bandwidth, and directly contributes to processing latency
- CWith larger datasets, the time token to move the data between map and reduce stages can exceed the actual processing undertaken by the map and reduce tasks
- DFor this reason the mapReduce engine provides an optional______________________ function that sumarizes a mapper´s output before it gets processed by the reducer
- EThe first stage of MapReduce is know as _________, during which the dataset file is divided multiple smaller splits
137MapReduce: Combine
- Aprocesses each key-value pair as per the user-defined logic and further generates a key-value pair as its output
- BA _________________ is essentially a reducer function that groups a mapper's output locally, on the same node as the mapper
- CA reducer function can be used as combiner function or a custom user-defined function can be used
- DThe mapReduce engine combines all values for a given key from the mapper output, creating multiple key-value pairs as input to the combiner where the key is not repeated and the value exists as a list of all corresponding values for that key
- EThe ___________ stage is only an optimization stage, therefore it may not even be called by the mapReduce engine.
138MapReduce: Partition
- ADuring this stage, if more than one reducer is involved, a partitioner divides the output from the mapper or combiner, if specified and called by the MapReducen engine, into partitions between reducer instances
- BThe number of partitions equals the number of reduces
- CIs only an optimization stage, therefore it may not even be called by the mapReduce engine
- DAlthough each partition contains multiple key-value pairs all records for a particular key are within the same partition
- EThe MapReduce engine guarantees a random and fair distribution between reducers while making sure that all of the same keys across multiple mappers end up with the same reducer instance
- Fdepending on the nature of the job, certain reducers can sometimes receive a large number of key-value pairs compared to others. As a result of this uneven workload, some reducers will finish earlier than others
- GOverall, this is less efficient and leads to longer job execution times than if the work was evenly split across reducers
- Hthis can be rectified by customizing the partitioning logic in order to guarantee a fair distribution of key-value pairs
- IThis is the last stage of the map task
139MapReduce: Shuffle & sort
- ADuring the first stage of the reduce task, output from all partitioner is copied across the network to the nodes running the reduce task. This is known as _____________
- BIs the final stage of the reduce task
- CThe list based key_value output from each partitioner can contains the same key multiple times
- DNext, The MapReducer engine automatically groups and sorts the key-value pairs according to the keys so that the output contains a sorted list of all input keys (and their values) with the same keys appearing together
- ETo view the full output from the mapReduce job, all the file parts must be combined
- FThe way keys are grouped and sorted can be customized
- GThe MapReduce engine then merges each group of keys together before the shuffle and sort output is processed by the reducer function
- HThis mere creates a single key-value pair per group, where key is the group key and the value is the list of all group values
140MapReduce: Reduce
- AThe way kwys are grouped and sorted can be customized
- BIs the final stage of the reduce task
- CDepending on the user-defined logic specified in the reduce function (reducer), the reducer will either further summarize its input or will emit the output without making any changes
- DThus, for each key-value pair that a reducer receives, the list of values stored in the value part of the pair is processed and another key-value pair is written out
- EThe output key can either be the same as the input key or a substring value from the input value, or another serializable user-defined object
141MapReduce:Reduce
- AThe output value can either be the same as the input value or a substring value from the input from the input value or another serializable user-defined object
- B__________________Just like the mapper, for the input key-value pair, a reducer may not produce any output key-value pair (filtering) or can generate multiple key-value pairs (demultiplexing)
- CIs essentially a reducer function that groups a mappers output locally, on the same node as the mapper
- DThe output of the reducer, that is tyhe key-value pairs, is then written out as a separate file one file per reducer
- ETo view the full output from the MapReduce job, all the file parts must be combined
- Fthe number of reducers can be customized
142Mapreduce: Reduce It is also possible to have a MapReduce job without a reducer, for example when performing filtering
- AVerdadeiro
- BFalso
143Somer NoSQL storage devices provide MapReduce support for batch processing out of box
- AVerdadeiro
- BFalso
144Making use of clustered deploymenr of NoSQL storage devices, the MapReduce processing engine distributes the query processing across multiple nodes.
This is generally achieved via the provision of map and reduce constructs that the user then needs to implement via respective API of the NoSQL storage devices- AVerdadeiro
- BFalso
145MapReduce works on the principle of divide-and-conquer. However, it is important to understand the semantics of this principle in the context of MapReduce. The divide-and-conquer principle can generally be achieved using one of the following approaches: Task parallelism and Data Parallelism
- AVerdadeiro
- BFalso
146(MapReduce) task parallelism
- ADifferent sub-datasets are spread across multiple nodes and are processed using the same algorthm
- BRefers to the parallelization of data processing by dividing a task into sub-tasks and running each sub-task on a separate processor, generally on a separate node in a cluster
- CEach sub-task generally executes a different algorithm, with its own copy of the same data or different data as its input, in parallel
- DGenerally the output from multiple sub-tasks is joined together to obtain the final set of results
147(MapReduce) Data Parallelim
- ARefers to the parallelization of data processing by dividing a dataset into multiple sub-datasets and processing each sub-dataset in parallel
- BEach sub task generally executes a different algorithm wich its own copy of the same data or different data as its inputs, in parallel
- CDifferent sub-datasets are spread across multiple nodes and are pocessed using the same algorithm
- DGenerally de output from each ptrocessed sub-dataset is joined together to obtain the final set of results
148MapReduce Algorithm Design
- AWithin Big Data enviroments, the same task generally needs to be performed repeatdly on a data unit, such as a record, where the complete dataset is distributed across multiple locations due to its large size
- BMapReduce adresses this requirement by employing the data parallelism approach, where the data is divided into splits
- CDifferent sub-datasets are spread across multiple nodes and are processed using the same algorithm
- DEach split is then processed by its own instance of the map function, wich contains the same processing logic as the other map functions
- EThe majority of traditional algorithmic development follows a sequential approach where operations on data are performed ona after the other in such a way that subsequent operations is dependent on its preceding operation
Here, operations are divided among the map and reduce functions - FMap an Reduce task are independent, and in turn, run isolated from each other
- GEach instance of a map or reduce function runs independently of other instances
149MapReduce Algorithm Design
- AThe logic within the reduce function is dependent on the output of the map function, in particular, which keys were emitted from the map function as the reduce function receives a unique key with a consolidated list of all of its values
- BRelatively simplistic algorithmic logic, such that he required result can be obtained bya pplying the same logic to different portions of a dataset in parallel, and then aggregating the results in some manner
- Crequires a highky scalable processing engine that can be linearly scaled
- DAvailability of the dataset in a distributed manner partitioned across a cluster so that multiple map functions can process diferent subset a datasets in parallel
- Eunderstanding of the data structure within the dataset so that a meaningful data unit (a single record) can be chosen
- FDividing algorithmic logic into map and reduce functions so that the logic in the map function is not dependent on the complete dataset, as only data within a single split is available
- GEmitting the correct key from the map function along with all the required data as value because the reduce function´s logic can only process those values that were emitted as part of the key-value pairs from the map function
- HEmiting the correct key from the reduce function along with the required data as value because the output from each reduce function becomes the final output of the mapReduce algorithm
Informações
- Descrição
- Fundamental Big Data Big Data I (carolina c) Quiz sobre Modulo 7: Fundamental Big Data engineering, criado por Carolina Colorado em 27-03-2017.
- Autor
- Carolina Colorado
- Idioma
- Español
- Visualizações
- 86
- Etiquetas
- Avaliação
- 3,0 de 5 · 1 avaliação Avalie este quiz