Flashcards
Big Data
Cards in this deck (23)
- 1What are the 3 shifts that take place towards "big data" world?more,messy, good enough
More- more data is available , is cheaper to gather, but accuracy is lost.
Messy - varies in quality, combining different types of data, or transforming data into
something else and distributed across countless servers and multiple ownerships. accuracy is decreased but we are satisfied with gains in knowledge of direction and trend
GOOD ENOUGH - we look for patterns and correlations, rather
than causality. eg.nowing when is the best time to buy a plane ticket is good enough, without the need to
know the cause and reason - 2What is big data?Big Data is a term that describes large volumes of high velocity, complex and variable data that
require advanced techniques and technologies to enable the capture, storage, distribution, management,
and analysis of the information.
n represents both a challenge in harnessing this volume of data, and an
opportunity for government agencies who seek to enhance their effectivenes. - 3What are the 4 Vs of big data>?volume, velocity, variety, and veracity
- 4In what ways can scaling of resources be categorized?vertical scaling (scale up) and horizontal
scaling (scale out)
VERTICAL SCALING - referes to increasing computational poiwer of a single cpu (suyper computer) which may be increasing number of cpus in it. --> expensive to build, operate and
maintain
Horizontal scaling -
create a cluster of inexpensive commodity computers, which increase the computation
power by adding more machines. eg cloud computing takes advantage of this, - CONS - harder to maintain ACID principles across thousands of nodes
- API need tp be parallely programmed. - 5What is cloud computing?Cloud computing takes advantage of concept of horizontal scaling and provide computing power on pay per use basis in which we only pay for how much we use the computer on cloud. Also its unique characterstic is its elasticity ie its hardware can be elastically increased or decreased.,
- 6What are types of cloud computing services?Infrastructure as a Service (IaaS),
Platform as a Service (-
PaaS),
and
Application/Software as a Service (SaaS). - 7Advantages of cloud computing?easy access to
--- virtually infinite computing,
---storage and
--- networking
resources at low cost,
---- which saves money on hardware costs, ---maintenance costs,
----software licenses, etc - 8What are the Concerns associated with scaling?--Management of replicas becomes a huge challenge in terms of maintaining consistency of the replicas
with acceptable overhead, especially when disconnection happens.
---maintain the ACID properties in transaction management - 9What are the 3 types of data processing?OLAP, OLTP, RTAP
OLTP -
This type of transaction involves large volumes of short transactions (read and write) and queries from
multiple-users, which is usually related to DBMSs and database applications (e.g. banking, airline
bookings system). ACID properties and data integrity are very important in OLTP
OLAP-
OLAP gets the data from data warehousing (historical data batches) and data is rarely updated (which
does not expect too much writing). It is supposed to answer complex and multi-dimensional analytical
queries. For example, business reports, marketing, budgeting, forecasting, etc
RTAP-
ming data is processed in real-time continuously - 10what are types of big data processing?batch processing :-
data is processed after it has been collected. MapReduce/Hadoop is totally based on batch
real-time
processing of data streams:-
data can be processed as it arrives eg - r, Storm, Spark, and Samza
hybrid
processing :-
handle queries that require for both batch (historical) data and streaming (real-time) data eg. Lambda Architecture - 11Explain the lambda architecture and its use?Lambda architecture basically complement batch processing by providing a layer called speed layer which can provide query results keeping in mind the most recent data which is streaming through.
Serving layer takes care of the view from a batch perspective. Together both views can provide a complete answer to a query.
Precomputing batch views is a high latency operation, so while querying to the serving layer, it is likely
that the most recent data will not be considered - 12what are various concepts introduced for managing consistency in big data?consistency tradeoffs, CAP theorem, BASE, and
PACELC - 13Brewer's BASEBasically Available Soft-state Eventually consistent. Available all the time, can be inconsistent, but without updates wiill eventually become consistent again.
not suitable for banking apps since they require full ACID - 14CAPintroduces 3 properties out of which only 2 need to be selected for DDBS.
Consistency - server returns correct response to requests
Availability - server eventually responds to the request
P- partial failure tolerence - server despite network partition will continue the operations
All 3 cannot be selected as they are not possible together - 15PACELCIncase of Partition failure choose trade off etween consistency and availability.
else choose tradeoff between latency and consistency.
Latency decision depends on replication of data.
If measures to handle inconsistency are incorporated then both C can be dropped. For social media applications Latency is important than consistency. - 16What are 4 types of NoSQL dbs?Key value stores, column based, document based, graph based.
- 17What challenges for relational dbs led to shift to NoSQL dbs?1. requires rigid ACID properties
2. e lacks of support for
new emerging types of data
3. lacks of support for processing and querying larger
amounts of data in shorter time. - 18What are some benefits to using NoSQl vs traditional relational dbs?arge volumes of rapidly changing structured, semi-structured, and
unstructured data; agile sprints, quick schema iteration, and frequent code pushes; object-oriented
programming that is easy to use and flexible; and geographically distributed scale-out architecture
instead of expensive, monolithic architecture. - 19whats a key value store- simple hash table (map)
Used when -all accesses to the database are via primary key.
Each row contains a key and a value
- distributed but persistent hashmap data structures
- The key is a numeric identifier and the associated value can be simple text or a complex list of
attributes
eg. Hyperdex and dynamo - 20Whats a document based db?- subclass of key value store
- value = document
Documents are self-describing pieces of data that can support data hierarchy and structure (e.g. nested
objects and arrays in JSON
Note that attributes can have types, in this case, unit is of
integer type when name brewer is text typed
As can be seen, data is in name/value pairs, followed by a colon, which is separated by commas. Curly
braces are used to hold objects while square brackets hold arrays
eg. Mongo Db
- 21Whats a column based db?data is grouped in columns rather than
rows, and stored contiguously on the disk
benefits - faster access as no need o access data sequential scanning
- better compression as all data of same data type
-different rows are stored in different places on disk while Columnar databases store all the
cells corresponding to a column as a continuous disk entry,
s column-oriented database makes the
search/access faster.
When to use column based? - when query accessed few fields, and number of rows are large. good for OLAP not OLTP eg for aggregation since all cell in one column are required for aggregation.
In the column-oriented system, the primary key is the data, mapping back to rowids
eg. Hbase and cassendra
- 22Whats column family?In column-oriented database, the simplest unit of storage is the column. Columns can be grouped into
column families (e.g., customer profile, address) that can be accessed by a row key. Rows can be built
using row keys (to pair up the column values). - 23Whats a graph database?data is represented as graph, which is composed of nodes and relationships. Each
node represents an entity whereas each relationship represents how two nodes are related. eg Neo4j.
- where nodes are instances of objects (e.g.,
user) that have properties (e.g., name), and edges have directional significance and types
eg. Twitter
Information
- Description
- Distributed Database Flashcards on Big Data, created by Varun m on 04/06/2017.
- Created by
- Varun m
- Views
- 6
- Rating