Flashcards
Big Data Technologies
Flashcards do conjunto (162)
- 1Was ist ein zentrales Problem im Big Data BereichDatenmengen steigen zu schnell an
- 2What are the typical Characteristics of Big Data?
- 3Which kind of scaling out exists?Horizonal (Scale out) vs. Vertical (Scale up)
- 4What are the advantages and disadvantages of Scaling up?Advantages:
Transparent for database system
Management complexity constant
Disadvantages:
Expensive hardware
Limited scalability - 5What are the advantages and disadvantages of Scaling out?Advantages:
Less expensive hardware
High scalability
Disadvantages:
Load balancing necessary
Data distribution necessary
Increased error rate
Increased management complexity
Distributed lock and replication
protocols - 6How do NoSQL Database Systems scale?they are ofcourse vertically but also horizontally scalable.
The speciality of them is that NoSQL DBMS additionally can be distributed - 7What were the ideas of NoSQL DBMS?non-relational,
distributed,
open-source and horizontally scalable, modern web-scale databases,
schema-free,
easy replication support,
simple API,
eventually consistent / BASE - 8What are column store DBMS?As the name implies column-store, structure their data in a different way than the usual row-store approach.
instead of saving them row by row, the data saved column by colum. - 9What are the advantages of column-store?When selecting columns (instead of SELECT *) less data needs to be processed
Higher Compression ration
suitable for read-mostly, read-intensive, large data repositories (OLAP) - 10What are the disadvantages of column-storesinsert / update require multiple accesses
expensive reads on entire records - 11What are the advantages of row store?It is very easy to insert / update a record
- 12What are the disadvantages of row-storemight read in unnecessary data
- 13For which use-cases should you use column-stores?suitable for read-mostly,
read-intensive
large data repositories (OLAP) - 14What is OLAP?Steht für Online Analytical Processing
Durchführung komplexer Analysevorhaben stehen im Vordergrund,
welche ein sehr hohes Datenaufkommen verursachen. - 15What is OLTP?Steht für Online-Transaction-Processing
bezeichnet ein Benutzungsparadigma, bei dem die Verarbeitung von Transaktionen direkt und prompt, also ohne nennenswerte Zeitverzögerung, stattfindet. - 16What is ETL?Steht für Extract, Transform, Load (ETL) ist ein Prozess bei dem Daten aus mehreren ggfs. unterschiedlich strukturierten Datenquellen in einer Zieldatenbank vereinigt werden.
- 17What is NoSQL in essence?Data Model
• non-relational
• schema-free
Scalability
• distributed and horizontally scalable
• easy replication support
• tunable consistency - 18What kind of NoSQL DBMS exist?Considered Categories of NoSQL DBMS:
Key-Value Database Systems
Document Database Systems
Column Family Database Systems
Graph Database Systems - 19What Data Model do Key-Value have?Key-value pairs: (key) → value
Values
– arbitrary type (serialized byte arrays) or
– strings, lists, sets, ordered sets (of strings) - 20What Storage Layout do Key-Value DBMS use?Hash-Maps, B-Trees ...
- 21What Indexes do Key-Value have?Primary indexes (Hash, B-tree) on key
Secondary indexes on values - 22What key-Value DBMS do you know?Redis, Amazon DynamoDB
Memcached - 23What Interface / Query Models do Key-Value DBMS use?Simple API
set (key, value), get (key), delete (key)
More complex operations
Language Bindings
MapReduce - 24What Datamodel do Document Store DBMS have?Key-value pairs with “documents”
(key) → document
Document format: JSON or BSON - 25What Storage Layout do Document Store DBMS use?B-Trees to store the documents
MongoDB: Documents in a single collection are stored together - 26What Indexes do Document Store DBMS use?Primary indexes on documentId (key)
Secondary indexes on JSON-names
• Default or user defined
• Composite indexes may be supported - 27What Interface / Query Models do Document Store DBMS use?Simple API: set/get/delete
Further query support differ widely
• Powerful ad-hoc queries with integrated query language
• No ad-hoc queries, predefined views with indexes only
Language Bindings
MapReduce - 28What Document Store DBMS do you know of?MongoDB, CouchDB, Couchbase
- 29What Datamodel do Column Family DBMS use?(rowkey, column, timestamp) → value
Loosely structured by columns and column families (“set of nested maps”)
Column families
set of columns grouped together into a bundle
• Column families have to be predefined
Column
• Not predefined; any type or data (can be nested)
Column family database systems support multiple versions of each cell by timestamps: - 30What Storage Layout do Column Family DBMS use?Data is stored by column family
- 31What Interface / Query Models do Column Family DBMS use?Simple API
• set (table, row, column, value)
• value = get (table, row, column)
• delete (table, row, column)
• timestamp optional
Language Bindings
More powerful query engines integrated or as additional software - 32What Indexes do Column Family DBMS use?Primary indexes (B-Trees -> sorted ordered)
Default or user defined secondary indexes - 33What Column Family DBMS do you know of?Google BigTable, HBase, Cassandra
- 34What Data Model do Graph DBMS use?Most popular graph data model in
graph databases today
G = (V, E): Property Graph Model
– Directed labeled multigraph
– V: Vertex (Nodes)
• Entities
– E: Edge (Relationships)
• Connect entities and structure domain
– Properties
• Attributes and metadata: (key, value) pairs
– (Labels)
• Group nodes by role - 35What Storage Layout do Graph DBMS use?Adjacency List (seldom Adjacency Matrix)
- 36What Indexes do Graph DBMS use?Graphs are their own indexes!
– But sometimes we want short-cuts to well-known nodes
– AND: Indexes for uniqueness constraints!
(properties of nodes or relationships)
– B-Trees, Hash Indexes, Quadtrees, R-Trees - 37What Interface / Query Models do Graph DBMS use?- Traversal algorithms
– Queries (Graph Query Language example: Cypher) - 38What Graph DBMS do you know of?Neo4J, Infinite Graph, OrientDB (Multi-Model)
- 39What are other (Soft) NoSQL DBMS than the most prominent ones?Search Platform
Object-Oriented Databases
XML-Databases
RDF-Stores - 40How to How to represent NoSQL data models?No industry standard yet!
But First tools emerging - 41What is the Object-relational impedance mismatch?Is a set of conceptual and technical difficulties that are often encountered when a RDBMS is being served by an application program written using object-orientation, because objects or class definitions must be mapped to database tables defined by a relational schema.
- 42What is the "Primary Decision" when modelling NoSQL databasesEmbedding vs. Referencing
- 43What are the advantages and disadvantages of Embedding?works really well when data access is aligned with the "aggregates"
but is not flexible when using for a different purpose - 44What are the advantages and disadvantages of Referencing?Generally higher mapping costs but more degrees of freedom
- 45What are Aggregate-oriented data models/systems?Have a fundamental unit of storage which is a rich structure of closely related data
Make it easier for the database to manage data storage over clusters, since the unit of data now could reside on any machine
Work best when most data interaction is done with the same aggregate
Make inter-aggregate relationships more difficult to handle than intra-aggregate relationships - 46How to data model with Graph DBMS?Nodes for Things
• Relationships for Structure
• Represent complex value types as nodes
• Iterative and incremental development
• Test-driven data model development - 47How to query embedded/nested data / array in NoSQL DBs?Using the "dot notation"
Collection operators / array query operators
Eventually Unnest (flatten) arrays - 48What is the problem with the Heterogeneous interfaces / query languages in the NoSQL World?- Different functionality
– Heterogeneity
– Vendor lock - 49What is an alternative to using the Heterogeneous interfaces / query languages in the NoSQL World?Object-NoSQL Mapper
f.e. JPQL or JPA - 50What cateogories of Object-NoSQL Mapper do you know of?Multi Data Store Mapper
Single Data Store Mapper - 51How good is the JPQL Support in Object-NoSQL Mappers?JPQL but often with limited functionality (depending from the underlying NoSQL database system)
Many systems do not support JOINS
Many systems do not offer aggregate functions, LIKE operator, or NOT operator, … - 52What are the different apporaches for Object-NoSQL Mappers to supporting multiple DBMS?Offer only the particular subset of features that is implemented by all supported NoSQL data stores, i.e. the intersection of features
Distinguish by data store and offer only the set of features implemented by a particular NoSQL data store
Offer the same set of features for all supported NoSQL data stores, possibly complementing missing features by implementing them inside the Object-NoSQL Mapper - 53What is Map reduce?A Programming Paradigm which is a old idea from functional programming which was presented 2004
An algorithm for processing and generating big data sets with a parallel, distributed algorithm on a cluster. - 54What are the advantages of the map reduce concept?Different operations on the same data do not influence each other
No concurrency conflicts
No deadlocks
No race conditions - 55What need developer to do if they want to use Map Reduce?implement two primary methods
Map: (key1, val1) → [(key2, val2)]
Reduce: (key2, [val2]) → [(key3, val3)] - 56What are the Alternatives to MapReduce?Dataflow Programming Languages
– HiveQL
– Pig
– (Jaql) - 57What are problems around Map-Recuce?MapReduce paradigm is too low-level
– Only two declarative primitives (map + reduce)
– Custom code for simple operations like projection and filtering
– Code is difficult to reuse and maintain - 58What is HDFS?Java based
distributed file system
• HDFS is the underlying storage
system for storing data sets in the
distributed environment when
using Hadoop
When using Hadoop, data is
always copied to HDFS first - 59What is Pig?A platform for analyzing large data sets
• Pig consists of two parts:
– PigLatin: A Data Processing Language
– Pig Infrastructure: An Evaluator for PigLatin programs
– Pig compiles Pig Latin into physical plans
– Plans are to be executed over Hadoop
Interface between the declarative style of SQL and low-level, procedural style of MapReduce - 60What kind of compressions do you know?Run-length Encoding
Bit-vector Encoding
Dictionary Encoding
Frame of Reference Encoding
Differential Encoding - 61What is NewSQL?Idea behind: Best of both worlds
• SQL
• ACID
• Non-locking concurrency control
• High per-node performance
• Scale out, shared nothing architecture - 62What is Polyglot Persistence?Having heterogenous DBMS for different purposes in production
" the right system for the right task"
Distributed file system (HDFS) for compressed log data
– NoSQL system (HBase) for storage and instant random access
– Full text search engine (Apache Solr) for queries on log messages
– MapReduce framework (Apache Hadoop) for analysis (usage statistics and error)
– Relational DBMS (Oracle) for meta data (user infos etc.) - 63What would you benchmark when comparing NoSQL DBMS?Persistence
Durability - 64Benchmarks for NoSQL DatabasesYahoo! Cloud Serving Benchmark YCSB
Yahoo! Cloud Serving Benchmark: Metrics - 65When to use Hadoop?Use HDFS to store all company relevant data. (social media, sensor data, logfiles ...)
Use Map Reduce to filter important data. Use this data in a relational Datawarehouse for ETL purposes - 66Why use NoSQL in developement enviornment?Agile processes Schema profit from the flexible NoSQL datamodel since it can change very quickly
But work does not stop here, the model still needs to be migrated. - 67Which kind(s) schema freedom does NoSQL offer?without native schema support
with optional schema support
with mandatory schema support - 68What workflows do you know to create an application with a database?Forward Engineering
Schema Creation (f.e. JSON Schema)
Reverse Engineering
Schema Overview
Data Exploration
Advanced Reverse Engineering
Schema Version Extraction - 69How to do Datamigration?Custom-coded Migration Scripts
Using Object-NoSQL-Mapper Annotations
(@AlsoLoad) - 70What operation does a NoSQL Schema Evolution Language consist of?Single type operations:
– add property
– rename property
– delete property
Multi-type operations:
– copy property (denormalization)
– move property (refactoring) - 71What Basic Strategies of Data Migration do you know of?Eager Migration
after introduction of a new schema version, all entities are migrated
Lazy Migration
evolution operations are stored,
data migration is done on request - 72What are the advantages and disadvantages of Eager Migration?Advantages:
all entities are in the current version
low latency (when entities are accessed)
Disadvantages:
entities that are not in use are migrated
high number (and costs) of migration operations - 73What are the advantages and disadvantages of Lazy Migration?Advantages:
only entities that are in use are migrated
no unnecessary data migration operations
composition of operations is possible
Disadvantages:
entities in the db have different versions
increased latency - 74How to reduce costs of data migration?In case of large amount of datasets:
update operations even for cold data (that is not in use)
In case of database as a service:
payable operations, monetary costs for all data migrations - 75How to reduce costs of data migration?Optimize Data Migration
Hybrid / Proactive Migration Approaches
• Predictive Migration
• Incremental Migration - 76What Proactive Migration Strategies do you know of?Incremental Migration
in some version, an eager migration is applied
Predictive Migration
Forecast function, which entities are accessed in near future (based on heuristics)
• Predictive migration of these entities - 77What are the advantages and disadvantages of Incremental Migration?Advantage:
composition of operations is possible
Disadvantage:
even entities that are not in use are migrated - 78What are the advantages and disadvantages of Predictive Migration?Advantages:
decreased average latency
reduced number of migration operations
Disadvantage:
additional migration operations in case of wrong predictions - 79Which Tradeoffs have the different Migration Strategies?
- 80What is the CAP theorem?A rule that states that in a distributed system it is only possible to have at most two of the following three characteristics:
Consistency
Availability
Partition tolerance - 81In relation to the CAP theorem what is meant by Consistency?all clients have the same view, even in case of updates
- 82In relation to the CAP theorem what is meant by Availability?every request received by a non-failing node in the system must result in a
response (i.e., even when severe network failures occur, every request must terminate) - 83In relation to the CAP theorem what is meant by Partition tolerance?system properties hold even when the network (system) is partitioned
(i.e., nodes can still function when communication with other groups of nodes is lost) - 84What is the main point of the CAP theorem?Fundamental tradeoff between consistency, availability, and latency ≡ tradeoff between consistency and latency
- 85What is the PACELC theorem?If Partition ->
Tradeoff Availability vs. Consistency
Else
Tradeoff Latency vs. Consistency - 86What kind consistency do you know of?Strong vs. Weak Consistency
- 87What is strong consistency?After the update completes, any subsequent access will return the updated value, i.e., any subsequent access from A, B, C will return D1
- 88What is weak consistency?The system does not guarantee that subsequent accesses will return the updated value D1 (a number of conditions need to be met before D1 is returned)
- 89What is Eventual Consistency?Specific form of weak consistency
– Guarantees that if no new updates are made, eventually all accesses will return D1
– If no failures occur, the maximum size of the inconsistency window can be determined based on factors such as communication delays, the load on the system, and the number of replicas involved in the replication scheme. - 90What is BASE?Basically available: Availability of the system even in case of failures
Soft-state: The state of the system may change over time, even without input (clients must accept stale state under certain circumstances.)
Eventually consistent: The system will become consistent over time, given that the system doesn't receive input during
that time - 91Which variations of Eventual consistencies exist?Causal consistency: – If A notifies B about the update, B will read D1 (but not C!)
Read-your-writes: – A will always read D1 after its own update
Session consistency: – Read-your-writes inside a session
Monotonic reads: If a process has seen Dk, any subsequent access will never return any Di with i < k
Monotonic writes: Guarantees to serialize the writes of the same process - 92What is ACID?Atomicity
Durability - 93What is the connection between Transactions and Atomicity and Durability?...
- 94What kind of Sharding do you know of?Hash-based
Range-based
Entity-Group
Consistent Hashing - 95What is Hash-based Sharding? What are the pros and cons?Hash of data values (e.g. key) determines partition (shard)
Pro: Even distribution
Contra: No data locality - 96What is Range-based Sharding? What are the pros and cons?Assign ranges defined over fields/attributes (shard keys) to partitions
Pro: Enables Range Scans and Sorting
Contra: Repartitioning/balancing required - 97What is Entity-Group Sharding? What are the pros and cons?Explicit data co-location for single-node-transactions
Pro: Enables ACID Transactions
Contra: Partitioning not easily changeable - 98What is Consistent Hashing? What are the pros and cons?Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web.
Keys and nodes (identified by id or server name) are mapped onto a “circle”
Keys are assigned to the node that is next to them in a clockwise direction - 99What is the Motivation / Benefit Replication?Performance enhancement
Availability enhancement (fault tolerance)
Tradeoff between benefits of replication and work required to keep replicas consistent - 100What basic (and orthogonal) parameters does Replication have?Two basic (and orthogonal) parameters:
• When are the updates propagated?
– synchronous vs. asynchronous
• Where is the update performed?
– update everywhere vs. selected copy - 101What is meant by "When" in the context of Replication?Asynchronous (lazy)
– Writes are acknowledged immediately
Synchronous (eager)
– The node accepting writes synchronously, propagates updates / transactions before acknowledging - 102What are the advantages and disadvantages of Synchronous (eager) Replication?Pro: Consistent
Contra: needs a commit protocol (more roundtrips), unavailable under
certain network partitions - 103What are the advantages and disadvantages of Asynchronous (lazy) Replication?Pro: Fast writes, no coordination needed
Contra: Replica data potentially stale (inconsistent / weak consistency) - 104How can you guarantee consistency without synchronous update propagation?Read only from master node (replicas for failover purposes only)
Quorum Consensus Protocols - 105What are Quorum Consensus Protocols?In a distributed database system, a transaction could be executing its operations at multiple sites. Since atomicity requires every distributed transaction to be atomic, the transaction must have the same fate (commit or abort) at every site. In case of network partitioning, sites are partitioned and the partitions may not be able to communicate with each other. This is where a quorum-based technique comes in. The fundamental idea is that a transaction is executed if the majority of sites vote to execute it.
N = number of replica store
W = number of replicas that need to ack an update before a commit is successful
R = number of replicas that are contacted on read
QCP states that if
W+R > N is true this state is called a Quorum Assembly
Strong consistency
If W+R <= N -> Weak consistency
has a role either
N = Replica
W = - 106What configurations of Quorum Consensus Protocols do you know of?W=N and R = 1 -> read optimized strong consistency (ROWA)
W=1 and R = N -> write optimized strong consistency
R = W = N/2 + 1 -> Majority Consensus - 107What kind of Quorum Consensus variations do you know of?Weighted votes: Each node is assigned a weight, e.g. for “better” replicas. Use sum of node weights instead of node sum N
Static vs. dynamic quorum
Dynamic Quorum: Quorums can be chosen separately for each item, e.g. in case of unavailable nodes - 108What Asynchronous Write Strategies do you know of?Write-Related Strategies
Hinted Handoff Algorithms
Read-Related Strategies
Read Repair
Anti-Entropy - 109What is the Read Repair Algorithm?A system may detect that several nodes are out of sync with older versions of the data requested in a read operation
Mark the nodes with the stale data with a Read Repair flag
Synchronizing the stale nodes with newest version of the data requested - 110What is AntiEntropy?AntiEntropy means comparing all the replicas of each piece of data that exist (or are supposed to) and updating each replica to the newest version.
- 111What Replication strategies do you know?Master-Slave (Primary Copy)
Multi-Master (Update anywhere) - 112What is Master-Slave?Only a dedicated master is allowed to accept writes
Slaves are read-replicas
Different data items can have different primary nodes - 113What are the pros and cons of Master-Slave?Pro: reads from the master are consistent
– Contra: master is a bottleneck and SPOF - 114What is Multi-Master?The server node accepting the writes synchronously propagates
the update or transaction before acknowledging (see also Quorum
Consensus) - 115What are the pros and cons of Multi-Master?– Pro: fast and highly-available
– Contra: either needs complicated coordination protocols (e.g. Paxos) or is inconsistent - 116How to detect older versions of data? How to detect concurrent writes?Timestamps!
- 117What is a common Common approach in NoSQL database systems to detect conflicts?Vector Clocks
- 118What are Vector Clocks?allow to determine whether one object is a direct descendant of the other /
direct descendant of a common parent / are unrelated in recent heritage - 119What is Local Secondary Indexing?An index of a non primary key attribute over the whole cluster which is stored on each cluster-node seperately to its own content
- 120What is Global Secondary Indexing?An index of a non primary key attribute which is saved clustered and each specific cluster only on one specific node
- 121What are the right criteriasData Analysis
Consistency Requirements
Query Requirements
Performance Requirements
Non-functional Requirements
Costs - 122What are functional requirements?Scan Queries
ACID
Conditional or Atomic Writes
Joins
Sorting
Filter Queries
Full-text Search
Aggregation and Analytics - 123What are Non-Functional requirements?Data Scalability
Write Scalability & Read Scalability
Elasticity
Consistency
Write Latency & Read Latency
Write Throughput
Write Availability & Read Availability
Durability - 124How to select the Right DBMS?Prototyping and performance analysis!
- 125What is the NoSQL Decision Tree?
- 126What is Polyglot Persistence?
- 127What is NewSQL?Idea behind: Best of both worlds
• SQL
• ACID
• Non-locking concurrency control
• High per-node performance
• Scale out, shared nothing architecture - 128What are the Key Features of a Column Store ?Columnar storage
-> Compression
-> Multiple sort orders
Execution Engine
– Avoid decompression -> operating directly on compressed data
– Early vs. late materialization
– Updates - 129What kinds of compression do you know of?Run-length Encoding
Bit-vector Encoding
Dictionary Encoding
Frame of Reference Encoding
Differential Encoding - 130What is Run-length Encoding?instead of using the default encoding
11110000 RLE counts the bit
-> 44 - 131What is Bit-vector Encoding?For each unique value v in column c, create bit-vector b: b[i] = 1 if c[i] = v
• Good for columns with few unique values
• Each bit-vector can be further compressed if sparse - 132What is Dictionary Encoding?For each unique value create dictionary entry
Dictionary can be per-block or per-column - 133What is Frame of Reference Encoding?Encodes values as b bit offset from chosen frame of reference
Special escape code (e.g. all bits set to 1)
indicates a difference larger than can be stored in b bits
After escape code, original (uncompressed)
value is written - 134What is Differential Encoding?Encodes values as b bit offset from previous value
Special escape code (just like frame of
reference encoding) indicates a difference larger than can be stored in b bits
After escape code, original (uncompressed) value is written
Performs well on columns containing
increasing/decreasing sequences - 135When to use which encoding?
- 136How to operate directly on compressed Data?
- 137What is Early Materialization?When collection a result set the rows are created first
- 138What are the drawbacks of Early Materialization?Need to construct ALL tuples
Need to decompress data
Poor memory bandwidth utilization - 139What is Late Materialization?When collection a result set the rows are created first
- 140When should always Late Materialization be used?For plans without joins, late materialization is a win
- 141Why is Late Material worse on joins then EM?is about 2X slower than EM join on typical queries (due to random I/O)
- 142What is an Invisible Join?...
- 143When to use an Invisible Join?Designed for typical joins when data is modeled using a star schema
- 144When to use a Jive/Flash Join?lnstead of probing projected columns from inner table out-of-order
Sort join index
• Probe projected columns in order
• Sort result using an added column - 145LM vs EM tradeoffsLM has the extra sorts (EM accesses all columns in order)
• LM only has to fit join columns into memory (EM needs join columns and all projected columns)
• LM only has to materialize relevant columns
• In many cases LM advantages outweigh disadvantages - 146Radix Cluster/Decluster JoinThe full sort from the Jive join is actually overkill
– We just want to access the storage blocks in order
(we don’t mind random access within a block) - 147Why are Column Stores update averse (abgeneigt)?– In-place: I/O for each column
+ Re-compression
+ Multiple sorted replicas
+ Sparse tree indices - 148How do Column-stores actually update in place?Differential lists/files or more advanced
– Updates buffered in RAM, merged on each query
– Checkpointing merges differences in bulk sequentially
• I/O trends favor this anyway (trade RAM for converting random into sequential I/O) - 149What are the typical bottlenecks in DBMS?in the past: Disk I/O Performance
today: CPU waiting for data to be loaded from memory into cache
-> Padding, (correct) Prefetching, - 150What part of a DBMS Layer Architecture is not necessary any more?
- 151What Data Layouts do you know?Row-store data layout
Column-store data layout
Hybrid Data Layout - 152What techniques does the Hybrid Data Layout use?horizontal and vertical paritioning
- 153What implementation Techniques do you know of to improve the performance of Databases?Compression
Reduce Locking Overhead
Differential Buffer
Insert only - 154What is a differential buffer?Instead of having directly a table (main store) the concept of a differential Buffer introduces an additional layer (buffer)
New values are written to a dedicated differential buffer (delta)
Cache sensitive B+ trees (CSB+) used for faster search on delta - 155What is the motivation behind the Differential Buffer?Inserting new tuples into a compressed structure can be expensive
– New values may require a reorganization of the dictionary
– Number of bits required to encode all dictionary values may change
– Attribute vector has to be reorganized
Deletion of tuples is expensive
– All attribute vectors have to be reorganized
– valuedIDs of subsequent tuples have to be adjusted - 156What are the advantages and disadvantages of a Differential Buffer?Advantages: Insert of new values are faster, because dictionary and attribute vectors need not be resorted
Disadvantages: Range select on differential buffer is expensive, based on unsorted dictionary
– Differential buffer requires more memory (no compression + CSB+ tree) - 157What needs to a Differential Buffer need to do to keep its performance over time?needs to be merged
periodically into the main store - 158Of what phases does the merge of the Differential Buffer consist of?Consist of 3 Phases
1. Prepare merge
2. Attribute merge
3. Commit merge - 159Are there any other Differential Buffer merging techqniques?-> Online Merge
Working on data copies allows asynchronous merge
Very limited interruption due to short lock
But: At least twice the memory needed! - 160What does a Differential Buffer consist of?Differential Buffer
Attribute Vector
Dictionary
Main Store
Attribute Vector
Dictionary - 161Are In-memory DBMS comparable to Disk-Based DBMS + Tons of MemoryNo since the optimizations for Disk Based databases are for mostly not useful for the other kind of hardware!
- 162What are the Usecases of Big DataFraud and Risk Management
Churn Management
Social Network Data
Gaming Data
Shopping Carts
Web Analytics
Log Analytics
IoT: Internet of Things
Real Time Analytics
Informações
- Descrição
- FlashCards sobre Big Data Technologies, criado por Vornamei Nachnamei em 12-07-2019.
- Criado por
- Vornamei Nachnamei · jul 2019
- Atualizado
- 16 jul 2019
- Visualizações
- 9
- Avaliação