Big Data - created from Mind Map

Page 1

Scale out, not up

limits of large shared-memory machines

Move processing to the data

cluster have limited bandwidth

Process data sequentially, avoid random access

Seamless scalability

from the mythical man-month to the tradable machine-hour

Row data

Calculated values

Metadata

Data Hierarchy

what is big data

definition of big data

how to store?

Challenges:

Provisioning for peak demand

Resource under-utilization

Capacity planning: too many variables

Storage management: a massive challenge

System upgrades: extremely time-consuming

Complex mine-field of software and hardware licensing

how to manage?

have Measured

might be Observed

Derived from raw data

Calculated as needed

Pre-calculated and stored

explain how data is to be interpreted

Data

Information

Raw facts

Unprocessed information

Processed to reveal meaning

size is beyond the ability

using tools to capture, store, manage, analyze

Volume

Terabyte 10**12

Petabyte 10**15

Exabyte 10**18

Velocity

Value

Variety

Data Volume/unit of time

Raw Data: Low value, but not valueless

Structured Data

Semi-structured Data

Unstructured Data

RDBMS technology

Cloud challenges: Elasticity

Distributed processing

is beyond traditional parallel programming models (MPI), parallel programming is not suited for big data

using simple way for simple data

MapReduce

Most popular implementation is Hadoop

Stream Computing

IBM infosphere streams, storm, spark

Making parallel distributed processing to reduce size of sets data

by using lots of different servers at lots of different places at the same time

reduce data into key value pairs

incoming data is constant

no way to do it in batches (separately)

Real-time analytics

analyze data before it is stored