BT4221 Lecture 1

BT4221 Lecture 1

1. What is Big Data?

   • Big Data: Definition
     • Velocity, Volume, Variety
       • ► Velocity: data in and out ► Variety: structured, unstructured, semistructured. ► Volume: size of the data
     • Value, Veracity
       • ► Value: data creating value? ► Veracity: true, credible data
   • Big Data: Growth
     • How has big data grown over the years
       • ► Require bigger storage space ► Increased processing power ► Availability of data Example: 2.5 quintillion of data every day, 90% of the data available created in the past 2 decades
   • Big Data: Volume
     • Examples
       • ► Typical PC in 2000 only 10 Gigabyte storage ► Facebook 500 Terabytes/ day ► Single cross country flight 240TB ► Boeing 737 10TB/ 30 min ► Smartphones: data created and consumed, sensors embedded objects, constantly updated data feeds that contain environment, locations, videos, other information
   • Big Data: Velocity
     • Examples
       • ► Clickstreams, ad impression capture customer behavior at millions of events/ second ► High frequency trading algorithms reflects market changes within microseconds. ► Machine to machine processing data between billions of devices ► Infrastructures and sensors generate massive log data in realtime ► Online gaming support millions of concurrent users
   • Big Data: Variety
     • Examples
       • ► Not just numbers, geospatial, 3D, audio and video, unstructured text, log files, social media ► Traditional database: smaller volume, structured data, fewer updates, predictable consistent data structure

2. Issues with Big Data

   • Big Data: Problems
     • Small vs Big Data
       • ► Small Data: only focuses on how to collect data and build statistical inference, deemed as Business Process Oriented operational efficiency ► Big Data: focuses on how to process big data and analytics, deemed as data driven business and innovations
     • Paradigm Shift
       • ► Shift from pre process to post process which challenges the system to store and analyse it all ► From sampling to enumeration: N is big; thus, everything is significant; finding a pattern vs. linear prediction ► From quality to quantity ► From casual relationship to correlation
     • 3 Problems
       • ► Cannot process big data in timely manner ► Market information can quickly become obsolete or unvailable ► The time window for sensing and responding to opportunities and threats close quickly
   • Big Data: Value & Potential
     • Value
       • ► We do not know who, how, what, and when to use the data for "result" ► Thus, we need to re-use and mash-up data (e.g. Google 2M times per minute) and at the same time, come up with innovative technology to reduce the cost of analysis and convert the redundancy to the core. ► Examples of innovative service: cinematch - movie recommendation; the long tail economy with personalized service (selling small but various products for a long time -> greater revenue); throttling - fairness algorithm that takes care of the most profitable customers while holding other customers
     • Potential
       • ► Big data presents a lot of potential (in term of job demand) as well as challenges
   • Big Data Applications
     • Organizations
       • ► Manufacturing: Enterprise Resource Planning, Knowledge Management System, Real-time Enterprise ► Finance: Enterprise Finance Application Integration, E-CRM, Enterprise Datawarehouse ► Logistics: Supply Chain Management, Balanced Score Card, Key Performance Indicator
     • Social Overhead Capital
       • ► Logistics: Commercial Vehical Operation System, Commerce at Light Speed, Electronic Data Interchange  ► Gepgraphic: Geographic Information Systm, Remote Sensing, Intelligence Transport System, Spatial Information Management
     • Medical
       • ► Diagnose: Data Driven Radiology, Picture Archiving and Communication System ► Health Management: Health Management Data Protocol, Ubiquitous- Health
   • Big Data Benefits, Strategy and Challenges
     • Benefits
       • ► Make better decisions and take meaningful actions ► Big data technology such as hadoop give you the scale and flexibility to store data before you know how you are going to process it ► Technologies such as MapReduce, Hive and Impala enable you to run queries without changing the data structure underneath
     • Strategy
       • ► Users: Customers/ Companies/ Machine - who can consume data the most efficiently and effectively? - which one is more important: consumer data or company data? ► Big is not the core of big data, the strategic information and knowledge derived from it is the core
     • Challenges
       • ► Garbage in - garbage out: data analysis on a useless business model always leads to a failure business model ► 3Vs is the key: need to use all to figure out the feasibility of the business model ► Will be overwhelemed ► Costs escalated too fast ► Many sources are private

3. Techniques & Technologies

   • Big Data Analytics Structures: ► Data Sources ► Data Warehouse ► Data Analytics
   • Big Data Storage
     • Features
       • ► Analyse your data characteristics: select data sources, eliminate redundant data, establish the role of NoSQL  ► Examples: Hadoop Distributed File System; HBase; Hive ► We need to: select the correct data storage for our data characteristics, move code to data, implement polyglot data store solutions, align business goals to appropriate data store
   • Big Data Technologies
     • Questions
       • ► Where is data stored? distributed storage - Amazon S3 ► Where is it processed? distributed servers/ cloud - Amazon EC2 ► What is the programming model? distributed processing - MapReduce ► How data is stored and indexed? High performance schema free data bases - MongoDB ► What are the operations? Analytics/ Semantic processing
     • Providers
       • ► IBM - Netezza ► EMC - Greenplum ► Oracle - Exadata
     • Processing
       • ► Integrating disparate data stores: mapping data into programming framework, connect and extract data for storage, trasnform data for processing, subdividing the data in preparation for Hadoop MapReduce ► Employing Hadoop MapReduce: Create components of MapReduce jobs, distribute data processing across server farms, execute, and monitor the progress of job flows
   • Analytics
     • Definition
       • ► Extensive use of data, statistical and quantitative analysis, explanatory and predictive models, and fact-based management to drive decisions and actions ►Examine large amount of data, obtain appropriate information, identify hidden patterns, unknown correlations, to...  ► Obtain competitive advantage, make better business decisions, leading to effective marketing, customer satisfaction, increased revenue
     • Techniques
       • ► User Interaction: Visualization, Exploration, Context and Relationship navigation, Automation ► Applied Analytics: Linguistic interpretation, Accurate models, Analyse stream data in motion, Algorithms uncover hidden patterns ► Data Management: Surface data directly from source, Query specialized systems, Data relationships and networks
   • Techniques
     • Techniques
       • ► Machine learning, data mining, predictive analytics, etc. use data to predict some variable as a function of other variables ► Examples: associate rule, genetic algorithm, classification tree, sentiment analysis
     • Challenges
       • ► May or may not care about insight, importance, patterns ► May or may not care about inference - how y changes as some x changes
     • Econometrics
       • ► Use statistical methods for prediction, inference, casual modeling of economic relationship  ► Casual inference is goal
     • Examples
       • ► Associate rule: customers buy beers and diapers together ► Genetic algorithm: maximise profits from stock market porfolio ► Classification tree: what kind of voters vote for certain party ► Sentiment analysis: ► Social network analysis: degree of separation ► Econometric: what kind of occupation impact to buy a car?