Wk 2 Data Cleaning & Integration

Wk 2 Data Cleaning & Integration

1. Data Curation:  Ingest •  Validate •  Transform •  Correct •  Consolidate (dedup) •  And visualize information to be integrated

2. Extract-Transform-and Load (ETL) tools

   • Current Situation:  Enterprises want to integrate more and more data sources
     • Data Preparation ToolsTrifacta Paxata Alteryx Cambridge Semantics Clear Story Informatica Springbok/Re
       • Enterprise Data Curation: Tamr, Deep Dive
       • Ingest (into a common place) Clean (-9999 means null) Transform (Euros to $) Schema match (your salary is my wages) De-duplicate (Mike Stonebraker, M.R. Stonebrak Export (to a downstream system of record) Data Curation Entails

3. Cloud Computing: Means computing resources available “on demand”

   • Examples: Low-level storage and computing – Amazon S3 and EC2; Google Compute Engine; Windows Azure; Rackspace •  Hosted services – Amazon Relational Database Service (MySQL/Oracle) – Google BigQuery, Amazon Redshift (in-house systems) •  Vertical applications – Salesforce, Splunk, Tableau
   • Benefits for Providers: Economies of scale, Fast deployment, Optimization across users. Statical multiplexing
   • Benefits for Users: Fast deployment, Outsourced management, Lower costs, Elasticity
   • Levels of Abstraction: Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS)
   • Multitenancy: Public Cloud, Private Cloud
   • Challenges: 1. Security. 2. Availability. 3. Data Transfer. 4. Lock-In

Images and files in the map