Uber

Uber

1. Safer & Reliable transportation

2. Gen 0

   • Limited data
     • Scattered
     • few terabytes
   • OLTP
     • Unrelated tables

3. Gen 1

   • Data driven
   • Data grew exponentialy
     • tons of terabytes
   • Data warehouse
     • Analisis
     • Centralization
     • Vertica
     • SQL interface
   • Problems
     • Reliability
     • adhoc communication
     • Modifiability
     • expensive
     • Scalability
     • Duplication

4. Gen 2

   • Hadoop
     • Data lake
     • Scalable ingestion
     • Interface
       • Presto
       • Hive
       • Spark
     • Parket
   • Scalable
     • Efficient
     • Stable
   • Schema
     • Service
   • Petabytes of data
     • tens of therabytes per day
   • Problems
     • HDFS small files
     • Latency

5. Gen 3

   • Interface
     • Presto
     • Hive
     • Veretica
     • Spark
     • Notebook
   • Research
     • HDFS scalability limitation
       • ViewFS
       • HDFS NameNode Federation
     • Faster data in Hadoop
       • Incremental ingestion of only updated and new data
     • Update & Delete in Hadoop & Parquet
       • Variable age of data
       • Complex data
     • Faster ETL & modeling
       • Incremental
   • Hudi
     • Efficient
   • No transformation of tata during ingetsion
     • Hadoop
   • Incremental data modeling
     • Generic ingestion
     • Views
       • Latest
       • Snapshot & changes
   • Standardized data model
     • Tables
       • Changelog / History
       • Latest

6. Gen 4

   • "Current"
   • Enhance
     • Latency
       • 5'
       • Hudi
     • Efficiency
       • Docker
       • Locality
     • Scalability & Reliability
       • Larger parquet files
       • Unification of content, format & metadata
     • Quality
       • Avoid non-schema-conforming
         • Mandatory schema checks
       • Content
         • Semantic

7. Uber’s Data Storage