GoConqr has a new look! Let us know what you think.

Sign up free

Flashcards

Big Data Management

Ash Z English 126 cards

Cards in this deck (126)

  1. 1ETL ProcessSelect
    Extract
    Transform
    Integrate
    Load
  2. 2DatawarehouseA physical repository where relational data is organized
    Cleansed Data
    DSS Functions
  3. 3DW CharacteristicsSubject Oriented
    Integrated
    Time-Variant
    Not Normalized
    NonVolatile
    Web-Based
  4. 4Data MartA subcategory of data that is only related to one another (e.g. Finance, Sales, Business)
  5. 5Types of DataMartDependent - A subset that is created directly from a data warehouse
    Independent - A small Date warehouse designed from a strategic business unit or a department
  6. 6ODSOperational data stores
    A type of database often used as an interim area for a data warehouse
  7. 7Oper MartsOperational Data mart
  8. 8EDWEnterprise Data Warehouse
  9. 9MetadataData about Data
  10. 10Operational DBs vs Data warehouseNormal relational DB's are used for operational purposes:
    - Support everyday operations
    - Mainly transactions
    - Online Transactional Processing (OLTP)
    Data warehouse allows for analysis of the data across multiple aspects.
    Online analytical processing (OLAP)
  11. 11OLTPOnline Transaction Processing
    - Get Data in
    - Daily activities
    - Small Data
    - Data Overwritten
  12. 12OLAPOnline Analytical Processing
    - Data Out
    - Decisions
    - A large amount of data
    - Regular Updates
  13. 133 VVolume
    Variety
    Velocity
  14. 14Data Problems- Typos
    - Field data not matching
    - Address variation
    - Names
    - Inconsistent spacing
  15. 15Most time costing process?ETL
    50 - 75% of time is required here
  16. 16ETL Problems- Complexity
    - Incorrect Mapping
    - Frequent Change
    - Conflicting business rules
    - Infrastructure issues
    - Cost
    - Skill Set
  17. 17Criteria to Select an ETL ToolCompatibility
    Functionality
    Reputation
    User-Friendly
  18. 18Dimensional ModelingA retrieval-based system that supports high-volume query access
  19. 19Star SchemaThe most common dimensional model
    Fact Table - Descriptive attributes needed to perform decisions
    Dimension Tables - Classification and aggregation information about the values in the fact table
  20. 20Snowflake SchemaAn extension of star schema
    Snowflake Shape
  21. 21Multi dimensionUsing several dimensions to analyze data
    Dimensions (Products, People)
    Measures (Money, Sales)
    Time (Daily, Monthly)
  22. 22Benefits of Hosted Data WarehousesRequires minimal investment
    Frees up the capacity of in house
    Frees up cash flow
    Powerful solution affordable
    Faster Connections
    Remote Access
  23. 23DW ArchitecturesIndependent Data Mart
    Data Mart Bus (Kimball Model)
    Hub and Spoke Architecture (Inmon Model)
    Centralized data warehouse
    Federated
  24. 24Independent Data MartNo consistency between data marts
    Easy development
    Difficult to query across more than one
  25. 25Data Mart Bus (Kimball Model)Bottom-Up
    Shared dimensions across data marts
    Dimensional modelling approach
    Queries across marts
    Virtual data warehouse
    Data is extracted from data sources
    Transformed and stored in separate data marts
  26. 26Kimballs 4 Step ProcessSelect the business process
    Identify the required grain of the data
    Choose the dimensions
    Identify Fact
    Repeat
  27. 27Hub and Spoke Architecture (Inmon Model)Most Common
    A central warehouse which then divided into dependent data marts
    Most queries use the data marts
    - Gather Data from data sources
    - Transforms and stores in centralized DB
    - Extracts data from DW to form data marts
    Top Down
  28. 28Centralized data warehouseNo data marts
    One big central datastore
    Queries are made through the data warehouse
    Variety of level to store data
  29. 29FederatedAd-Hoc Architecture
    Integrate pre-existing decision support systems
    Links smaller ones together
    Developers need to identify shared keys
  30. 30Breadbox AnalysisAnalyzing the size and granularity of the data
  31. 31Selecting an Architecture- Level of information interdependence
    - Information Needs
    - Nature of end-user
    - Constraint on resources
    - The ability of in house staff
    - Compatibility with existing systems
  32. 323 Tier Architecture1. Data acquisition
    2. A data warehouse that contains the data
    3. Client
  33. 33OLAP CubesFinal Stage
    User queries are directed to the OLAP Cube
    Rapid Response to queries
    Table Based
    Less flexible queries
    More than 2 dimensions
  34. 34OLAP OperationsSlicing - Selecting a slice from the cube by fixing the value to one dimension
    Dicing - Sub-Cube
    Drill Down/Up - Changing the level at which dimension is being analyzed
    Roll-Up - Summarising data along a dimension
    Pivot - Changing the visualization of a cube to examine different dimensions
  35. 35Normalized databases3NF
    Divide the database
    Discrete Entities
    Minimise data duplication
    Transnational database
    Not suited for OLAP
  36. 36Fact TableSingle business process
    Stores measurement generated by that process
    Generally Numeric
    Contains FK
    Lowest Possible Granual
  37. 37Periodic Snapshot Fact TablesSingle row for something the enterprise tracks closely
  38. 38Dimension TablesQuery Filtering
    Query Result Labelling
    If the user wants to query by something (e.g. Date, Time, Product) this uses a dimension
    Does create redundancies however
  39. 39Surrogate KeyAuto Increment Key
    Short Lifetime
    Can Recycle
  40. 40Conformed DimensionsConsistency across tables
    Support Integration from multiple parts
    Can be linked to multiple fact tables
  41. 41Bridge TableMany to Many Relationships
    Each row contains:
    - Group Key
    - Dimension Key
  42. 42Outrigger SchemaA mid-way point between star and snowflake
  43. 43ExtractionReading the required data
    Wide variety of sources
    Data is added to a staging area where it can be transformed
  44. 44Staging2 Options:
    - Relational DB
    - Flat Dimension
  45. 45Extraction Sub-SystemsData Profiling
    Change Data Capture System
    Extract System
  46. 46TransformationCleaning
    Conforming
    Fixing Data
    Staradising Formats
    Eliminating Duplicates
  47. 47Transformation Sub-SystemsData Cleansing
    Error Event Schema
    Audit Dimension Assembler
    Deduplication
    Conforming
  48. 48LoadingPreparing data to load in the DW
    Appropriate dimension tables
    Make dimensions and load the data
  49. 49Loading Sub-Systems9. Slowly Changing Dimension Manager
    10. Surrogate Key Generator
    11. Hierarchy Manager
    12. Special Dimension Manager
    13. Fact Table Builder
    14. Surrogate Key Pipeline
    15. Multivalued Dimension Bridge Table Manager
    16. Late Arriving Data Handler
    17. Dimension Manager
    18. Fact Provider
    19. Aggregate Builder
    20. OLAP Cube Builder
    21. Data Propagation Manger
  50. 50ManagementBI relies on timely, consistently reliable data
    Which needs managing
  51. 51What is Big Data?Data in which it is hard or impossible to use traditional methods to store, process or transform the data.
  52. 52Big Data Factors of GrowthCloud
    Social Media
    Mobile
  53. 53Why traditional approaches cannot work?The server cannot handle the volume
    The database cannot handle the volume
  54. 54Data Processing ModelsScale Up
    Scale-Out
  55. 55Scale UpAdding Bigger and more powerful machines
    Expensive
    Less Storage
    Low Fault
  56. 56Scale-OutTaking advantage of existing hardware already in inventory, or by purchasing more of the smaller, less expensive machines
    Cheap
    High Fault
    Petabytes
  57. 57HadoopReliable
    Scalable
    Distributed
    Processing across clusters of computers
    Scale-up from a single server
  58. 58Why Apache HadoopOpen Source
    Simple
    Supports higher parallel processing
    Scalable
    Low cost
    Fault Tolerant
  59. 59Hadoop SubsystemsApache HBase
    Sqoop
    ZooKeeper
    Mahout
    Pig
    Hive
  60. 60Hadoop EcosystemMultiple software components operate with Hadoop
  61. 61Name Node (NN)The centerpiece of the HDFS
    Has RAM, CPU, and storage
    More powerful than a Data node
    Stores HDFS metadata
    Does not store the data
  62. 62Data Node (DN)RAM
    CPU
    Storage
    Hosts Task Tracker
    Stores Data
  63. 63Client Node (C)Access from non-Admin
    Hadoop access components
    No role in a cluster
  64. 64HDFSHadoop File System (HDFS)
    Storage across clusters
  65. 65HDFS ISThe current directory
    $HDFS DFS -Is
  66. 66HDFS PUTCopy a local file to HDFS
    $ hdfs dfs -put
  67. 67HDFS GETCopy HDFS file to local
    $ hdfs dfs -get
  68. 68HDFS Remove$ hdfs dfs -rm
  69. 69HUEHadoop User Experience
    Web-based interface to enable users to perform Hadoop tasks
  70. 70HUE ComponentsFile Browser
    Query Editors
    Job Browser
  71. 71MapReduceMap Reduce is a software system for parallel processing of distributed data in HDFS
  72. 72Hadoop TrackersJob Tracker
    - Manages resources and the Task Tracker
    Task Tracker
    - Takes orders from Job Tracker
    - Updates Job Tracker
  73. 73YARNYet Another Resource Negotiator
  74. 74SqoopCommand Line interface application for transferring data between RDBS
    Moving RDBS data into Hadoop
  75. 75Sqoop Syntax$ sqoop import \
    --connect jdbc:mysql://localhost:port/db \
    --username foo \
    --table TEST
  76. 76Sqoop Export$ sqoop export \
    --connect jdbc:mysql://localhost:port/db \
    --table bar \
    --export-dir /results/bar_data
  77. 77HIVEDatabase infrastructure fo HDFS
    Easy Data Summarization
    HIVEQL
    Easy ETL
    Access to HDFS
    Mapreduce Code
  78. 78HIVE ArchitectureInstalled on Client Node
  79. 79Hive ModulesCompiler
    Metastore - Metadata
    Command Line Interface (CLI)
    JDBC and ODBC
    Thrift Server
  80. 80MetastoreSeparate Database outside Hadoop
    How to read and write from Hive
    Metadata
  81. 81Hiveserver 2 UpgradesImproved JDBC and ODBC
    Concurrent Thrift Client
    Authentication
    Authorization
  82. 82Accessing HiveBeeline
  83. 83Managed Hive TableMetastore is managed by Hive
  84. 84External Hive TableIt is external and not managed by Hive
  85. 85PigLanguage: Pig Latin
    Data Flow application for Hadoop
    High-level language interface
    SQL like interface
  86. 86Pig Data TypesStructured (database Files)
    Unstructured (Web Logs)
    Semi-structured (XML, Email)
  87. 87Pig ComponentsPig Latin
    Grunt
  88. 88Pig LatinCommand Based language mainly used for data transformation and flow expression
  89. 89GruntCommand-line tool for running and debugging pig statements
  90. 90Pig ArchitectureInstalled on the client node
    Converts pig program into MapReduce
  91. 91Pig Advantages95% reduction of code
    less dev time
    less execution time
    Easy to learn
  92. 92SAS InterfacesFILENAME - Read and write HDFS data
    PROC HADOOP - Between SAS and Hadoop
    SQL Pass-Through - HiveQL Queries
    LIBNAME - Access Hive Tables
  93. 93SASHas A data loader
    Better UI
  94. 94FILENAMEUpload local data using Hadoop
    Read Data from Hadoop using DATA step
  95. 95PROC HADOOPHDFS Commands
    Mapreduce Programs
    Pig Language Code
  96. 96PROC SQOOPSqoop Commands
  97. 97Pass-Through Statementproc sql;
    connect to hadoop ();
    select * from connection to hadoop
    (select employee_name,salary
    from salesstaff
    where salary > 50000);
    disconnect from hadoop;
    quit;
  98. 98DS2New SAS Programming Language
    SAS DATA
    Automatic and User-defined method
    Stored in packages
  99. 99When to use DS2Threaded Processing
  100. 100In-Memory ComputingFaster than batch
    Cost getting better
    Apache Spark
    Caching frequently used data
  101. 101Columnar DatabaseCan retrieve just the data for the required attribute
    Maybe improved data compression
  102. 102Batch ProcessingStatic Data
    Analysis has access to all data
    Complex Queries
    High Latencies
  103. 103Stream ProcessingData processed instant
    Analysis based on short time window
    Simple Queries
    Very low latency
  104. 104Min-Batch ProcessingBetween Stream and Batch Processing
  105. 105Types of stream processingEvent Stream Processing (ESP)
    Complex Event Programming (CEP)
    Live Data-Marts
  106. 106ESPStateless
    Continuous Queries
  107. 107CEPStateful
    Designed for event by event
    Aggregation of data
  108. 108Live Data-MartsRecent Development
    End-user access
  109. 109Challenges for stream processingMassive Data
    Real time Responce
    Scalability
    Continous queries
  110. 110Requirements for stream processingMoving Data
    StreamSQL
    Stream Imperfections
    Predictable Outcomes
  111. 111BI DesignBusiness Intelligence
    Work with business to identify applications
  112. 112Project RisksWrong Sponsorship
    Unrealistic expectations
  113. 113Business Requirements DefinitionTo be successful the data infrastructure must serve the needs of the business
  114. 114Chief Data OfficerControl Data
  115. 115CDO RolesDetermining Data strategies
    Overseeing implementation and monitoring of strategy
  116. 116Data StrategyA data strategy is coordinated of executing multiple data management domains to help manage revenue, cost, compliance, and risk
  117. 117DMBOKData Architecture
    Data Modelling and Design
    Data storage and operations
    Data Security
    Data Integration and Inexorability
    Documents and Content
    Reference and Master Data
    Data Warehousing and business Intelligence
    Meta-Data
    Data Quality
  118. 118Data GovernancePlanning
    Oversight and control
    over the management of data
    Multiple levels of Governance
  119. 119Data ArchitectureOverall structure of data
  120. 120Data Modeling and DesignAnalysis
    Design
    building
    Testing
    Maintenance
  121. 121Data Storage and OperationsStructured physical data
  122. 122Data SecurityPrivacy
    Confidentiality
  123. 123Data integration and interoperabilityacquisition, extraction, transformation, movement, delivery,
  124. 124Data Management MaturityData Management Strategy
    Data Quality
    Data Operations
    Platform and Architecture
    Data Governance
  125. 125Data Strategy DriversProblems
    Environment
    Infrstructure
    Target
  126. 126—

Information

Description
Flashcards on Big Data Management , created by Ash Z on 12/11/2019.
Created by
Ash Z · Nov 2019
Updated
Nov 13, 2019
Views
6
Rating

Feedback

What is it about?

What do you think of the new look?

How do you like it?