Flashcards
Big Data Rev
Flashcards do conjunto (53)
- 1Database Life Cycle1. Initial Study
2. Database Design
3. Implementation and Loading
4. Testing and Evaluation
5. Operating
6. Maintenance and Evolution - 2Stage 1: Conceptual Modelling- Expressed via ERD
- Sometimes non key attributes are shown
- NO FK - 3Stage 2: Logical Modelling- Now specific to an implementation model
- FK's have been added
- Expressed visually via extended Bachman diagram - 4Stage 3:Physical Modelling- Specific to relational DBMS
- MySQL can run against the database to create table structures - 5What is Data?- Basis of all information systems
- Raw Data
- Data turns to information once processed - 6What is BIG Data?- Big data is a term for data in which the data is too large to be stored or manipulated in generic data programs. Big data is generally used by companies receiving 100's of GB of data on an everyday basis.
- Volume
- Variety
- Velocity - 7Structured Data- Organised data
- Computer readable and usable
- Data can be sourced from a number of places including databases and spreadsheets
- Computers can understand it and use the data values - 8Unstructured data- Do not conform to data models and associated metadata
- Computers have a hard time reading it
- Pictures
- Videos
- Blocks of text - 9Problems with spreadsheets?- Data Redundancy
- Changing data is difficult
- Limited security - 10What is a database?- Related data sorted in a single repository
- Eliminate inconsistency
- Stores data structures and relationships
- Entities - Items we want to store (Student)
- Attributes - Piece of data we want to store (Student Name)
- Relationships (Relationships between entities - Student may be enrolled in a course - 11Data Hierarchy- Wisdom
- Knowledge
- Information
- Data - 12Types of Big DataStructured
Unstructured
Semi-Structured - 13Database Management System- Collection of Programs
- Modify
- Store
- Extract
- Database
- Small and Large Systems - 14Disadvantages of DBMS- Time consuming
- Specialised knowledge needed
- Cost may be large
- File size is large
- More difficult to recover data - 15Types of DBMS:- Oracle
- MS-Access
- SQL Server - 16Steps to developing a ERD:1. List the major entities in the system and represent them graphically with a rectangle
2. Search for relationships between the entities and represent them with a diamond
3. Add a PK for every entity
4. Model relationship connectivity between each pair of entities
5. Model relationship cardinalities between each pair of entities
6. Determine if composite entities are to be created
7. Verify the ERD - 17SQL SyntaxSELECT (Column)
FROM (Table)
WHERE (Conditions)
GROUP BY (Columns)
HAVING (Group By Conditions)
ORDER BY (Columns)
LIMIT (Limits rows returned)
DISTINCT (Removes repeating data) - 18LIMIT- LIMIT (Constrain the number of rows returned) SELECT ProductID, UnitPrice FROM product ORDER BY UnitPrice DESC LIMIT 0,2;
- 19SQL Example ExplainedSELECT * (Select all)
FROM Product (The Table)
WHERE P_CODE = 311452 (Will only display these) - 20Comparison Operators:= Equal
> Greater Then
< Less Than
>= Greater than or equal
<= Less than or equal
<> Not equal - 21Characteristics Big Data:- Volume
- Variety
- Velocity
- Veracity
- Variability
- Volatility - 22Big Data Analysis Techniques:- Text analytics / Mining
- Audio Analytics
- Video Analytics
- Social Media Analytics
- Predictive Analytics - 23Text analytics / mining- Used to answer business questions and improve day to day operations
- Text mining in a computer workshop can be used to identify component defects and inform technicians to avoid the same problem
- Identify Patterns and Trends - 24Audio Analytics- Audio to text
- Recognise certain sounds
- Could be used in police footage
- Computer can pull things out of sound - 25Video Analytics- IBM intelligent video analytics
- Sorts the videos in to data that can be used for patterns
- Can be used by police to find a suspects car
- Learns events and interests - 26Social Media Analytics- Gathering data from conversations had on social networks
- Business Decisions
- Customer Centrality
- Facebook, Twitter etc. - 27Predictive Analytics- Extracting data from data sets to determine patterns and predicts future outcomes and trends
- Does not tell what will happen in the future
- Weather - 28Data Warehouse- A physical repository
- Cleansed data in standardised format
- Integrated subject orientated databases
- Non-volatile - 29Data Mart- A departmental data warehouse that stores only relevant data
- Dependant data mart - A subset is created from a data warehouse
- Independent data mart - A small data warehouse designed for strategic business unit or a department - 30Data Mining- CRISP - Cross industry Standard process for data mining
- SEMMA - Sample, Explore, Modify, Model and Assess
- KDD - Knowledge Discovery Database
- Sources of data are often consolidated data warehouse
- Usually client, server or web based
- Miner is often an end user - 31CRISPStep 1. Business Understanding
Step 2. Data Understanding
Step 3. Data Preparation
Step 4. Model Building
Step 5. Testing and Evaluation
Step 6. Deployment - 32SEMMASample
Explore
Modify
Model
Assess - 33Parallel Processing- Multiple computers work on the same task
- Same operation simultaneously
- Improves performance - 34Data Mining Benefits- Customer Relationship Management
- Banking and other financial
- Retailing and Logistic
- Manufacturing and Maintenance
- Brokerage and Securities
- Insurance
- Healthcare
- Medicine - 35Data Mining Myths- Provides instant solutions
- Not viable for business applications
- Is only for larger business - 36Data Mining Problems- Selecting wrong problem
- Ignoring sponsor
- Not enough time
- Looking at wrong data - 37Decision TreeEmploys the divide and conquer method
- 38Cluster Analysis- Used for automatic identification or natural grouping
- Machine learning
- No output variable
- Segmentation - 39Hadoop- Open Source
- Data processing
- Storage
- Big Data
- Clustered System
- Batch Processing
- Processing data uses MapReduce - 40Hadoop Advantages- Scalable
- Cost Effective
- Flexible
- Fast
- Resilient to failure - 41Hadoop disadvantages- Security
- Not Fit for small data
- Stability issues
- Limitations - 42MapReduce- Map - Takes input data and processes to produce key value pairs
- Reduce - Takes the key value pairs and reduces or combines them to bring final result
- Multiple maps - Bring results to the reducer - 43Text Mining Concepts- Specific Questions
- Impossible for humans
- Filters large amount of data and extract relevant information
- Patterns and Trends
- Extra Information
- Process
- Integrity - 44Text Mining Process- Establish the Corpus
- Create the term Documents matrix
- Extract Knowledge - 45Web Mining- Largest Repository of data
- HRML, XML and Text format
- Too big
- Too Complex
Opportunities are great
Challenges are huge - 46IOT- Network enabled physical objects
- Any time connection
- Any place
- Internet of things - 47IOT Driving Forces- Sensor Technology
- Cheap Miniature computers
- Low power connectivity
- Capable Mobile Devices
- Power of the cloud - 48IOT Challenges- Security
- Legal
- Social
- Market
- Technical Control - 49Big Data Governance- Formulates policy
- Privacy
- Monetisation
- Clarity of data access
- Integration
- Usage
- Management - 50Many To ManyResolved By Bridging
- 51NullsRepresent a missing value
- 52A primary key that consists of more than one field is called a ____ key.Composite
- 53A table is perceived as a ______.two-dimensional structure
Informações
- Descrição
- FlashCards sobre Big Data Rev, criado por Ash Z em 16-11-2018.
- Criado por
- Ash Z
- Visualizações
- 31
- Avaliação