Karteikarten
Databases Topic 10
Karten in diesem Stapel (35)
- 1Introduction to Data WarehousingValue of data to an organisation and how it needs to be managed as any other valuable resource.
Focus changes to adding value to the data resource. - 2Introduction to Business Intelligence (BI)BI is about 'getting the right data, to the right people, in the right format, at the right time' for the purpose of supporting management decision making.
This decision making might require analysis of past and current activities, or prediction of things in the future. - 3Operational SystemsThese type of systems allow organizations to conduct their daily activities efficiently.
- 4BI SystemsDecisions tend to be longer-term in their focus.
- Capacity planning
- Product development
- Outlet locations
- Product promotions
Operational system tend not to be able to support these kinds. - 5Two Broad Categories of BI SystemReporting Systems: Filter, sort, simple calculations, summarize current status.
Data Mining Systems: Usually use statistical and mathematical techniques. - 6ETL: Extract, Transform, LoadProcess consisting of many sub-processes that will:
EXTRACT data from the sources
TRANSFORM the data into a format suitable for the data warehouse
LOAD the data into the data warehouse
FEED the data to the users. - 7ExtractionThe data need to be 'collected' from the various operational systems and possible external data sources.
Issues that arise here include:
Identification of sources
Data ownership - 8TransformationData are extracted from various sources, there is no guarantee that there is a common data dictionary.
Eg. Domains may differ
As such, data need to be transformed into a common format.
Big problem: "Dirty" Data. - 9"Dirty Data"Data that is dirty will exhibit features such as:
Multiple identifiers
Multiple field names
Different units
Missing values
Orphaned values - 10Source of DirtIt comes from:
Unclean legacy systems
Improper Integration
Aging of the data warehouse
Changing user requirements
Changing expectations of the data warehouse. - 11Cleaning and Transforming Dirty Data.Cleansing in the legacy environment
Cleansing at the point of integration.
Cleansing at and after the point of loading. - 12LoadThe transformed and cleaned data then need to be loaded into the data warehouse.
- 13FeedData then need to be prepared for 'feeding' to the users of the data.
- 14What is a data warehouse?Traditional definition is that it is a database that is:
Integrated
Subject-oriented
Time-variant
Non volatile - 15IntegratedCentralized, Consolidated database.
Integrates data from multiple sources.
-Data in the organizational data sources is often inconsistent.
-Integration allows a consistent view of the data to be presented to the users. - 16Subject-orientedOperational systems tend to be function or process-oriented.
DW contains data organized and summarized by topic. - 17Time-VariantOperational systems concentrate on current transactions.
DW are concerned with the flow of data through time.
When new data are uploaded to the DW, time-based suggestion are recalculated. - 18Non-volatileOnce data enter the DW they stay there.
-The data are not updated in real-time, but refreshed from operational system on a regular basis.
-DW must be able to deal with lots of data. - 19Problems with DWUnderestimation of resources for data loading.
Hidden problems with source systems.
Data required for analysis not captured.
Increased end-user demands.
Data ownership.
Complexity of integration. - 20Data MartA subset of DW that supports the requirements of a particular department or business function.
-Focus on only the requirements of one.
- Does not contain detailed operational data as per DW.
-More easily understood and navigated. - 21Reasons for creating a Data MartGive users access to the data they need to analyze most often.
Provide data in a form that matches the collective view of the data.
Cost of implementing data marts is normally less than required to establish a DW. - 22Issues with Data MartsData Mart Functionality
Data mart size
Data mart load performance
Data mart Internet
Data mart administration
User access to data in multiple data marts - 23Designing Data WarehousesStart by building one or more data marts.
Interviewing appropriate members of staff to enable the identification of a prioritized set of requirements that the DW must meet.
*Dimensionality Modelling. - 24Dimensionality Modelling (1/2)Logical design technique that aims to present data in a standard form that allows for high-performance access.
DM is composed of fact table, and dimension tables. - 25Dimensionality Modelling (2/2)Star schema is a logical structure that has a fact table in the center surrounded by dimension tables which can be denormalised.
Snowflake schema is a variant of the star schema where dimension tables do not contain denoamalized data.
Starflake schema is a hybrid stucture. - 26Fact TablesFacts are generated by events that occurred in the past.
Bulk of data in DW is in fact tables which can be extremely large.
Important to treat fact data as read-only reference data that will not change over time. - 27Dimension tablesDimension tables usually contain descriptive textual information.
Dimension attributes are used as the constraints in DW queries. - 28Advantage of DMEfficiency.
Ability to handle changing requirements.
Extensibility.
Ability to model common business situations.
Predictable query processing. - 29Reporting Systems: OnLine Analytical Processing (OLAP)OLAP report has measures and dimensions:
Measure - a data item of interest
Dimension - a characteristic of a measure
OLAP cube - a presentation of a measure with associated dimensions. - 30OLAPAn OLAP cube can have any number of axes.
OLAP allows drill-down, a further division of the data into more detail. - 31Data Mining ApplicationsData Mining applications use sophisticated statistical and mathematical techniques to find patterns and relationships that can be used to classify and predict.
- 32Unsupervised data miningStatistical techniques are used to identify groups of entities with similar characteristics.
Cluster Analysis. - 33Supervised data miningA model is developed.
Statistical techniques are used to estimate parameter values of the model.
Regression analysis. - 34Popular Data Mining TechniquesDecision Tree Analysis: Classifies entities into groups based on past history.
Logistic Regression: Produces equations that offer probabilities that certain events will occur.
Neural Networks: Complex statistical prediction techniques.
Market Basket Analysis: Determines patterns of associated buying behavior. - 35Market Basket AnalysisSupport: The probability that two items will be purchased together.
Confidence: Probability that an item will be purchased given the fact that customer has already purchased another item.
Lift: Ration of confidence to the basic probability a item will be purchased.
Informationen
- Beschreibung
- Data Warehousing
- Erstellt von
- jaydeeees
- Aufrufe
- 13
- Bewertung
Mehr zu Data Warehousing and Emerging Database Technologies