Karteikarten
Data Pre-Processing
Karten in diesem Stapel (22)
- 1Pre-Processing Tasks* Data Cleaning
* Data Integration
*Data Transformation
*Data Reduction - 2Why Pre-Process Data?Data is mostly
-Noisy
-Inconsistent
-Incomplete
-Duplicate
Preprocessing removes the inconsistency in the dataset as well as any noise. The data-warehouse needs information which is consistent. - 3What is noisy Data?Noisy data contains random errors, or outliers that is dissimilar to the dataset. Possible faults include human error and data transmission.
- 4Incomplete Data- Missing attributes values within the dataset
- Lacking certain attributes of interest
- Only contains aggregate data - 5Task 1: Data CleaningAll noise and inconsistency within the dataset is removed.
Involving:
-Missing values
-Smoothing noisy data
-Identifying and removing outliers
-Resolving inconsistencies - 6Handling missing data-Ignore tuples with missing values
-Manually enter missing values
- Replace the missing values with a constant
-Predict the missing value using a model - 7Ignore the truples with missing valuesDone when the class label is missing (classification)
- 8Advantage of Ignoring the truples with missing values-Simple and effective if number of records missing are small.
- 9Disadvantage of Ignoring truples with missing values-Result in the dataset being bias due to the large amount of missing values.
- 10Manually filling in the missing values* Time-consuming
*Error proning
*Not feasible for large dataset with missing values. - 11Replacing the missing value with a constant- New class label "Unknown"
- The attribute mean for numeric values, or attribute mode for categorical values - 12Advantage of replacing the missing value with a constant- Simple
- Smaller bias than throwing away the records with missing values. - 13Disadvantage of replacing the missing value with a constant-Noise in the dataset will be introduced.
- 14Predict the missing value using a modelBuild a model based on other attributes to predict missing ones with using Bayesian or decision tree.
Advantage: - Most effective
Disadvantage: - Complex and Computationally expensive - 15How to handle noisy data?Binning: Smooth a sorted data value using the values around it.
Regression: Smooth by fitting the data to a regression function
Clustering: Detect and remove outliers
Combined computer and human inspection: Detect suspicious values and check by human - 16Task 2: Data IntegrationData integration combines data from multiple sources into a coherent data store.
Redundancies and inconsistencies should be removed to improve mining speed and quality. - 17Problems with Integration* Schema Integration; integrating metadata from different sources.
* Entity identification problem; identifying real world problems from multiple data sources.
* Detecting and resolving data value conflicts.
* Redundant data: -Object identification and derivable data. - 18Task 3: Data TransformationData is transformed into appropriate forms for mining..
Such as
Smoothing: Removing the noise from data
Aggregation: Summarization of data, data cube and construction.
Normalisation: Scaling data to fall within a small specific range.
Changing data types: numerical to categorical.
Attribute construction - 19Normalisation* Min-max normalisation
* Z-score normalisation - 20Min-Max NormalisationMaps a dataset to a new range [new Minumum and new Maximum]
Not suitable for outliers in a dataset or if the min and max is unknown. - 21Z-score NormalisationNormalise dataset using mean and standard deviation to map a new dataset with 0 and 1.
- 22Task 4: Data Reduction-Reduces data volume to maintain the integrity of original data.
-Strategies include: Data aggregation and attribute selection. Data Sampling and Discretization.
Informationen
- Beschreibung
- Bachelors Degree Data Mining Karteikarten am Data Pre-Processing, erstellt von Jason Dsilva am 01/05/2016.
- Erstellt von
- Jason Dsilva
- Aufrufe
- 0
- Bewertung