Data Preprocessing

Data Preprocessing

1. Data Cleaning

   • Missing Values
     • Ignore the tuple
       • Ignore the tuple : This is usually done when the class label is missing
     • Fill in the missing value manually
       • Fill in the missing approach is time-consuming and may not be feasible given a large data set with many missing values.
     • Use a global constant to fill in the missing value
       • Replace all missing attribute values by the same constant
     • Use the attribute mean to fill in the missing value
     • Use the attribute mean for all samples belonging to the same class as the given tuple
     • Use the most probable value to fill in the missing value
   • Noisy Data
     • Noise is a random error or variance in a measured variable
     • Binning
       • Binning methods smooth a sorted data value by consulting its “neighborhood,”
     • Regression
       • Data can be smoothed by fitting the data to a function, such as with regression
       • Linear Regression
         • Linear regression involves finding the “best” line to fit two attributes (or variables), so that one attribute can be used to predict the other
       • Multiple Linear Regression
         • Multiple linear regression is an extension of linear regression, where more than two attributes are involved and the data are fit to a multidimensional surface.
     • Clustering
       • Outliers may be detected by clustering, where similar values are organized into groups, or “clusters.”
   • Cleaning as a Process
   • Fill in missing values & correct inconsistencies in the data

2. Data Integration

   • Entity Identification Problem
   • Redundancy & Correlation Analysis
     • X^2 Correlation Test for Norminal Data
     • Correlation Coefficient for Numeric Data
     • Covariance of Numeric Data
   • Tuple Duplication
   • Data Value Conflict Detection & Resolution

3. Data Transformation

   • Smoothing
     • Smoothing, which works to remove noise from the data.
   • Aggregation
     • Aggregation, where summary or aggregation operations are applied to the data.
   • Generalization
     • Generalization of the data, where low-level or “primitive” (raw) data are replaced by higher-level concepts through the use of concept hierarchies.
   • Normalization
     • Normalization, where the attribute data are scaled so as to fall within a small specified range, such as
   • Attribute construction
     • Attribute construction (or feature construction),where new attributes are constructed and added from the given set of attributes to help the mining process.

4. Data Reduction

   • Data Cube Aggregation
   • Attribute Subset Selection
     • Stepwise forward selection. Stepwise backward elimination. Combination of forward selection and backward elimination. Decision Tree induction.
   • Dimensionality Reduction
   • Numerosity Reduction

5. Data Discretization & Concept Hierarchy Generation

   • Discretization and Concept Hierarchy Generation for Numerical Data
   • Concept Hierarchy Generation for Categorical Data