Data Preprocessing

Data Preprocessing

1. Data Cleaning

   •          Data comes from many source as internal and external.            Process of dealing with duplicate data issues.            
   • Missing Values
     •            Data is not always available.
   • Noisy Data
     •              Noise is a random error or variance in a measured variable.
   • Inconsistent Data
     •             It use domain knowledge or expert decision.
   • Unified date format
     •             We want to transform all dates to the same format internally.             Frequently, just the year (YYYY) is sufficient

2. Data Integration

   •             It is likely that your data analysis task will involve data integration, which combines data from multiple sources into a coherent data store as in data warehouse.
   • Redundant Data
     •              It  may be able to be detected by correlational analysis
   • Schema Integration
     •             It integrates metadata from different sources.
   • Data Integration
     •               It combines data from multiple sources into a coherent store.

3. Data Transformation

   •            The data are transformed or consolidated into forms appropriate for mining.
   • Smoothing
     •             It works to remove noise from the data.
   • Aggregation
     •               It is used to moving up in the concept hierarchy on numeric attributes.
   • Generalization
     •                 It is used for moving up in the concept hierarchy on nominal attributes.              
   • Normalization
     •                 The attribute data are scaled so as to fall within a smaller range such as -1.0 to 1.0 or 0.0 to 1.0.
   • Attribute Construction
     •               It used for replacing or adding new attributes inferred by existing attributes.

4. Data Reduce

   •         It is a techniques that can be applied to obtain a reduced representation of the data set that is much smaller in volume.
   • Data Cube
     •              It used for applying roll-up, slice or dice operations.
   • Histogram
     •             It reduce the number of attributes by grouping them into intervals (bins).
   • Data Compression
     •             Encoding mechanism is used to reduce the data set size.
   • Clustering
     •                It is used to  grouping values in clusters.
   • Removing Irrelavant Attribute
     •             It is used for attribute selection (filtering and wrapper methods), searching the attribute space.
   • Sampling
     •              It allows a mining algorithm to run in complexity that is potentially sub-linear to the size of the data.