Chap. 3 - Data Preprocessing

Chap. 3 - Data Preprocessing

1. Well- accepted multidimensional view

   • Accuracy
     • Inaccurate or noisy (containing error or outlier values that deviate from the expected
   • Completeness
     • Incomplete - attribute of interest may not always be available
   • Consistency
     • Incorrect data may result from inconsistencies in naming conventions or data code used or inconsistent formats for input field.
   • Timeliness
     • Data is not updated in a timely fashion has a negative impact on data quality
   • Believability
     • Reflects how much data are trusted by user
   • Interpretability
     • Reflects how easy the data are understood

2. Data Preprocessing

   • Data cleaning
     • Handling data that are incomplete, noisy and inconsistent
       • Incomplete - lacking attribute values, lacking certain attribute of interest or contain only aggregate data (missing data)
         • handling missing data by fill in manually or automatically
       • Noise - contain noise, error or outlier
         • method of handling noisy data
           • binning
           • regression
             • finding the best line to fit two attribute so that one attribute can be used to predict one another
           • cluster analysis
             • Unsupervised - outlier
       • Inconsistent - contain discrepancies in codes or name
   • Data integration
     • Combines data from multiple sources into coherent data store
       • Schema integration - integrate metadata from different sources
       • Detecting and resolving data value conflict
       • Redundancy - redundant attributes may be able to be detected by correlation analysis and covariance analysis
   • Data transformation
     • Data are transformed into forms appropriate for mining
       • Smoothing - remove noise from data
       • Aggregation - summarization, data cube construction
       • Generalization - concept hierarchy climbing
       • Normalization - scaled to fall within a small, specific range
         • MINMAX normalization
         • z-score normalization/ z = X - μ / σ
         • decimal scaling
       • Attribute/ feature construction
   • Data reduction
     • To obtain a reduced representation of the data set that is much smaller in volume but closely maintains the integrity of original data.
       • CONTINUE - 3.1 (data reduction)

Images and files in the map