DATA CLEANING

DATA CLEANING

1. Why need to clean data?

   • Because Data quality may be compromised by:
     • incomplete/Missing datas (e.g., data with missing values)
       • What cause it?
         • OR Because some data were not recorded at the source
         • Because of a failure of the system to capture such records.
       • How to correct it?
         • Mthd 1: Eliminate records with missing values
           • When not to use?
             • if dataset are small and there are too many records with missing values, omitting such records may lead to a substantial loss of information, and result in poor mining outcome.
         • Mthd 2: Ask domain expert
           • To obtain recommendations of possible values to substitute for the missing values.
             • Why may not be good mthd?
               • This approach is subjective and arbitrary
         • 3. To encode and identify missing values
           • Why it is gd mthd?
             • The entire records are still retained in the dataset
           • When to use?
             • When you know that the missing values carry certain meaning.
               • EG: an “allergy” field with no values could means “no known drug allergy”.
                 • Thus, make sense to encode the field rather than to leave it blank.
         • 4. Replace missing values with some values
           • CONTINUOUS variables
             • EG of values
               • The MEAN of a continuous attribute for the rest of the non-missing observations.
           • CATEGORICAL variables
             • can use MODE
     • Noisy data
     • Or Inconsistent data