DATA PREPROCESSING

DATA PREPROCESSING

1. Why Preprocess the Data?

   • Incomplete, noisy, and inconsistent data are commonplace properties of large realworld databases and data warehouses. Incomplete data can occur for a number of reasons. Attributes of interest may not always be available, such as customer information for sales transaction data.

2. Descriptive Data Summarization

   • Measuring the Central Tendency
   • Measuring the Dispersion of Data
   • Graphic Displays of Basic Descriptive Data Summaries

3. Data Cleaning

   • Missing Values
   • Noisy Data
   • Data Cleaning as a Process
     • Missing values, noise, and inconsistencies contribute to inaccurate data. So far, we have looked at techniques for handling missing data and for smoothing data. “But data cleaning is a big job. What about data cleaning as a process? How exactly does one proceed in tackling this task? Are there any tools out there to help?”

4. Discretization and Concept Hierarchy for Numerical Data

5. Data Reduction

   • Data Cube Aggregation
     • Data reduction techniques can be applied to obtain a reduced representation of the data set that ismuch smaller in volume, yet closely maintains the integrity of the original data. That is, mining on the reduced data set should be more efficient yet produce the same (or almost the same) analytical results
   • Attribute Subset Selection
     • Attribute subset selection, where irrelevant, weakly relevant, or redundant attributes or dimensions may be detected and removed.
   • Dimensionality Reduction
     • Dimensionality reduction, where encoding mechanisms are used to reduce the data set size.
   • Numerosity Reduction
     • Numerosity reduction,where the data are replaced or estimated by alternative, smaller data representations such as parametric models (which need store only the model parameters instead of the actual data) or nonparametric methods such as clustering, sampling, and the use of histograms.

6. Data Discretization and Concept Hierarchy Generation

   • It is difficult and laborious to specify concept hierarchies for numerical attributes because of the wide diversity of possible data ranges and the frequent updates of data values. Such manual specification can also be quite arbitrary. Concept hierarchies for numerical attributes can be constructed automatically based on data discretization. We examine the following methods: binning, histogram analysis, entropy-based discretization, c2-merging, cluster analysis, and discretization by intuitive partitioning. In general, each method assumes that the values to be discretized are sorted in ascending order.
   • Concept Hierarchy Generation for Categorical Data
     • Categorical data are discrete data. Categorical attributes have a finite (but possibly large) number of distinct values, with no ordering among the values. Examples include geographic location, job category, and itemtype. There are several methods for the generation of concept hierarchies for categorical

7. Data Integration and Transformation

   • Data mining often requires data integration—the merging of data from multiple data stores. The data may also need to be transformed into forms appropriate for mining. This section describes both data integration and data transformation.
   • Data Integration
     • It is likely that your data analysis task will involve data integration, which combines data from multiple sources into a coherent data store, as in data warehousing. These sources may include multiple databases, data cubes, or flat files.
   • Data Transformation
     • Smoothing, which works to remove noise from the data. Such techniques include binning, regression, and clustering. Aggregation, where summary or aggregation operations are applied to the data. For example, the daily sales data may be aggregated so as to compute monthly and annual total amounts. This step is typically used in constructing a data cube for analysis of the data at multiple granularities. Generalization of the data, where low-level or “primitive” (raw) data are replaced by higher-level concepts through the use of concept hierarchies. For example, categorical 2.4 Data Integration and Transformation 71 attributes, like street, can be generalized to higher-level concepts, like city or country. Similarly, values for numerical attributes, like age, may be mapped to higher-level concepts, like youth, middle-aged, and senior. Normalization, where the attribute data are scaled so as to fall within a small specified range, such as