Data Cleaning in Data Mining
Quality of your data is critical in getting to
final analysis.Any data which tend to be incomplete, noisy and inconsistent can
effect your result.
Data cleaning in data mining is the process of
detecting and removing corrupt or inaccurate records from a record set, table
or database.
Some data cleaning methods :-
1 You can ignore the tuple.This is done when
class label is missing.This method is not very effective , unless the tuple
contains several attributes with missing values.
2 You can fill in the missing value
manually.This approach is effective on small data set with some missing values.
3 You can replace all missing attribute values
with global constant, such as a label like “Unknown” or minus infinity.
4 You can use the attribute mean to fill in
the missing value.For example customer average income is 25000 then you can use
this value to replace missing value for income.
5 Use the most probable value to fill in the
missing value.
Noisy Data
Noise is a random error or variance in a
measured variable. Noisy Data may be due to faulty data collection instruments,
data entry problems and technology limitation.
How to Handle Noisy Data?
Binning:
Binning methods sorted data value by
consulting its “neighbor- hood,” that is, the values around it.The sorted
values are distributed into a number of “buckets,” or bins.
For example
Price = 4, 8, 15, 21, 21, 24, 25, 28, 34
Partition into (equal-frequency) bins:
Bin a: 4, 8, 15
Bin b: 21, 21, 24
Bin c: 25, 28, 34
In this example, the data for price are first
sorted and then partitioned into equal-frequency bins of size 3.
Smoothing by bin means:
Bin a: 9, 9, 9
Bin b: 22, 22, 22
Bin c: 29, 29, 29
In smoothing by bin means, each value in a bin
is replaced by the mean value of the bin.
Smoothing by bin boundaries:
Bin a: 4, 4, 15
Bin b: 21, 21, 24
Bin c: 25, 25, 34
In smoothing by bin boundaries, each bin value
is replaced by the closest boundary value.
Regression
Data can be smoothed by fitting the data into
a regression functions.
Clustering:
Outliers may be detected by clustering,where
similar values are organized into groups, or “clusters.Values that fall outside
of the set of clusters may be considered outliers.
Clustering