Wednesday, 16 October 2019

Data Mining Reduction


Data Reduction In Data Mining
A database or date warehouse may store terabytes of data.So it may take very long to perform data analysis and mining on such huge amounts of data.
Data reduction techniques can be applied to obtain a reduced representation of the data set that is much smaller in volume but still contain critical information.
Data Reduction Strategies:-
1 Data Cube Aggregation
Aggregation operations are applied to the data in the construction of a data cube.
2 Dimensionality Reduction
In dimensionality reduction redundant attributes are detected and removed which reduce the data set size.
3 Data Compression
Encoding mechanisms are used to reduce the data set size.
4 Numerosity Reduction
In numerosity reduction where the data are replaced or estimated by alternative.
5 Discretisation and concept hierarchy generation
Where raw data values for attributes are replaced by ranges or higher conceptual levels.


Data Mining Concepts


Data Cleaning in Data Mining
Quality of your data is critical in getting to final analysis.Any data which tend to be incomplete, noisy and inconsistent can effect your result.
Data cleaning in data mining is the process of detecting and removing corrupt or inaccurate records from a record set, table or database.
Some data cleaning methods :-
1 You can ignore the tuple.This is done when class label is missing.This method is not very effective , unless the tuple contains several attributes with missing values.
2 You can fill in the missing value manually.This approach is effective on small data set with some missing values.
3 You can replace all missing attribute values with global constant, such as a label like “Unknown” or minus infinity.
4 You can use the attribute mean to fill in the missing value.For example customer average income is 25000 then you can use this value to replace missing value for income.
5 Use the most probable value to fill in the missing value.
Noisy Data
Noise is a random error or variance in a measured variable. Noisy Data may be due to faulty data collection instruments, data entry problems and technology limitation.
How to Handle Noisy Data?
Binning:
Binning methods sorted data value by consulting its “neighbor- hood,” that is, the values around it.The sorted values are distributed into a number of “buckets,” or bins.
For example
Price = 4, 8, 15, 21, 21, 24, 25, 28, 34
Partition into (equal-frequency) bins:
Bin a: 4, 8, 15
Bin b: 21, 21, 24
Bin c: 25, 28, 34
In this example, the data for price are first sorted and then partitioned into equal-frequency bins of size 3.
Smoothing by bin means:
Bin a: 9, 9, 9
Bin b: 22, 22, 22
Bin c: 29, 29, 29
In smoothing by bin means, each value in a bin is replaced by the mean value of the bin.
Smoothing by bin boundaries:
Bin a: 4, 4, 15
Bin b: 21, 21, 24
Bin c: 25, 25, 34
In smoothing by bin boundaries, each bin value is replaced by the closest boundary value.
Regression
Data can be smoothed by fitting the data into a regression functions.
Clustering:
Outliers may be detected by clustering,where similar values are organized into groups, or “clusters.Values that fall outside of the set of clusters may be considered outliers.
Description: Clustering : last night study
Clustering


Data Mining Reduction

Data Reduction In Data Mining A database or date warehouse may store terabytes of data.So it may take very long to perform data analysi...