Analysis of Dimensionality Reduction Techniques on Big Data

By G. T. Reddy, M. P. K. Reddy, K. Lakshmanna, Rajesh Kaluri, D. Rajput, Gautam Srivastava, Ieee Thar Baker Senior Member. IEEE Access, 2020

703 citations recorded by the source corpus.

Due to digitization, a huge volume of data is being generated across several sectors such as healthcare, production, sales, IoT devices, Web, organizations. Machine learning algorithms are used to uncover patterns among the attributes of this data. Hence, they can be used to make predictions that can be used by medical practitioners and people at managerial level to make executive decisions. Not all the attributes in the datasets generated are important for training the machine learning algorithms. Some attributes might be irrelevant and some might not affect the outcome of the prediction. Ignoring or removing these irrelevant or less important attributes reduces the burden on machine learning algorithms. In this work two of the prominent dimensionality reduction techniques, Linear Discriminant Analysis (LDA) and Principal Component Analysis (PCA) are investigated on four popular Machine Learning (ML) algorithms, Decision Tree Induction, Support Vector Machine (SVM), Naive Bayes Classifier and Random Forest Classifier using publicly available Cardiotocography (CTG) dataset from University of California and Irvine Machine Learning Repository. The experimentation results prove that PCA outperforms LDA in all the measures. Also, the performance of the classifiers, Decision Tree, Random Forest examined is not affected much by using PCA and LDA.To further analyze the performance of PCA and LDA the eperimentation is carried out on Diabetic Retinopathy (DR) and Intrusion Detection System (IDS) datasets. Experimentation results prove that ML algorithms with PCA produce better results when dimensionality of the datasets is high. When dimensionality of datasets is low it is observed that the ML algorithms without dimensionality reduction yields better results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Health Information Management

View Paper: https://www.semanticscholar.org/paper/b34fc78de28be598e21118d7cb9d84d63374addc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Health Information Management

Citations in the corpus

Metaverse for Healthcare: A Survey on Potential Applications, Challenges and Future Directions (2022, background): “storage, and big data analysis [50], [51], [52].”
A Survey on Blockchain for Big Data: Approaches, Opportunities, and Future Directions (2020, methodology): “ata analytics, which uses data mining and statistical models to analyze massive datasets, is playing a major role in helping the industries to gain insights into the purchase patterns of the customers[57]. However, the tremendous growth in the big data presented its own challenges. Some of the key c”
Harnessing the power of synthetic data in healthcare: innovation, application, and privacy (2023, background): “Techniques such as distribution matching 50 , correlation analysis 51 , and dimensionality reduction 52 can capture the intricate correlations and patterns inherent in real-world medical scenarios, enhancing the data ’ s representativeness.”
Machine Learning in Aerodynamic Shape Optimization (2022, methodology): “Unsupervised learning has prevailed in various important problems, such as anomaly detection [173, 174] and novelty detection [175, 176], clustering [177, 178], and dimensionality reduction[179, 180].”