Challenges of Big Data Analysis

By Jianqing Fan, Fang Han, Han Liu. National Science Review, 2013

1,427 citations recorded by the source corpus.

Big Data bring new opportunities to modern society and challenges to data scientists. On one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data introduce unique computational and statistical challenges, including scalability and storage bottleneck, noise accumulation, spurious correlation, incidental endogeneity, and measurement errors. These challenges are distinguished and require new computational and statistical paradigm. This article gives overviews on the salient features of Big Data and how these features impact on paradigm change on statistical and computational methods as well as computing architectures. We also provide various new perspectives on the Big Data analysis and computation. In particular, we emphasize on the viability of the sparsest solution in high-confidence set and point out that exogeneous assumptions in most statistical methods for Big Data can not be validated due to incidental endogeneity. They can lead to wrong statistical inferences and consequently wrong scientific conclusions.

Subfields: Artificial Intelligence, Computer Networks and Communications, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/547df0f649d5b8fcb64ac06611e5bbe7b14518f7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Statistics and Probability

Citations in the corpus

Big Data for Remote Sensing: Challenges and Opportunities (2016, background): “Due to the aforementioned heterogeneity and high dimensionality of big data in remote sensing, we also face important computational and statistical challenges related to processing scalability, noise accumulation, spurious correlation, incidental endogeneity, and measurement errors [19], [20].”
The impact of big data on world-class sustainable manufacturing (2015, methodology): “‘Big Data and Analytics’ (BDA) has attracted the attention of scholars from every field including, genomics, neuroscience, economics and finance (Fan et al. 2014).”
Artificial Intelligence and Management: The Automation–Augmentation Paradox (2020, background): “Overfitting is a key risk in this context, which means that a machine may learn a complete model that also explains the errors, consequently failing to generalize appropriately beyond its training data (Fan, Han, & Liu, 2014).”
An Overview on the Estimation of Large Covariance and Precision Matrices (2015, neutral): “Before introducing the estimator of Σ in the case of latent factors, we first elucidate why PCA can be used for the factor analysis when the number of variables is large.”
Assessing the Methods, Tools, and Statistical Approaches in Google Trends Research: Systematic Review (2018, background): “Healthcare is one of the fields in which big data are widely applied [24,25], with the number of publications in this field showing a high increase [26].”
Big Data: Deep Learning for financial sentiment analysis (2018, background): “Each data source can collect information without any centralized control [62].”
Majorization-Minimization Algorithms in Signal Processing, Communications, and Machine Learning (2017, background): “From an optimization perspective, these can result in large-scale problems due to the tremendous amount of data and variables, which cause challenges to traditional algorithms [1].”
A survey on missing data in machine learning (2021, background): “However, it is important to note that the only suitable solution comes down to a virtuous design and good analysis [16].”
Next-Generation Big Data Analytics: State of the Art, Challenges, and Future Research Topics (2017, background): “The current research outcomes have indicated that data are no longer just data [57], [67], [68].”
Random Forests for Big Data (2015, methodology): “Recently some statistical methods have been adapted to process Big Data, like linear regression models, clustering methods and bootstrapping schemes.”