Machine Learning With Big Data: Challenges and Approaches

By Alexandra L’Heureux, Katarina Grolinger, Hany F. ElYamany, Miriam A. M. Capretz. 2017

835 citations recorded by the source corpus.

The Big Data revolution promises to transform how we live, work, and think by enabling process optimization, empowering insight discovery and improving decision making. The realization of this grand potential relies on the ability to extract value from such massive data through data analytics; machine learning is at its core because of its ability to learn from data and provide data driven insights, decisions, and predictions. However, traditional machine learning approaches were developed in a different era, and thus are based upon multiple assumptions, such as the data set fitting entirely into memory, what unfortunately no longer holds true in this new context. These broken assumptions, together with the Big Data characteristics, are creating obstacles for the traditional techniques. Consequently, this paper compiles, summarizes, and organizes machine learning challenges with Big Data. In contrast to other research that discusses challenges, this work highlights the cause–effect relationship by organizing challenges according to Big Data Vs or dimensions that instigated the issue: volume, velocity, variety, or veracity. Moreover, emerging machine learning approaches and techniques are discussed in terms of how they are capable of handling the various challenges with the ultimate objective of helping practitioners select appropriate solutions for their use cases. Finally, a matrix relating the challenges and approaches is presented. Through this process, this paper provides a perspective on the domain, identifies research gaps and opportunities, and provides a strong foundation and encouragement for further research in the field of machine learning with Big Data.

Subfields: Artificial Intelligence, Management Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Challenges and Future Directions of Big Data and Artificial Intelligence in Education (2020, background): “Moreover, it is also shifting from a single domain (e.g., domain expertise, or expert systems) to a cross-disciplinary approach through collaboration (Spikol et al., 2018; Krouska et al., 2019) and domain transfers (L’heureux et al., 2017).”
Machine Learning in IoT Security: Current Solutions and Future Challenges (2019, background): “Also, predictive ability of an algorithm decreases with the increase in the dimensionality of data [102].”
Truth from the machine: artificial intelligence and the materialization of identity (2021, positive): “Consequently, algorithmi-cally revealed correlations way may not actually be accurate (L ’ heureux et al. 2017).”
A Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View (2018, background): “worth noticing that new intelligent systems mainly focus on learning from massive amounts of data with the goals of high efficiency, minimum computational cost and considerable predictive or classification accuracy [3], [5].”
Interpreting Interpretability: Understanding Data Scientists' Use of Interpretability Tools for Machine Learning (2020, result): “With only six interviews, we cannot claim that this list is by any means exhaustive, but it is in line with other analyses of ML pitfalls [42].”