Challenges and Opportunities with Big Data

By Alexandros Labrinidis, Hosagrahar Visvesvaraya Jagadish. 2017

924 citations recorded by the source corpus.

- Data is exploding at a rapid rate. Big data is the term for data sets so large and complicated that it becomes difficult to process using traditional data management tools or processing applications. Heterogeneity, scale, timeliness, complexity, and privacy problems with Big Data impede progress at all phases of the pipeline that can create value from data. The problems start right away during data acquisition, when the data tsunami requires us to make decisions, currently in an ad hoc manner, about what data to keep and what to discard, and how to store what we keep reliably with the right metadata. Much data today is not natively in structured format, tweets and blogs are weakly structured pieces of text, while images and video are structured for storage and display, but not for semantic content and search. Transforming such content into a structured format for later analysis is a major challenge. The value of data explodes when it can be linked with other data, thus data integration is a major creator of value. Since most data is directly generated in digital format today, we have the opportunity and the challenge both to influence the creation to facilitate later linkage and to automatically link previously created data. Data analysis, organization, retrieval, and modeling are other foundational challenges. Data analysis is a clear bottleneck in many applications, both due to lack of scalability of the underlying algorithms and due to the complexity of the data that needs to be analyzed. Finally, presentation of the results and its interpretation by non-technical domain experts is crucial to extracting actionable knowledge.

Subfields: Health Information Management, Information Systems and Management, Management Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Health Information Management

In Information Systems and Management

In Management Information Systems

Citations in the corpus

A Survey on Cleaning Dirty Data Using Machine Learning Paradigm for Big Data Analytics (2018, background): “Semi structured and unstructured data remain mostly unfamiliar for Data quality problems [28, 33].”
Transforming the communication between citizens and government through AI-guided chatbots (2019, background): “As reported in (Computing Research Association, 2012), such approaches are not all computational; rather, they are designed explicitly to have a human in the loop.”
Critical analysis of Big Data challenges and analytical methods (2017, methodology): “According to Labrinidis and Jagadish (2012) developing and maintaining this extraction method is a continuous challenge.”
Profiling relational data: a survey (2015, background): “With adequate metadata, there is some hope, but even so, challenges will remain […] [7].””
Smart Contract Development: Challenges and Opportunities (2021, background): “Labrinidis and Jagadish discussed some controversies and myths surrounding big data and summarized some challenges and opportunities with big data [82].”
Computational approaches in target identification and drug discovery (2016, background): “To address the above-mentioned interplay in high-throughput formats, we feel that information technologies and chemoinformatic tools need to be employed on the basis of a synergy that even extends to artificial and human intelligence interplay - humans can detect patterns, which computer algorithms ”
Machine Learning With Big Data: Challenges and Approaches (2017, methodology): “3 shows a representation of the pipeline based on the work of Labrinidis and Jagadish [76], along with the three types of manipulations to be discussed in this section: data manipulations, processing manipulations, and algorithm manipulations.”
Big Data and cloud computing: innovation opportunities and challenges (2017, background): “Processing large volumes of data requires dedicated computing resources and this is partially handled by the increasing speed of CPU, network and storage (Bertino et al. 2011).”
A Survey on Deep Learning in Big Data (2017, background): “Labrinidis and Jagadish in [7] described some challenges and research problems with respect to Scalability, Heterogeneity aspects of Big Data management.”