Big Data: Astronomical or Genomical?

By Z. Stephens, Sk Lee, F. Faghri, Roy H. Campbell, Chengxiang Zhai, Miles Efron, R. Iyer, Michael C. Schatz, S. Sinha, Gene E. Robinson. 2015

1,313 citations recorded by the source corpus.

Genomics is a Big Data science and is going to get much bigger, very soon, but it is not known whether the needs of genomics will exceed other Big Data domains. Projecting to the year 2025, we compared genomics with three other major generators of Big Data: astronomy, YouTube, and Twitter. Our estimates show that genomics is a “four-headed beast”—it is either on par with or the most demanding of the domains analyzed here in terms of data acquisition, storage, distribution, and analysis. We discuss aspects of new technologies that will need to be developed to rise up and meet the computational challenges that genomics poses for the near future. Now is the time for concerted, community-wide planning for the “genomical” challenges of the next decade.

Subfields: Genetics, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Genetics

In Molecular Biology

Citations in the corpus

Opportunities and obstacles for deep learning in biology and medicine (2017, background): “comparison of genomics with social media, online videos, and other dataintensive disciplines suggests that genomics alone will equal or surpass other fields in data generation and analysis within the next decade [1].”
A Berkeley View of Systems Challenges for AI (2017, background): “According to a recent study [100], by 2025, we will need a three-to-four orders of magnitude improvement in compute throughput to process the aggregate output of all genome sequencers in the world. would require computation resources to at least double every year.”
Comparison of Strategies for Scalable Causal Discovery of Latent Variable Models from Mixed Data (2018, background): “Thus, in order for causal inference algorithms to be practical, they need to be able to process big data efficiently [28].”
TYGS and LPSN: a database tandem for fast and reliable genome-based classification and nomenclature of prokaryotes (2021, neutral): “The everincreasing growth of genome data (16) demands technical”
Feature selection methods for big data bioinformatics: A survey from the search perspective. (2016, background): “[11] Z.D. Stephens, S.Y. Lee, F. Faghri, R.H. Campbell, C. Zhai, M.J. Efron, R. Iyer, M.C. Schatz, S. Sinha, G.E. Robinson, Big data: astronomical or genomical?, PLoS Biol 13 (7) (2015) e1002195, http://dx.doi.org/10.1371/journal.pbio.1002195.”
Evaluation of template-based modeling in CASP13 (2019, background): “contrast, genome databases are currently doubling about once every 7 months.(3) A large fraction of proteins lacking an experimental structure will”
Searching ECOD for homologous domains by sequence and structure (2018, methodology): “Advances in genome sequencing, along with techniques such as cryo-electron microscopy, have dramatically increased the amount of available protein sequence and structure data (Bai, McMullan, & Scheres, 2015; Stephens et al. 2015).”
The SIB Swiss Institute of Bioinformatics’ resources: focus on curated databases (2015, background): “As the data deluge from genomics may well outgrow the amount of data produced in astronomy (1,2), the importance of bioinformatics is expected to increase even further in the coming years.”
Will we run out of data? Limits of LLM scaling based on human-generated data (2022, background): “For illustration, (Stephens et al., 2015) forecasts growth rates of between 2-40 million terabytes of genomics data every year by 2025.”
Efficient Coalescent Simulation and Genealogical Analysis for Large Sample Sizes (2015, background): “…unprecedented scale (Genome of the Netherlands Consortium et al., 2014; UK10K Consortium et al., 2015; 1000 Genomes Project Consortium et al., 2015; Gudbjartsson et al., 2015; Eisenstein, 2015; Stephens et al., 2015), and the storage and analysis of these data pose serious computational challenges.”