Computational solutions to large-scale data management and analysis

By Eric E. Schadt, Michael D. Linderman, J. Sorenson, Lawrence Lee, Garry P. Nolan. Nature reviews genetics, 2010

646 citations recorded by the source corpus.

Today we can generate hundreds of gigabases of DNA and RNA sequencing data in a week for less than US$5,000. The astonishing rate of data generation by these low-cost, high-throughput technologies in genomics is being matched by that of other technologies, such as real-time imaging and mass spectrometry-based flow cytometry. Success in the life sciences will depend on our ability to properly interpret the large-scale, high-dimensional data sets that are generated by these technologies, which in turn requires us to adopt advances in informatics. Here we discuss how we can master the different types of computational environments that exist — such as cloud and heterogeneous computing — to successfully tackle our big data problems.

Subfields: Information Systems and Management, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/c178754399c3a04a32d6151c2f0d7ddcf13d7d7a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems and Management

In Molecular Biology

Citations in the corpus

Data-intensive applications, challenges, techniques and technologies: A survey on Big Data (2014, methodology): “Furthermore, there are privacy concerns relating to the hosting of data sets on publicly accessible servers, as well as issues related to storage of data from human studies [159].”
Toward enhanced understanding and projections of climate extremes using physics-guided data mining techniques (2014, background): “Today, however, the major technical barriers for mining massive data lie in scalable data-intensive analysis capabilities, where fast storage and scalable input/output are major concerns (Schadt et al., 2010; Trelles et al., 2011), as well as mathematical and algorithmic capabilities.”
From RNA-seq reads to differential expression results (2010, background): “However, the cost of infor matics to house, process and analyze the data is sub stantial [89].”
A survey of tools for variant analysis of next-generation genome sequencing data (2013, neutral): “tional analysis of the experimental data [23].”
'Big data', Hadoop and cloud computing in genomics (2013, background): “Jeff Hammerbacher, Cloudera’s chief scientist and co-founder, has committed to dedicating 25% of his time to the initiative with Mount Sinai also heavily committed to the project with Eric Schadt (a visionary in the use of computational biology in genomics [52]) coordinating the initiative.”
A window into third-generation sequencing. (2010, methodology): “For biology to accurately model biological systems, advances are needed in data transfer, access control and management, standardization of data formats and integration of data from multiple different dimensions (57).”
Dissecting the Phenotypic Components of Crop Plant Growth and Drought Responses Based on High-Throughput Image Analysis[W][OPEN] (2014, background): “Thus, we are now facing the “big data problems” (Schadt et al., 2010) brought about by such real-time imaging technologies in the phenomics era.”
To Milliseconds and Beyond: Challenges in the Simulation of Protein Folding (2012, background): “As sampling becomes less of an issue, a new challenge in folding simulations raises its head — given the massive amount of data extensive sampling provides, how does one make sense of it all? MD simulations are a high-dimensional time series, and therefore present a ‘Big Data’ challenge [28,29].”
Fractal MapReduce decomposition of sequence alignment (2012, methodology): “Accordingly, the use the MapReduce functional pattern now underlie many of the leading genomic analysis packages such as GATK [12] and CloudBurst [13] and is the key cloud computing abstraction for large scale data management and analysis [11,14].”
How ‘Big Data’ Can Make Big Impact: Findings from a Systematic Review and a Longitudinal Case Study (2015, neutral): “(Gorton, et al., 2008), (Field, et al., 2009), (Jacobs, 2009), (Bughin, et al., 2010), (Schadt, et al., 2010), (Acker, et al., 2011), (Alexander, et al., 2011), (Bradbury, 2011), (Brown, et al., 2011), (Bughin, et al., 2011), (Long & Siemens, 2011), (Sobek, et al., 2011), (Anderson & Blanke, 2012),…”