Efficient storage of high throughput DNA sequencing data using reference-based compression.

By Markus Hsi-Yang Fritz, Rasko Leinonen, Guy Cochrane, Ewan Birney. Genome Research, 2011

447 citations recorded by the source corpus.

Data storage costs have become an appreciable proportion of total cost in the creation and analysis of DNA sequence data. Of particular concern is that the rate of increase in DNA sequencing is significantly outstripping the rate of increase in disk storage capacity. In this paper we present a new reference-based compression method that efficiently compresses DNA sequences for storage. Our approach works for resequencing experiments that target well-studied genomes. We align new sequences to a reference genome and then encode the differences between the new sequence and the reference genome for storage. Our compression method is most efficient when we allow controlled loss of data in the saving of quality information and unaligned sequences. With this new compression method we observe exponential efficiency gains as read lengths increase, and the magnitude of this efficiency gain can be controlled by changing the amount of quality information stored. Our compression method is tunable: The storage of quality scores and unaligned sequences may be adjusted for different experiments to conserve information or to minimize storage costs, and provides one opportunity to address the threat that increasing DNA sequence volumes will overcome our ability to store the sequences.

Subfields: Artificial Intelligence, Computer Networks and Communications, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/341535366d39fee18d31e09b768da469f7a2dfac

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Molecular Biology

Citations in the corpus

The real cost of sequencing: scaling computation to keep pace with data generation (2016, methodology): “A widely accepted compression method, CRAM (compression algorithm), is able to shrink BAM files by ~30 % without any data loss ( ‘ losslessly ’ ) and by more if compression is allowed to lose some information ( ‘ lossy ’ ), typically in the quality scores [41].”
Twelve years of SAMtools and BCFtools (2020, methodology): “0 release introduced support for the better-compressed CRAM format [8].”
The Harvest suite for rapid core-genome alignment and visualization of thousands of intraspecific microbial genomes (2014, methodology): “This equates to roughly a 100,000X (lossy) compression factor from raw reads to compressed archive, requiring only 0.08 bits per base to store the full core-genome alignment plus other related information, which is competitive with related techniques like CRAM [102].”
Countering GATTACA: efficient and secure testing of fully-sequenced human genomes (2011, background): “correctness of the reference genome that is only used for compression [39].”
Big Data: Astronomical or Genomical? (2015, background): “These needs can be diminished with effective data compression [27], but decompression times and fidelity are a major concern in compressive genomics [28].”
Ensembl Genomes 2018: an integrated omics infrastructure for non-vertebrate species (2017, methodology): “These alignments are stored in Compressed Read Alignment Map (CRAM) format (22) and are resubmitted to the ENA for persistent archiving.”
HTSlib: C library for reading/writing high-throughput sequencing data (2021, unknown): “For example, HTSJDK is the Java equivalent [7] and is used extensively in Java applications; Sambamba [8] is written in the D language and focuses primarily on efficient multi-threaded work; Scramble [9] has BAM and SAM capability and is the primary source for experimental CRAM [10, 11] development;”
The sequence read archive: explosive growth of sequencing data (2011, methodology): “Among these possibilities, SRA is exploring approaches based on reference alignment and compression of reads, and on the preservation of only the most valuable base quality information (8), and is also actively participating in experiments assessing the effect of quality score quantization.”