Fast and Efficient Compression of Floating-Point Data
By Peter Lindstrom, Martin Isenburg. IEEE Transactions on Visualization and Computer Graphics, 2006
461 citations recorded by the source corpus.
Large scale scientific simulation codes typically run on a cluster of CPUs that write/read time steps to/from a single file system. As data sets are constantly growing in size, this increasingly leads to I/O bottlenecks. When the rate at which data is produced exceeds the available I/O bandwidth, the simulation stalls and the CPUs are idle. Data compression can alleviate this problem by using some CPU cycles to reduce the amount of data needed to be transfered. Most compression schemes, however, are designed to operate offline and seek to maximize compression, not throughput. Furthermore, they often require quantizing floating-point values onto a uniform integer grid, which disqualifies their use in applications where exact values must be retained. We propose a simple scheme for lossless, online compression of floating-point data that transparently integrates into the I/O of many applications. A plug-in scheme for data-dependent prediction makes our scheme applicable to a wide variety of data used in visualization, such as unstructured meshes, point sets, images, and voxel grids. We achieve state-of-the-art compression rates and speeds, the latter in part due to an improved entropy coder. We demonstrate that this significantly accelerates I/O throughput in real simulation runs. Unlike previous schemes, our method also adapts well to variable-precision floating-point and integer data
Subfields: Artificial Intelligence, Computer Graphics and Computer-Aided Design, Computer Networks and Communications
View Paper: https://www.semanticscholar.org/paper/09b71bc8d83e2583319b5bd42838e6c4ffa0bd70
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Graphics and Computer-Aided Design
In Computer Networks and Communications
Citations in the corpus
- Fast Error-Bounded Lossy HPC Data Compression with SZ (2016, methodology): “Fpzip [12] was proposed to compress the HPC data compression by”
- Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization (2017, methodology): “1’s 8.9, 1675% higher than ISABELA’s 1.2, 788% higher than FPZIP’s 2.4, and 1538% higher than GZIP’s 1.3.”
- Gorilla: A Fast, Scalable, In-Memory Time Series Database (2015, methodology): “This scheme leveraged XOR comparison with previous values to generate a delta encoding [25, 17].”