Shedding Light on the Dark Data in the Long Tail of Science
By M. Cragin, P. Heidorn. Library Trends, 2008
447 citations recorded by the source corpus.
One of the primary outputs of the scientific enterprise is data, but many institutions such as libraries that are charged with preserving and disseminating scholarly output have largely ignored this form of documentation of scholarly activity. This paper focuses on a particularly troublesome class of data, termed dark data. “Dark data” is not carefully indexed and stored so it becomes nearly invisible to scientists and other potential users and therefore is more likely to remain underutilized and eventually lost. The article discusses how the concepts from long-tail economics can be used to understand potential solutions for better curation of this data. The paper describes why this data is critical to scientific progress, some of the properties of this data, as well as some social and technical barriers to proper management of this class of data. Many potentially useful institutional, social, and technical solutions are under development and are introduced in the last sections of the paper, but these solutions are largely unproven and require additional research and development.
Subfields: Economics and Econometrics, Information Systems, Statistics and Probability
View Paper: https://www.semanticscholar.org/paper/d886362c98b22d2110d7c0d0da62511c5b315f12
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Economics and Econometrics
In Information Systems
In Statistics and Probability
Citations in the corpus
- Ecoinformatics: supporting ecology as a data-intensive science. (2012, background): “The diversity in scales studied and the ways in which studies are carried out results in large numbers of small, idiosyncratic data sets that accumulate from the thousands of scientists that collect relevant biological, ecological and environmental data [18].”
- Designing forest biodiversity experiments: general considerations illustrated by a new large experiment in subtropical China (2014, background): “Adequate combination and aggregation of such data – with their long tail typical for extensive data set compilations (Heidorn 2008), that is, with few data sets with very large amounts of similarly structured data (e.”
- Motivating Online Publication of Data (2009, methodology): “The cost of saving and reusing data published online is also likely to be lower than the cost of collecting them again (Heidorn 2008).”
- Will it rise or will it fall? Managing the complex effects of urbanization on base flow (2016, background): “Data may be available but may be unpublished or unavailable in publicly served electronic format (i.e., ‘dark data’; Heidorn 2008).”