Google Dataset Search: Building a search engine for datasets in an open Web ecosystem

By D. Brickley, M. Burgess, Natasha Noy. The Web Conference, 2019

352 citations recorded by the source corpus.

There are thousands of data repositories on the Web, providing access to millions of datasets. National and regional governments, scientific publishers and consortia, commercial data providers, and others publish data for fields ranging from social science to life science to high-energy physics to climate science and more. Access to this data is critical to facilitating reproducibility of research results, enabling scientists to build on others' work, and providing data journalists easier access to information and its provenance. In this paper, we discuss Google Dataset Search, a dataset-discovery tool that provides search capabilities over potentially all datasets published on the Web. The approach relies on an open ecosystem, where dataset owners and providers publish semantically enhanced metadata on their own sites. We then aggregate, normalize, and reconcile this metadata, providing a search engine that lets users find datasets in the “long tail” of the Web. In this paper, we discuss both social and technical challenges in building this type of tool, and the lessons that we learned from this experience.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/bcbf86a3653270d5fe826ea9c5266e9c3a01613d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yuji Roh
2. Steven Euijong Whang
3. Hernisa Kacorri
4. Jae-Gil Lee
5. Hwanjun Song

In Information Systems

1. Jae-Gil Lee
2. Hwanjun Song
3. Yuji Roh
4. Steven Euijong Whang

In Management Science and Operations Research

1. Hernisa Kacorri
2. Jae-Gil Lee
3. Hwanjun Song
4. Yuji Roh
5. Steven Euijong Whang
6. Peter Christen

Citations in the corpus

• IncluSet: A Data Surfacing Repository for Accessibility Datasets (2020, methodology): “Half-way through our dataset collection, we were excited to see Google deploy the new Dataset Search engine [23].”
• Data collection and quality challenges in deep learning: a data-centric AI perspective (2021, background): “More recently, a public version of Goods called Google Dataset Search [29], which facilitates search primarily for science datasets, has been released.”
• Croissant: A Metadata Format for ML-Ready Datasets (2024, neutral): “In addition to the support from individual data repositories, Croissant is also supported by Google Dataset Search [26].”
• Continuous Online Evaluation of Recommendation Strategies in Social Science Academic Search (2026, unknown): “These platforms provide access to subject-specific information, such as publications, as well as other types of content, such as research data [11, 20], images, videos & audio [74], and research output & software [20, 26].”