Data Engineering for Everyone
By Vijay Janappa Reddi, G. Diamos, Pete Warden, Peter Mattson, David Kanter. 2021
10 citations recorded by the source corpus.
Data engineering is one of the fastest-growing fields within machine learning (ML). As ML becomes more common, the appetite for data grows more ravenous. But ML requires more data than individual teams of data engineers can readily produce, which presents a severe challenge to ML deployment at scale. Much like the software-engineering revolution, where mass adoption of open-source software replaced the closed, in-house development model for infrastructure code, there is a growing need to enable rapid development and open contribution to massive machine learning data sets. This article shows that open-source data sets are the rocket fuel for research and innovation at even some of the largest AI organizations. Our analysis of nearly 2000 research publications from Facebook, Google and Microsoft over the past five years shows the widespread use and adoption of open data sets. Open data sets that are easily accessible to the public are vital to accelerating ML innovation for everyone. But such open resources are scarce in the wild. So, what if we are able to accelerate data-set creation via automatic data set generation tools?
Subfields: Computer Networks and Communications, Information Systems, Information Systems and Management, Software Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Networks and Communications
In Information Systems
Citations in the corpus
- The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage (2021, background): “This work is a first step towards creating a speech dataset that covers all speakers and environments (not just English audiobooks [22, 25, 26]) for open use [27].”