Foofah: Transforming Data By Example

By Zhongjun (Mark) Jin, Michael R. Anderson, Michael J. Cafarella, Hosagrahar Visvesvaraya Jagadish. 2017

137 citations recorded by the source corpus.

Data transformation is a critical first step in modern data analysis: before any analysis can be done, data from a variety of sources must be wrangled into a uniform format that is amenable to the intended analysis and analytical software package. This data transformation task is tedious, time-consuming, and often requires programming skills beyond the expertise of data analysts. In this paper, we develop a technique to synthesize data transformation programs by example, reducing this burden by allowing the analyst to describe the transformation with a small input-output example pair, without being concerned with the transformation steps required to get there. We implemented our technique in a system, FOOFAH, that efficiently searches the space of possible data transformation operations to generate a program that will perform the desired transformation. We experimentally show that data transformation programs can be created quickly with FOOFAH for a wide variety of cases, with 60% less user effort than the well-known WRANGLER system.

Subfields: Computer Networks and Communications, Information Systems and Management, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

1. Aditya G. Parameswaran
2. Yihan Gao
3. Silu Huang
4. Sanjay Krishnan
5. Eugene Wu
6. Steven Euijong Whang

In Information Systems and Management

1. Sanjay Krishnan
2. Eugene Wu
3. Yinjun Wu
4. Susan B. Davidson
5. Francesca Toni
6. Piyawat Lertvittayakumjorn

In Management Science and Operations Research

1. Eugene Wu
2. Sanjay Krishnan
3. Aditya G. Parameswaran
4. Yihan Gao
5. Silu Huang
6. M. Franklin
7. Ken Goldberg
8. Jiannan Wang

Citations in the corpus

• AlphaClean: Automatic Generation of Data Cleaning Pipelines (2019, background): “Wrangler [27] and Foofah [26] are text extraction and transformation systems that similarly formulate their problems as search over a language of text transformations, and develop manual pruning heuristics to reduce the search space.”
• Navigating the Data Lake with DATAMARAN: Automatically Extracting Structure from Log Datasets (2017, methodology): “Work from program synthesis has developed techniques to perform extraction or transformation from examples [24, 26, 33, 36], while some others [34, 44] require the users to provide the transformation steps; instead, we are opting for a fully unsupervised approach.”