Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections
By Yi-An Lai, Xuan Zhu, Yi Zhang, Mona T. Diab. 2020
29 citations recorded by the source corpus.
Summarizing data samples by quantitative measures has a long history, with descriptive statistics being a case in point. However, as natural language processing methods flourish, there are still insufficient characteristic metrics to describe a collection of texts in terms of the words, sentences, or paragraphs they comprise. In this work, we propose metrics of diversity, density, and homogeneity that quantitatively measure the dispersion, sparsity, and uniformity of a text collection. We conduct a series of simulations to verify that each metric holds desired properties and resonates with human intuitions. Experiments on real-world datasets demonstrate that the proposed characteristic metrics are highly correlated with text classification performance of a renowned model, BERT, which could inspire future applications.
Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Douwe Kiela
- Mohit Bansal
- Nazneen Rajani
- Tristan Thrush
- Yacine Jernite
- Yejin Choi
- Margaret Mitchell
- Angelina McMillan-Major
In Natural Language Processing
- Tristan Thrush
- Douwe Kiela
- Nazneen Rajani
- Angelina McMillan-Major
- Yacine Jernite
- Alexandra Sasha Luccioni
- Margaret Mitchell
- Jason Aaron Edward Weston
In Evaluation & Benchmarks
- Tristan Thrush
- Douwe Kiela
- Angelina McMillan-Major
- Yacine Jernite
- Alexandra Sasha Luccioni
- Margaret Mitchell
- Nazneen Rajani
- Mohit Bansal
Citations in the corpus
- Measuring Data (2022, background): “, 2002); “characteristic metrics”, which the authors define as “unsupervised measures” to quantitatively describe the properties of a collection of data or a dataset (Lai et al., 2020); “intrinsic metrics”, to quantify the properties of a dataset (Grusky et al.”