Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings

By Bikash Gyawali, Lucas Anastasiou, Petr Knoth. 2020

26 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

1. Daphne Ippolito
2. Nicholas Carlini
3. Katherine Lee
4. Chiyuan Zhang
5. Chris Callison-Burch
6. Andrew Nystrom
7. Douglas Eck
8. Mohit Bansal

In Evaluation & Benchmarks

1. Chiyuan Zhang
2. Chris Callison-Burch
3. Andrew Nystrom
4. Douglas Eck
5. Nicholas Carlini
6. Katherine Lee
7. Daphne Ippolito
8. Philipp Koehn

Citations in the corpus

• Deduplicating Training Data Makes Language Models Better (2021, methodology): “MinHash (Broder, 1997) is an approximate matching algorithm widely used in large-scale deduplication tasks (Versley and Panchenko, 2012; Gabriel et al., 2018; Gyawali et al., 2020), including to deduplicate the training set for a large Chinese-language LM (Zeng et al., 2021).”
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “Furthermore, MinHash (Broder, 1997) stands out as a prevalent algorithm for large-scale deduplication tasks (Gyawali et al., 2020).”