ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms

By Martin Aumüller, Erik Bernhardsson, Alexander John Faithfull. Similarity Search and Applications, 2018

590 citations recorded by the source corpus.

This paper describes ANN-Benchmarks, a tool for evaluating the performance of in-memory approximate nearest neighbor algorithms. It provides a standard interface for measuring the performance and quality achieved by nearest neighbor algorithms on different standard data sets. It supports several different ways of integrating k-NN algorithms, and its configuration system automatically tests a range of parameter settings for each algorithm. Algorithms are compared with respect to many different (approximate) quality measures, and adding more is easy and fast; the included plotting front-ends can visualise these as images, Open image in new window plots, and websites with interactive plots. ANN-Benchmarks aims to provide a constantly updated overview of the current state of the art of k-NN algorithms. In the short term, this overview allows users to choose the correct k-NN algorithm and parameters for their similarity search task; in the longer term, algorithm designers will be able to use this overview to test and refine automatic parameter tuning. The paper gives an overview of the system, evaluates the results of the benchmark, and points out directions for future work. Interestingly, very different approaches to k-NN search yield comparable quality-performance trade-offs. The system is available at http://sss.projects.itu.dk/ann-benchmarks/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/8a65888ea8199749f05b95471138fe9f1d946fb0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

An anatomy for neural search engines (2019, methodology): “That rules out exact and efficient nearest neighbor search algorithms [Aumüller et al., 2017].”
Milvus: A Purpose-Built Vector Data Management System (2021, background): “Vector similarity search (a.k.a high-dimensional nearest neighbor search) is an extensively studied topic both for approximate search (e.g., [7, 41]) and exact search (e.g., [38, 42]).”
Accelerating Large-Scale Inference with Anisotropic Vector Quantization (2019, methodology): “Our implementation is open-source and available at https://github.com/google-research/ google-research/tree/master/scann and furthermore the exact conﬁgurations used to produce our benchmark numbers are part of the ANN-Benchmarks GitHub repository.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, methodology): “ScaNN [268] is developed with quantizationbased approaches, which achieves excellent performance on ANN-Benchmarks.”
Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks (2019, methodology): “These models are also amenable to precomputation of document representations—and fast retrieval using approximate nearest neighbor search [Aumüller et al., 2017, Boytsov et al., 2016].”
Fast geometric learning with symbolic matrices (2020, unknown): “MNIST [73], SIFT [62], GloVe-25 and GloVe-100 [88] were taken from the ANN-benchmarks repository [4], while HyperE-10 and HyperE-50 are hyperbolic embeddings processed from WordNet datasets [95].”
Search Result Clustering in Collaborative Sound Collections (2020, unknown): “[7] or similar approximate methods [1, 21]).”
A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Search (2021, methodology): “For these algorithms, existing surveys [10, 57, 79] provide some meaningful”
The Faiss Library (2024, unknown): “Most of the research works on vector search have been open-sourced, and some of these evolved in relatively comprehensive software packages for vector search.”
VIBE: Vector Index Benchmark for Embeddings (2025, unknown): “In line with the earlier performance evaluations (e.g., Li et al., 2019; Aumüller et al., 2020), the general trend is that the graph and clustering methods significantly outperform the hashing and tree methods: the top-performing methods contain only graph methods and clustering methods 5 .”
MINT: Multi-Vector Search Index Tuning (2025, unknown): “Graph-based techniques organize items in graphs and are prevalent these days because of their efficiency and high recall [5, 6, 29–31, 55, 56].”