Bayesian Optimization of Text Representations

By Dani Yogatama, Lingpeng Kong, Noah A. Smith. 2015

45 citations recorded by the source corpus.

When applying machine learning to problems in NLP, there are many choices to make about how to represent input texts. These choices can have a big effect on performance, but they are often uninteresting to researchers or practitioners who simply need a module that performs well. We propose an approach to optimizing over this space of choices, formulating the problem as global optimization. We apply a sequential model-based optimization technique and show that our method makes standard linear models competitive with more sophisticated, expensive state-of-the-art methods based on latent variable models or neural networks on various topic classification and sentiment analysis problems. Our approach is a first step towards black-box NLP systems that work with raw text and do not require manual tuning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification (2015, methodology): “Emerging research has begun to explore hyperparameter optimization methods, including random search (Bengio, 2012), and Bayesian optimization (Yogatama and Smith, 2015; Bergstra et al., 2013).”
Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016, neutral): “A natural choice for the prior distribution is a product of K beta distributions: as this is the conjugate prior to the Bernoulli likelihood, and it leads to efﬁcient posterior updating.”
Show Your Work: Improved Reporting of Experimental Results (2019, methodology): “We start by applying our technique on a text classification task in order to confirm a well-established observation (Yogatama and Smith, 2015): logistic”
Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell than You Might Think! (2020, background): “…are an opaque tool for analysis, especially for deep neural networks: performance differences versus baselines, frequently small in magnitude, can often be attributed to hyperparameter search schemes, random seeds, the number of models compared, etc. (Yogatama and Smith, 2015; Dodge et al., 2019).”