Unifying Orthogonal Monte Carlo Methods

By Krzysztof Marcin Choromanski, Mark Rowland, Wenyu Chen, Adrian Weller. 2019

29 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Rethinking Attention with Performers (2020, background): “(2) Hadamard/Givens ORFs [H/G-ORFs]: Applies random Hadamard (Choromanski et al., 2017) or Givens matrices (Choromanski et al., 2019b).”
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (2020, methodology): “(2) Hadamard/Givens ORFs [H/G-ORFs]: Applies random Hadamard [13]/Givens matrices [11].”
Hybrid Random Features (2021, methodology): “…Carlo (QMC) methods, where directions and/or lengths of Gaussian vectors used to produce features are correlated, often through geometric conditions such as orthogonality (Choromanski et al., 2017; Rowland et al., 2018; Yu et al., 2016; Choromanski et al., 2019; Choromanski & Sindhwani, 2016).”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, methodology): “Butterfly, Sparse + Low-rank Matrices: Butterfly matrices have been used in numerical linear algebra [Parker, 1995, Li et al., 2015] and machine learning [Mathieu and LeCun, 2014, Jing et al., 2017, Munkhoeva et al., 2018, Dao et al., 2019, Choromanski et al., 2019].”
Monarch: Expressive Structured Matrices for Efficient and Accurate Training (2022, methodology): “The class of butterfly matrices (Parker, 1995) have also been used in kernel models (Munkhoeva et al., 2018; Choromanski et al., 2019) and deep learning models (Vahid et al.”
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps (2020, methodology): “In machine learning, butterﬂy matrices have been use to approximate the Hessian for fast optimization (Mathieu & LeCun, 2014), and to perform fast random projection (Jing et al., 2017; Munkhoeva et al., 2018; Choromanski et al., 2019).”
Demystifying Orthogonal Monte Carlo and Beyond (2020, methodology): “OMC relies on ensembles of mutually orthogonal random samples for variance reduction and turns out to be very effective in virtually all applications of MC in machine learning involving isotropic distributions [36, 11, 18, 14, 35].”