signSGD via Zeroth-Order Oracle

By Sijia Liu, Pin-Yu Chen, Xiangyi Chen, Mingyi Hong. 2019

173 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, background): “…performing updates only based on the sign of each coordinate of the gradient have recently gaining popularity for training deep learning models Seide et al. [2014], Carlson et al. [2015], Wen et al. [2017], Balles and Hennig [2018], Bernstein et al. [2018], Zaheer et al. [2018], Liu et al. [2018].”
Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressor (2020, unknown): “…(Alistarh et al., 2018), standard random dithering (Goodall, 1951; Roberts, 1962; Alistarh et al., 2017), natural dithering (Horv´ath et al., 2019a), ternary quantization (Wen et al., 2017), and scaled sign quantization (Karimireddy et al., 2019b; Bernstein et al., 2018, 2019; Liu et al., 2019).”
A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning: Principals, Recent Advances, and Applications (2020, methodology): “Using the sign operation scales down the (coordinate-wise) estimation errors [36], [42].”
Fine-Tuning Language Models with Just Forward Passes (2023, background): “Particularly salient applications of ZO to deep learning are distributed methods [88, 41] and black-box adversarial example generation [13, 61, 16, 62].”
Communication-Efficient Edge AI: Algorithms and Systems (2020, background): “red a distributed gradientfree algorithm for multi-agent convex optimization, where the agents can only exchange quantized data information due to limited bandwidth. In the extreme case considered in [156], each estimated gradient is further quantized into 1 bit, which enjoys high communication efﬁc”