Adaptive Posterior Learning: few-shot learning with a surprise-based memory module

By Tiago Ramalho, Marta Garnelo. 2019

83 citations recorded by the source corpus.

The ability to generalize quickly from few observations is crucial for intelligent systems. In this paper we introduce APL, an algorithm that approximates probability distributions by remembering the most surprising observations it has encountered. These past observations are recalled from an external memory module and processed by a decoder network that can combine information from different memory slots to generalize beyond direct recall. We show this algorithm can perform as well as state of the art baselines on few-shot classification benchmarks with a smaller memory footprint. In addition, its memory compression allows it to scale to thousands of unknown labels. Finally, we introduce a meta-learning reasoning task which is more challenging than direct classification. In this setting, APL is able to generalize with fewer than one example per class via deductive reasoning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Efficient Meta Lifelong-Learning with Limited Memory (2020, methodology): ““Uncertainty” utilizes model’s conﬁdence level (Ramalho and Garnelo, 2019).”
Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, background): “Ramalho and Garnelo (2019) proposed to store samples that the model is least conﬁdent.”
Generalizing from a Few Examples (2019, methodology): “FSL has also been used in curve fitting [39, 48, 114, 156] and understanding number analogy by logic reasoning to perform calculations [104].”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “APL [158] further summarizes the model-based knowledge as a probability distribution.”
Meta-Neighborhoods (2019, background): “For instance, previous works tackling few-shot classifications add a new slot to the memory when the label of a given data does not match the labels of its k-nearest neighbors from the memory [1] or the given data is misclassified [29].”
Few-Shot Learning With Global Class Representations (2019, unknown): “7 APL [22] 97.”
Meta-Learning with Sparse Experience Replay for Lifelong Language Learning (2020, unknown): “Other strategies such as those based on surprise (Ramalho and Garnelo, 2019) and forgetting (Toneva et al.”
ADER: Adaptively Distilled Exemplar Replay Towards Continual Learning for Session-based Recommendation (2020, background): “[30] proposes to store the most “surprising” samples that the model is least confident.”