Modulating early visual processing by language

By Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, Aaron C. Courville. 2017

551 citations recorded by the source corpus.

It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic input are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the \emph{entire visual processing} by linguistic input. Specifically, we condition the batch normalization parameters of a pretrained residual network (ResNet) on a language embedding. This approach, which we call MOdulated RESnet (\MRN), significantly improves strong baselines on two visual question answering tasks. Our ablation study shows that modulating from the early stages of the visual processing is beneficial.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Parameter-Efficient Transfer Learning for NLP (2019, methodology): “This technique, similar to conditional batch normalization (De Vries et al., 2017), FiLM (Perez et al., 2018), and selfmodulation (Chen et al., 2019), also yields parameterefficient adaptation of a network; with only 2d parameters per layer.”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “2, we do miss some families of tasks including visually grounded dialog [4, 45], embodied tasks like question answering [7] and instruction following [8], and text generation tasks like image and video captioning [5].”
Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “We provide class information to G with class-conditional BatchNorm (Dumoulin et al., 2017; de Vries et al., 2017) and to D with projection (Miyato & Koyama, 2018).”
Spectral Normalization for Generative Adversarial Networks (2018, methodology): “For the generator network of conditional GANs, we used conditional batch normalization (CBN) (Dumoulin et al., 2017; de Vries et al., 2017).”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “This often takes the form of class-conditional normalization statistics [14, 10] as well as discriminators with heads that are explicitly designed to behave like classifiers p(y|x) [34].”
Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis (2020, background): “In [5], [15], [42], [43], conditional information is provided to the generator by conditional gains and bias in the Batch Normalization [29] layers.”
Object-Centric Image Generation from Layouts (2020, background): “As in [41,53], the parameters γ, β of our batch-normalization layers are conditional and determined on a per-pixel level (as opposed to classical conditional batch-normalization [6]).”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, methodology): “Conditional normalization layers include the Conditional Batch Normalization (Conditional BN) [12] and Adaptive Instance Normalization (AdaIN) [19], Both were first used in the style transfer task and later adopted in various vision tasks [5,9,11,20,25,34,36,39,46,51].”
cGANs with Projection Discriminator (2018, methodology): “The method with auxiliary classiﬁer loss seems effective in improving the visual appearance, but not in training faithful generative model. et al., 2017b; de Vries et al., 2017) was used in place of the standard batch normalization layer.”
Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, methodology): “We implemented our occupancy network using a fully-connected neural network with 5 ResNet blocks [29] and condition it on the input using conditional batch normalization [13, 16].”
Uncertainty in Multitask Transfer Learning (2018, methodology): “In this formulation, we use a ResNet [16] network, where we inserted FILM layers [26, 9] between each residual block to condition on the task.”
Learning Rapid-Temporal Adaptations (2017, methodology): “As a form of feature-wise transformation method, the adaptive neurons are broadly related to conditional normalization techniques [20, 6, 28].”
Bayesian Hypernetworks (2017, background): “These works condition the primary net’s behaviour on information about the task, in the form of task IDs (Dumoulin et al., 2016; Kirkpatrick et al., 2016), natural language (de Vries et al., 2017; Perez et al., 2017), or task-exemplars (Huang and Belongie, 2017).”
Overcoming Catastrophic Forgetting by Neuron-level Plasticity Control (2019, background): “There are a few alternatives available in such cases, such as conditional batch normalization(De Vries et al. 2017) and virtual batch normalization(Salimans et al. 2016).”
End-to-End Adversarial Text-to-Speech (2020, neutral): “The latents z and speaker embedding s modulate the scale and shift parameters of the batch normalisation layers [13, 15].”