FiLM: Visual Reasoning with a General Conditioning Layer
By Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, Aaron C. Courville. 2017
4,130 citations recorded by the source corpus.
We introduce a general-purpose conditioning method for neural networks called FiLM: Feature-wise Linear Modulation. FiLM layers influence neural network computation via a simple, feature-wise affine transformation based on conditioning information. We show that FiLM layers are highly effective for visual reasoning - answering image-related questions which require a multi-step, high-level process - a task which has proven difficult for standard deep learning methods that do not explicitly model reasoning. Specifically, we show on visual reasoning tasks that FiLM layers 1) halve state-of-the-art error for the CLEVR benchmark, 2) modulate features in a coherent manner, 3) are robust to ablations and architectural modifications, and 4) generalize well to challenging, new data from few examples or even zero-shot.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Percy Liang
- Trevor Darrell
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Bernt Schiele
In Multimodal Learning
Citations in the corpus
- Parameter-Efficient Transfer Learning for NLP (2019, methodology): “, 2017), FiLM (Perez et al., 2018), and selfmodulation (Chen et al.”
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers (2019, methodology): “7 NLVR 2 NLVR 2 (Suhr et al., 2019) is a challenging visual reasoning dataset where some existing approaches (Hu et al., 2017; Perez et al., 2018) fail, and the SotA method is ‘MaxEnt’ in Suhr et al. (2019).”
- Large Scale GAN Training for High Fidelity Natural Image Synthesis (2018, methodology): “Instead of having a separate layer for each embedding (Miyato et al., 2018; Zhang et al., 2018), we opt to use a shared embedding, which is linearly projected to each layer’s gains and biases (Perez et al., 2018).”
- Diffusion Models Beat GANs on Image Synthesis (2021, methodology): “We also experiment with a layer [37] that we refer to as adaptive group normalization (AdaGN), which incorporates the timestep and class embedding into each residual block after a group normalization operation [61], similar to adaptive instance norm [21] and FiLM [41].”
- Image Super-Resolution via Iterative Refinement (2021, methodology): “We experimented with more sophisticated methods of conditioning, including FiLM [42], but found that the simple concatenation yielded similar genera-”
- Symbolic Music Generation with Diffusion Models (2021, methodology): “We use feature-wise linear modulation (Perez et al., 2018) to generate γ (scale) and ξ (shift) parameters given a noise encoding and apply the transformation γφ+ ξ to the output φ of each layer normalization block in each residual layer, allowing for effective conditioning of the diffusion model.”
- LILA: Language-Informed Latent Actions (2021, methodology): “We incorporate these embeddings using feature-wise linear modulation (FiLM) layers [46] that fuse language information with other features h by mapping language embeddings to parameters ( γ, β ) of an affine transformation: h (cid:48) = γ ∗ h + β (Fig.”
- Imitating Interactive Intelligence (2020, methodology): “For this model, we use a ResNet augmented with FiLM (Perez et al., 2018), which has elsewhere proved effective for visual reasoning tasks.”
- Scalable Diffusion Models with Transformers (2022, methodology): “Dhariwal and Nichol [9] ablated several architecture choices for the U-Net, such as the use of adaptive normalization layers [40] to inject conditional information and channel counts for convolutional layers.”
- ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition (2021, methodology): “[36] Ethan Perez, Florian Strub, Harm De Vries, Vincent Du-”
- Fast Model Editing at Scale (2021, methodology): “To enable some layer-wise specialization, MEND applies a layer-specific scale s` and offset o` to the editor network hidden state and output, similar to FiLM layers (Perez et al., 2018).”
- Dynamic Neural Networks: A Survey (2021, background): “Cross-Field Image captioning [130], [242], Video captioning [243], [244], Visual question answering [123], [124], [245], Multi-modal sentiment analysis [246], [247]”
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, background): “Modulation-based vision-and-language models (Perez et al., 2018; Nguyen et al., 2020) also fall under Figure 2c, with their visual CNN stems corresponding to visual embedder, RNNs producing the modulation parameters to textual embedder, and modulated CNNs to modality interaction.”
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding (2019, methodology): “This includes simple linear models using ranking loss [16, 36], bi-linear pooling meth-ods [45, 20, 55, 17, 37], attention-based methods [1, 52, 64] and reasoning based approaches [54, 27, 33, 38, 29] on CLEVER and VQA-v1.”
- Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “Feature gating has been used for many tasks, such as machine translation [18], VQA [19], reinforcement learning [20], image classification [21,22], and action recognition [23].”