RNN Fisher Vectors for Action Recognition and Image Annotation

By Guy Lev, Gil Sadeh, Benjamin Klein, Lior Wolf. 2015

172 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) have had considerable success in classifying and predicting sequences. We demonstrate that RNNs can be effectively used in order to encode sequences and provide effective representations. The methodology we use is based on Fisher Vectors, where the RNNs are the generative probabilistic models and the partial derivatives are computed using backpropagation. State of the art results are obtained in two central but distant tasks, which both rely on sequences: video action recognition and image annotation. We also show a surprising transfer learning result from the task of image annotation to the task of video action recognition.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “To exploit this dependency, Liu et al. (2016a) proposed a spatio-temporal LSTM (STLSTM) network which extends the traditional LSTM-based learning to both temporal and spatial domains. Rather than concatenate the jointbased input features, ST-LSTM explicitly models the dependencies between the joints”
VideoLSTM convolves, attends and flows for action recognition (2016, neutral): “implies that our end-to-end prediction is highly complementary to hand-crafted approaches, as previously also observed by others (Feichtenhofer et al., 2016b; Lev et al., 2016; de Souza et al., 2016).”
Learnable pooling with Context Gating for video classification (2017, methodology): “The remaining question is: how to aggregate video features over the entire and potentially long video? One way to achieve this is to employ recurrent neural networks, such as long short-term memory (LSTM) [16] or gated recurrent unit (GRU) [17]), on top of the extracted frame-level features to captu”
Stacked Cross Attention for Image-Text Matching (2018, background): “A rich line of studies have explored mapping whole images and full sentences to a common semantic vector space for image-text matching [22, 38, 2, 39, 23, 27, 44, 10, 33, 13, 9, 8, 11].”
ECO: Efficient Convolutional Network for Online Video Understanding (2018, background): “An alternative way to model the temporal relation between frames is by using recurrent networks [6,12,13].”
Enhancing Video Summarization via Vision-Language Embedding (2017, methodology): “Such models have been used for cross-modal retrieval [16, 19, 26, 29, 27, 37, 39], image caption generation [12, 13, 27, 38, 44], and visual storytelling [11, 50].”
Instance-Aware Image and Sentence Matching with Selective Multimodal LSTM (2016, methodology): “[22] exploit RNN to encode FV for further performance improvement.”
Learning Semantic Concepts and Order for Image and Sentence Matching (2017, methodology): “0 RNN+FV [18] 34.”
Dual-Path Convolutional Image-Text Embedding (2017, methodology): “In this paper, in place of RNNs which are more commonly seen in image-text matching, we explore the usage of CNNs for text representation learning.”
Spatiotemporal Multiplier Networks for Video Action Recognition (2017, unknown): “RNN-FV [21] + C3D [35] + IDT 94.”
Movie Description (2016, methodology): “2015) and for the visual representation it uses RNN Fisher Vector (Lev et al. 2015), encoding video frames with the 19-layer VGG.”
ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification (2017, methodology): “More related to our approach, the temporal structure of the video can be also represented implicitly by an appropriate aggregation of descriptors across the video [16, 29, 31, 34, 55, 61].”
Predicting Visual Features From Text for Image and Video Caption Retrieval (2017, methodology): “Image caption retrieval: We compare a number of recently developed models for image caption retrieval [4]–[7], [42], [48], [67].”
Human Action Recognition From Various Data Modalities: A Review (2020, unknown): “3 RNN-FV+iDT [128] 2016 94.”
Linking Image and Text with 2-Way Nets (2016, unknown): “9 RNN-FV [20] 23.”
Spatial Temporal Transformer Network for Skeleton-based Action Recognition (2020, methodology): “Methods that fall into this category rely on different aspects of skeleton data: (1) RNN-based methods [21]–[24] leverage on the sequentiality of joint coordinates, treating input skeleton data as time series.”
InfoGCN: Representation Learning for Human Skeleton-based Action Recognition (2022, methodology): “In the early stages of deep learning-based approaches for skeleton-based action recognition, convolutional neural networks (CNNs) [10, 32, 46] and recurrent neural networks (RNNs) [11,25,31,53] were the standard models to adopt.”
Dual-path Convolutional Image-Text Embeddings with Instance Loss (2017, unknown): “0 3 RNN-FV [19] fixed VGG-19 feature from [17] 41.”