Only Time Can Tell: Discovering Temporal Data for Temporal Modeling
By Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, Lorenzo Torresani. 2019
98 citations recorded by the source corpus.
Understanding temporal information and how the visual world changes over time, is a fundamental ability of intelligent systems. In video understanding, temporal information is at the core of many current challenges, including compression, efficient inference, motion estimation or summarization. However, in current video datasets it has been observed that action classes can often be recognized without any temporal information, from a single frame of video. As a result, both benchmarking and training in these datasets may give an unintentional advantage to models with strong image understanding capabilities, as opposed to those with strong temporal understanding. In other words, current datasets may not reward good temporal understanding, potentially hindering progress. In this paper we address this problem head on by identifying action classes where temporal information is actually necessary to recognize them and call these "temporal classes". Selecting temporal classes using a computational method would bias the process. Instead, we propose a methodology based on a simple and effective human annotation experiment. We remove just the temporal information, by shuffling frames in time, and measure if the action can still be recognized. Classes that cannot be recognized when frames are not in order, are included in the temporal set. We observe that this set is statistically different from other static classes, and that performance in it correlates with a network's ability to capture temporal information. Thus we use it as a benchmark on current popular networks, which reveals a series of interesting facts, like inflated convolutions bias networks towards classes where motion is not important. We also explore the effect of training on the temporal set, and observe that this leads to better generalization in unseen classes, demonstrating the need for more temporal data. We hope that the proposed dataset of temporal categories will help guide future research in temporal modeling for better video understanding.
Subfields: Time-Series & Forecasting
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Time-Series & Forecasting
Citations in the corpus
- Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, background): “For example, although classifying some human actions requires the temporal ordering of video frames, many actions can be classified from just their distribution over frames or even from a single frame [51].”
- Is Space-Time Attention All You Need for Video Understanding? (2021, background): “Indeed, prior work (Sevilla-Lara et al., 2021) has shown that on K400, spatial cues are more important than temporal information in order to achieve strong accuracy.”
- TempCompass: Do Video LLMs Really Understand Videos? (2024, background): “…age of LLMs, a lot of studies (Goyal et al., 2017; Yi et al., 2020; Yu et al., 2019; Bagad et al., 2023; Buch et al., 2022; Hendricks et al., 2018; Sevilla-Lara et al., 2019; Jang et al., 2017; Ren et al., 2023a) have already been conducted to evaluate the temporal perception performance of…”
- Learning De-biased Representations with Biased Representations (2019, background): “3D CNNs have proven their state-of-the-art performances on action recognition benchmarks such as Kinetics (Carreira & Zisserman, 2017), but recent studies (Sevilla-Lara et al., 2019; Li et al., 2018; Li & Vasconcelos, 2019) have shown that such action datasets have strong static biases towards the s”
- ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “In contrast to K400, SSv2 presents richer temporal information with much higher significance [60].”
- Video Transformer Network (2021, background): “We also argue that this could mean that Kinetics-400 is primarily a static frame, appearance based classification problem rather than a motion problem [31].”