BSN: Boundary Sensitive Network for Temporal Action Proposal Generation

By Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang. 2018

794 citations recorded by the source corpus.

Temporal action proposal generation is an important yet challenging problem, since temporal proposals with rich action content are indispensable for analysing real-world videos with long duration and high proportion irrelevant content. This problem requires methods not only generating proposals with precise temporal boundaries, but also retrieving proposals to cover truth action instances with high recall and high overlap using relatively fewer proposals. To address these difficulties, we introduce an effective proposal generation method, named Boundary-Sensitive Network (BSN), which adopts "local to global" fashion. Locally, BSN first locates temporal boundaries with high probabilities, then directly combines these boundaries as proposals. Globally, with Boundary-Sensitive Proposal feature, BSN retrieves proposals by evaluating the confidence of whether a proposal contains an action within its region. We conduct experiments on two challenging datasets: ActivityNet-1.3 and THUMOS14, where BSN outperforms other state-of-the-art temporal action proposal generation methods with high recall and high temporal precision. Finally, further experiments demonstrate that by combining existing action classifiers, our method significantly improves the state-of-the-art temporal action detection performance.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Finally, MQ can be seen as activity detection [141, 229, 237] but for the activities of the camera wearer.”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, methodology): “Span Predictor: To predict temporal spans, we follow existing works [21, 33, 41] to predict the probability of each position being the start or end of the span.”
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval (2020, methodology): “Inspired by span predictors in reading comprehension [36,3] and action localization [27], we use start-end predictors to predict start-end probabilities from early fused query-video representations.”
Gaussian Temporal Awareness Networks for Action Localization (2019, methodology): “(4) Boundary Sensitive Network (BSN) [21] locates temporal boundaries with three action-ness curves and reranks proposals with neural networks.”
Completeness Modeling and Context Separation for Weakly Supervised Temporal Action Localization (2019, background): “Fully supervised temporal action localization has witnessed remarkable progress recently [39, 48, 15, 9, 37, 51, 47, 8, 2, 31].”
Video Self-Stitching Graph Network for Temporal Action Localization (2020, methodology): “M scr , comprised of a stack of Conv1d (3, 1), ReLU and Conv1d (1, 1), predicts action-ness/startness/endness scores [20] for each sequence.”
ActionFormer: Localizing Moments of Actions with Transformers (2022, methodology): “Follow the common practice in [36, 38, 76], we train our model on the training set and report the performance on the validation set.”
G-TAD: Sub-Graph Localization for Temporal Action Detection (2019, methodology): “In inference, following [30] to leverage the global video context, we take the video classification scores from [42] and [51], and multiply them by the fused confidence score pj for evaluation.”
Graph Convolutional Networks for Temporal Action Localization (2019, methodology): “Upon the I3D features, we further apply max pooling across segments to obtain one 1024-dimensional feature vector for each proposal that is obtained by the BSN method [27].”
SCSampler: Sampling Salient Clips From Video for Efficient Action Recognition (2019, background): “Action localization is often approached through a two-step mechanism [5, 8, 5, 14, 15, 21, 28, 1], where first an action proposal method identifies candidate action segments, and then a more sophisticated approach validates the class of each candidate and refines its temporal boundaries.”
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning (2023, background): “Dense video captioning lies at the intersection of event localization [25, 29, 33, 62, 65, 66, 84, 127] and event captioning [30,63,75,97,104].”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, methodology): “Two action proposal generation methods are evaluated: Boundary Sensitive Network (BSN) [35] and Temporal Actionness Grouping (TAG) [61].”
Listen to Look: Action Recognition by Previewing Audio (2019, background): “They generate a large number of clip proposals from the video, and then use ﬂow [40] or deep features [8, 79]) for proposal selection.”
End-to-End Temporal Action Detection With Transformer (2021, background): “The candidate segments are generated by dense uniform sampling [16] or grouping local frames that may contain actions [12].”
Temporal Action Detection with Structured Segment Networks (2017, methodology): “Among the methods after 2018, BSN (Lin et al. 2018) is also a bottom-up approach but estimates the confidence of”