A Practical Online Framework for Extracting Running Video Summaries under a Fixed Memory Budget

By Chandrashekhar Lavania, Kai Wei, Rishabh K. Iyer, J. Bilmes. 2021

5 citations recorded by the source corpus.

We study the problem of summarizing a video stream (potentially of unbounded duration) on the fly, where the summarization system must operate under a fixed memory bud-get and should produce an appropriate summary of its past at each time step. This problem is motivated by applications that have access to only limited memory and compute resources (e.g., sensor networks, smart devices and phones). We approach this problem as constrained streaming maximization of a natural submodular objective function. In particular, we propose a novel feature-based submodular function for the summarization objective — this function is instantiated by deep-model-learnt features. We solve the constrained streaming maximization problem with an algo-rithm that abides by the memory budget. Our streaming algorithm, is unique in that it uses both an “adding gain” (to determine something new) and a “swapping gain” (to determine something better) relative to a current summary. Based on a time dependent F-measure method to gauge the performance of streaming summarization techniques, we demonstrate that our approach provides significant improvement over a number of state-of-the-art baseline methods that utilize comparable resources on both the TVSum50 and SumMe data sets.

Citations in the corpus