Foveated convolutional neural networks for video summarization

TitleFoveated convolutional neural networks for video summarization
Publication TypeJournal Article
Year of Publication2018
AuthorsWu, J, Sheng-hua, Z, Ma, Z, Heinen, SJ, Jiang, J
JournalMultimedia Tools and Applications
Volume77
Issue22
Start Page29245
Pagination29245-29267
Date Published11/2018
ISSN1380-7501
Abstract

With the proliferation of video data, video summarization is an ideal tool for users to browse video content rapidly. In this paper, we propose a novel foveated convolutional neural networks for dynamic video summarization. We are the first to integrate gaze information into a deep learning network for video summarization. Foveated images are constructed based on subjects’ eye movements to represent the spatial information of the input video. Multi-frame motion vectors are stacked across several adjacent frames to convey the motion clues. To evaluate the proposed method, experiments are conducted on two video summarization benchmark datasets. The experimental results validate the effectiveness of the gaze information for video summarization despite the fact that the eye movements are collected from different subjects from those who generated summaries. Empirical validations also demonstrate that our proposed foveated convolutional neural networks for video summarization can achieve state-of-the-art performances on these benchmark datasets.

DOI10.1007/s11042-018-5953-1
Alternate JournalSpringer Nature 2018

Related Centers, Labs, Projects