ICANet: A Method of Short Video Emotion Recognition Driven by Multimodal Data
^††thanks: Xuecheng Wu, Mengmeng Tian, and Lanhang Zhai are all with school of Cyber Science and Engineering, Zhengzhou University. (Corresponding authors: Xuecheng Wu.)

In this paper, the audio signal is preprocessed by generating the LFCC spectrograms. LFCC spectrogram is a three-dimensional spectrum representing the voice frequency graph changing with time. Its abscissa is time, its ordinate is frequency, and the coordinate point value is voice data energy. Since the spectrogram deploys a two-dimensional plane to express three-dimensional information, the energy value is expressed by the depth of the colors. Firstly, this paper preprocesses the Wav files of the audio section of the IEMOCAP dataset, using the scipy voice processing tool and python_speech_features library to read the speech information and extract the LFCC speech features, and finally converts the ordinary Wav speech signals into LFCC spectrograms.

Optical Flow is the apparent motion mode of two consecutive interframe images caused by the motion of an object or a camera. It is a 2D vector field, where each vector is a displacement vector, representing the motion of the point from frame $X$ to frame $X+1$. In this paper, we deploy the interface of the optical flow estimation algorithm provided in the visual algorithm library OpenCV, including the sparse flow estimation algorithm cv2.Calcopticalflowpyrlk() and dense flow estimation algorithm cv2.calcOpticalFlowFarneback(). Specifically, we deploy the sparse flow estimation algorithm, which is the Lucas Kanade algorithm [1].

In this paper, the I3D [2] is deployed to extract the specific features of the characters in the videos, and the model training method of separating the video stream and the optical flow stream is adopted. I3D (two-stream inflated 3D CNN) [2] is a video action recognition model proposed by Google DeepMind in 2017, using Inception-V1 as the backbone. Its specific network structure is shown in Fig. 2. In Inception-V1, the stride of the first convolution transformation layer is 2. In addition to the parallel maximum pooling layer, there are four maximum pooling layers with a stride of $2\times 2$ and $7\times 7$. The structure of the initial space sub module “Inc.” is shown as Fig. 3.

This paper adopts the form of separate training. One I3D network training takes RGB (video) stream as input, and the other takes FLOW (optical flow) stream as input, carrying optimized and smooth stream information. In this paper, the two I3D networks are first trained, and the pre-trained weights are then loaded during model validation. The models are trained on the IEMOCAP multimodal dataset, and all the video frames have been adjusted to the appropriate size, number and channels before input into the network. Seventy-nine video frames are extracted at uniform time intervals, and the entire video content is captured. Finally, the pre-trained weights of RGB and FLOW modalities are obtained.

This paper first converts audio files into the LFCC spectrograms, and then evolve it into an image classification task. We deploy the independently improved VGG16 denoted as CA-VGG16. We mainly introduce the Coordinate Attention Module (CA module) to improve the performance of VGG16, adding the CA Module after the first, fourth and fifth convolution transformation blocks of VGG16 to improve the model performance. The overall network structure of CA-VGG16 is shown in Fig. 4.

The Coordinate Attention is a novel attention mechanism proposed by Hou et al. [3]. The Coordinate Attention innovatively embeds the precise positional information into the inter-channel attention and then improves the capability of the backbone network to capture the detailed object structures, enrich the semantic information of shallow feature maps and obtain the precise positional information of larger areas.

VGG16 [4] mainly deploys the small convolution filters to build a new convolutional neural network, including the convolution transformation layers, the pooling layers, and the full connection layers.

Convolution Transformation Layer   In the VGG16, it proposes to utilize two $3\times 3$ convolution kernels to replace one specific $5\times 5$ convolution kernel, and deploy three $3\times 3$ convolution kernels to replace one $7\times 7$ convolution kernel. In this approach, we can significantly reduce the parameters of whole network, increase the depth and improve the feature expression ability of the overall network to a certain extent. The calculation formula of the convolution transformation layer are as follows:

Pooling Layer   VGG16 deploys the maximum pooling layers, followed by a maximum pooling layer after several convolution transformation layers to form a block for compressing the shape of feature maps. The general formula for the pooling layer are shown in the following:

From the Eq. 1 to the Eq. 4, where $W$ and $H$ represent the width and height, respectively. $P$, $K$, and $S$ represent the filter size, the convolution kernel size, and the stride, respectively.

Full Connection Layer   In the CNN, the full connection layer is responsible for the task of “Classifier”, which classifies the feature tensors. VGG16 deploys three full connection layers at the end, and the last full connection layer uses the SoftMax layer for classification.

In this paper, we add the CA modules after the first, fourth and fifth blocks of VGG16, denoted as CA-VGG16-3. At the same time, we also attempt to add the CA modules after each block, denoted as CA-VGG16-5. However, the ablation experimental results show that CA-VGG16-3 performs better than CA-VGG-5. Therefore, we select CA-VGG16-3 as the audio feature extraction network, denoted as CA-VGG16. The detailed network structure of CA-VGG16 is shown in Fig. 4.