Deep Sparse Conformer for Speech Recognition^††thanks: Accepted by InterSpeech 2022.

There are a list of work that proposed new attention mechanisms to replace the $O\left(L^2\right)$ time/space complexities into $O\left(L\text{log}L\right)$ or even $O\left(L\right)$ [Li2021EfficientCS_linear_conformer, wang21ha_interspeech_probsparse]. Motivated by [informer-DBLP:conf/aaai/ZhouZPZLXZ21] for modeling long sequences for time-series forecasting, we adapt the ProbSparse self-attention mechanism to replace the MHSA function in Equation 2. In ProbSparse, the original tensor $\text{Q}\in R^{b\times L\times d}$ ($b$ is batch size, $L$ is input sequence length, and $d$ is hidden dimension. For simplicity, we set $b=1$ and omit $b$ hereafter.) is replaced by a same shape tensor $\overline{\text{Q}}$ which only contains top-$u$ ($=O\left(\text{log}L\right)$) queries under a sparsity measurement $M(\text{q},\text{K})$. That is, each key is only allowed to attend to $u$ dominant queries:

Here, ${\text{q}}_i,{\text{k}}_j\in R^{1\times h}$ are row vectors, and $k({\text{q}}_i\in \overline{\text{Q}},{\text{k}}_j)$ is selected to be an exponential kernel $\text{exp}\left({\text{q}}_i{\text{k}}_j^{\top }/\sqrt{d}\right)$.

The query sparsity measure $M(\text{q},\text{K})$ is motivated by (1) the dominant dot-product pairs encouraging the corresponding query’s attention probability distribution $p\left({\text{k}}_j|{\text{q}}_i\right)$ away from the uniform distribution, and (2) Kullback-Leibler divergence $KL\left(q||p\right)$ measuring the “distance” between $p\left({\text{k}}_j|{\text{q}}_i\right)$ and uniform distribution $q\left({\text{k}}_j|{\text{q}}_i\right)=1/L$. An empirical approximation for efficiently computing the query sparsity measurement for ${\text{q}}_i$ is defined as the following “max-mean” equation:

The range of top-$u$ approximately holds as proved in [informer-DBLP:conf/aaai/ZhouZPZLXZ21]. Under the long tail distribution, it has been empirically testified that randomly sample $U=L\text{log}L$ dot-product pairs to compute the query sparsity measure and then select sparse top-$u$ from it and form $\overline{\text{Q}}$ yielded acceptable results [wang21ha_interspeech_probsparse].

We use the sparse attention function (MHSA-Sparse) defined by Equation 7 and assisted by Equation 8 to replace the original MHSA function used in Equation 2:

Here, the linear projections are trainable parameter matrices ${\text{W}}_i^{\text{Q}}$ and ${\text{W}}_i^{\text{K}}\in R^{d\times d_k}$, ${\text{W}}_i^{\text{V}}\in R^{d\times d_v}$, ${\text{W}}^{\text{O}}\in R^{h{d}_v\times d}$ and $d_k=d_v=d/h$.

Algorithm 1 describes the pseudo-code of a multi-head attention layer using ProbSparse (lines 12 to 19, 21) and relative position encoding (lines 10, 11, 19). One additional linear layer with weight matrix ${\text{W}}^{\text{P}}\in R^{d\times d}$ is introduced to parameterize the memory tensors’ position information. Following [transformer-xl-dai-etal-2019-transformer], two learnable vectors ${\text{U}}_1$, ${\text{U}}_2\in R^d$ are used in shapes of $(h,d_q)$ in line 10 and 11. They are introduced to assist relative position encoding by alleviating attentive bias towards different words in different positions.

Equation 8 suggests an inner-product computation of between a query vector to all the key vectors to determine the importance of the query vector. This blocks the reduction of time complexity. Instead, we only use $L_{\text{K}}^{\prime }$ number of key vectors through sampling from the whole $L_{\text{K}}$ (line 12, 13). This ensures a time complexity of $O\left(L_{\text{Q}}\text{ln}{L}_{\text{K}}\right)$ for computing Equation 8 in line 15. A hyper-parameter $c_1$ (= 5.0) determines the size of the key vector subset.

When the query sparseness measurement is computed, we can easily pick the top-$L_{\text{Q}}^{\prime }$ queries where $L_{\text{Q}}^{\prime }$ is the number of query vectors selected. A hyper-parameter $c_2$ (= 5.0) determines the size of the query vector subset. Line 19 combines the “similarity scores” between the reduced query vector set and the original key and position tensors. Then, the scores are masked by the lengths of sequences in a batch, normalized into probability-style scores under Softmax and finally dropped out with a given probability $p_{dropout}$.

The Conformer-Large defined in [conformer_gulati20_interspeech] contains 118.8M parameters in which there are 17 encoder layers with encoder dimension of 512, 1 decoder layer with decoder dimension of 640. Deeper models alike (1) GPT-3 [gpt3_NEURIPS2020_1457c0d6] model with 175B parameters and 96 transformer layers has achieved impressive scores both for classification and generation tasks through few-shot prompt-based learning, (2) DeepNet [deepnet-wang2022deepnet] with 1,000 transformer layers has shown significantly better BLEU scores [papineni-etal-2002-bleu] in multilingual neural machine translation, compared with shallow architectures. Inspired by these, we aim at enlarging the number of layers in Conformer to as much as one-hundred levels and testifying its representation ability for ASR.

DeepNet [deepnet-wang2022deepnet] combines the good performance of Post-LN and stable training of Pre-LN by leveraging a new normalization function named DeepNorm when performing residual connections [resnet-He2016DeepRL] in a Transformer layer:

In order to ensure the model updating in DeepNet to be constrained by bounds for stable training, $\alpha$ is assigned with various values alike $(2N{)}^{\frac{1}{4}}$ for encoder-only framework (e.g., BERT [bert-devlin-etal-2019-bert]), and $(2M{)}^{\frac{1}{4}}$ for decoder-only framework (e.g., GPT [gpt3_NEURIPS2020_1457c0d6]) where $N$ and $M$ respectively stand for the number of $f(\cdot )$, such as self-attention, cross-attention or FFN sub-layers.

We adapt this DeepNorm function in our deep sparse Conformer. The updated Conformer block is mathematically expressed as:

Here, we utilize ${\alpha }_{1,2,3,4}$, all take a value of $0.81(N^{4}M{)}^{1/16}$ ($N$/$M$=encoder/decoder layer numbers), to scale the temporary output from former sub-layer for residual connection. MHSA-Sparse function is defined in Equation 7 and described in Algorithm 1. Figure 1 depicts the encoder model architecture of our deep sparse Conformer in which one Conformer block is extended into a pipeline of four modules with four residual connections controlled by DeepNorm.

Our experiments are performed under an open-source ASR platform, WeNet¹¹1https://github.com/wenet-e2e/wenet [wenet-DBLP:journals/corr/abs-2102-01547]. We release code and pretrained models²²2https://github.com/Xianchao-Wu/wenet-deep-sparse-conformer. We select a Conformer baseline model consists of a 12-block Conformer encoder ($d_{\text{FFN}}$=2048, $h$=8, $d$=512, CNN${}_{\text{kernel}}$=31) and a 3-block bidirectional Transformer decoder ($d_{\text{FFN}}$=2048, $h$=8, $d$=512) which encodes the textual sequences in left-to-right ($l2r$) and right-to-left ($r2l$) directions. The objective is to minimize $L$, a linear combination of the CTC loss [ctc-10.1145/1143844.1143891] ($\lambda$=0.3) and attention losses (ATT) computed by point-wise KL-divergence [KL-10.1214/aoms/1177729694]:

Label smoothing with $\delta =0.1$ is applied to the attention objective so that the references are discounted by (1-$\delta$).

During training, we select the Adam optimizer [Kingma2015AdamAM] with a maximum learning rate of 0.002. The Noam learning rate scheduler with 30K warm-up steps is used. The models are trained with static batching skill for 120 epochs. The top 30 best checkpoints ranked by validation set losses are averaged to be the final model. For decoding, we use CTC greedy search and Attention rescoring. All our experiments were performed under NVIDIA DGX-A100 with 8*A100-80GB GPUs.

We evaluate deep sparse Conformer variants on the Japanese ‘‘Corpus of Spontaneous Japanese’’ (CSJ) dataset³³3https://ccd.ninjal.ac.jp/csj/en/, which consists of 500 hours of labeled speech. We use sentencepiece-bpe [kudo-richardson-2018-sentencepiece] to segment the text sequences and select a vocabulary size of 4,096. During data preparation, we generate 80-dimension FBank feature vectors with a 25ms window, a 10ms frame stride and dither=1.0. SpecAugment [spec-aug-2019] is adapted with 2 frequency masks ($F$=10) and 2 time masks ($T$=50). Global Cepstral Mean and Variance Normalization [cmvn-10.1016/S0167-6393(98)00033-8] technique is employed to normalize the 80-dimension FBank feature vectors.

Table 1 lists the CERs of five of our deep sparse Conformer (DSC) variants and four baselines. The first baseline is a Transformer with enhanced information from speakers and speech [Li2019ImprovingTS] (refer to Section 2). The second baseline is Espnet’s implementation of deep VGGBLSTM with CTC joint decoding and LM rescoring⁴⁴4https://github.com/espnet/espnet/blob/master/egs/csj/asr1/RESULTS.md. Note that only this baseline uses an external language model among all the nine models. The third baseline is Citrinet [Majumdar2021CitrinetCT] (channel size = 384), a recent pure CNN-based model with 1D time-channel separable convolutions combined with squeeze-and-excitation and has achieved comparable results compared with Conformer. The fourth baseline is Conformer-Large [conformer_gulati20_interspeech] with a bi-decoder.

Our first DSC variant is a Conformer-Small model with the same configurations of Conformer-S [conformer_gulati20_interspeech] with 16 layers except that we use a bidirectional decoder. The other four variants differ only at the encoder part of different layers: 12, 17, 50, and 100. These 5 variants use a bidirectional decoder in which both $l2r$ and $r2l$ decoders have 3 Transformer decoder layers.

We have the following observations. First, Conformer performs well among the baselines, with a relatively large model size. Second, considering that both Conformer and DSC-12en have 135M parameter sizes, we can compare their CERs and training/inferencing time directly. DSC-12en achieved an absolute CER improvement of 1.45%, 0.62% and 0.79% on the three test sets. Also, the training time is only 80% of Conformer’s and the inference speed is 1.23 times faster. These improvements of accuracy and training/inferencing speed reflect the effectiveness of the deep sparse Conformer blocks. Third, using DeepNorm strategy, we could successfully train 50 and 100 layers of Conformer. However, the training time increases significantly: the training time for DSC-17en, DSC-50en and DSC-100en are respectively 1.6, 5.25 and 16.5 times of DSC-12en’s. The accuracies slightly dropped when we enlarged the model parameter sizes. This observation aligns with that reported in [Pham2019VeryDS]. Following [Pham2019VeryDS], we also performed 6-gram LM enhanced ensembles of 30-best outputs from the 5 variants and achieved the best CERs for the 3 test sets: 4.16%, 2.84% and 3.20%.