IMCI: Integrate Multi-view Contextual Information
for Fact Extraction and Verification

Document retrieval includes coarse document retrieval in iteration 1 and refined document retrieval in iteration 2.

Coarse document retrieval aims to quickly obtain most related documents from millions of open-domain Wikipedia documents with as high as possible recall and acceptable precision. Inspired by UKP-Athene Hanselowski et al. (2018) and SR-MRS Nie et al. (2019b), coarse document retrieval is a combination of constituency-based Wikipedia search and TF-IDF retrieval. These two respectively utilize search engine power and statistical word frequency information. For constituency-based Wikipedia search, we also conduct mention filtering like UKP-Athene Hanselowski et al. (2018). That is, if the title of a document is not explicitly mentioned in the claim, then we consider it as weakly related and remove it.

Refined document retrieval aims to retrieve documents with improved performance than coarse retrieval, namely higher recall and also higher precision and F1 score. It adopts dense semantic retrieval and utilizes Wikipedia hyperlinks. Specifically, in iteration 2, we decide refined candidate documents according to corresponding candidate evidences from iteration 1. That is, for each claim, all documents which contain at least one candidate evidence will be taken into account. Furthermore, as top one candidate evidences show pretty high precision (86.11%, in Table 5), we regard them as gold evidence, and take all documents which have hyperlinks with them as refined candidate documents to process multi-hop problem.

Sentence retrieval aims at selecting most related sentences as evidences from candidate documents. In previous models, during sentence retrieval, it is required to design sampling strategy to obtain negative samples for neural retrieval model training. Besides, these models respectively encode and score each claim-sentence pair.

Differently, in our framework, to avoid sampling strategy design and also utilize multi-view contextual information, we encode each sentence within its corresponding intra-document context. Moreover, as mentioned, top one candidate evidences of iteration 1 show pretty high precision (86.11%, in Table 5). Therefore, in iteration 2 we take them as inter-document context, and insert them into the input sequence to process multi-hop problem.

Formally, as shown in Figure 2, the claim, the document title and all the sentences in the document are concatenated:

where sen${}^{\ast }$ denotes top one candidate evidence of iteration 1, which are taken as inter-document context in iteration 2. The sequence is encoded by BERT encoder. For the claim, we take the hidden state of the first claim token as claim representation $E_c$. For the title, we take the hidden state of the first title token as title representation $E_t$. For each sentence, we take the hidden state of the first sentence token as the sentence representation $E_s$. The sentence representation is enhanced through alignment with the title representation:

and the claim representation:

where $\odot$ means element-wise Hadamard product. Then, the score of sentence $\hat{y}$ is obtained through a Multi Layer Perceptron (MLP) with sigmoid activation function:

The training objective of sentence retrieval is defined as binary cross entropy loss, to maximize the probability of groundtruth evidence sentences:

where m is the batch size, $n_i$ is the sentence number of document i, and y is the sentence label, 1 for groundtruth evidence sentences while 0 for non-evidence sentences.

In each iteration, we have scored different sentences as candidate evidences. To obtain better performance, for each claim, we merge the results of different iterations, and rerank the sentences through their scores. Finally, according to the original setup of the task, we keep at most top 5 sentences as evidences, for further fact verification.

For each evidence sentence, we respectively obtain its representations through intra-document encoding and inter-document encoding.

$\bullet$ Intra-document Encoding aims to capture intra-document contextual information of each evidence sentence. It is similar to the sentence retrieval model in Section 2.2. Each evidence sentence is encoded within its intra-document context. Then its intra-document representation is also obtained through alignment.

$\bullet$ Inter-document Encoding is utilized to capture token-level information interaction among different evidence sentences to handle multi-hop problem. The claim, all evidence sentences and their document titles are concatenated as another input sequence:

The concatenation is also encoded by BERT encoder. Then, similarly, we obtain claim, title or evidence representation from the hidden state of the first token. Finally, for each evidence, we obtain its inter-document representation through alignment with the claim representation and its corresponding title representation.

Through multi-view contextual encoding, for each evidence, we can obtain two alignment representations from different contextual views. To further integrate multi-view evidence information to handle multi-hop problem, inspired by multi-relational graph convolutional network Cao et al. (2019); Tu et al. (2019, 2020), we propose dual evidence fusion graph network. As shown in Figure 3, one evidence sentence corresponds to two different nodes in this graph, whose initial representations respectively come from intra-document encoding and inter-document encoding. For each evidence sentence, the noun phrases and named entities are extracted as keywords through spaCy¹¹1https://spacy.io/ tool. Then for a pair of nodes, the links between them are decided according to following rules:

$\bullet$ Common Document Two nodes are linked if they come from the same document.

$\bullet$ Common Keyword Two nodes are linked if they share overlapped keywords.

$\bullet$ Claim Jump Two nodes are linked if they respectively share overlapped keywords with the claim.

For each claim, N selected evidence sentences introduce 2N evidence nodes. Let $H_i$ $\in$ $R^{2N\times d}$ denotes the node representations at i-th graph layer, where $d$ refers to the hidden dimension. The initial representation $H_0$ is the claim-title-evidence alignment representation through multi-view contextual encoding. Updated information $U_i$ $\in$ $R^{2N\times d}$ after a single graph layer is defined as :

where ${\tilde{A}}_j$ $\in$ $R^{2N\times 2N}$ denotes corresponding row normalized adjacent matrix for different kinds of edges. Then the forget ratio $G_i$ $\in$ $R^{2N\times d}$between the updated and old information is:

And the updated evidence representation through the graph layer is:

In this way, with several stacked layers, evidence representations are updated and multi-view evidence information is fused.

Aggregation aims to combines the evidence representations for final inference representation to verify the claim. Among the selected evidence sentences of a claim, some are groundtruth ones while others are not. To utilize evidence label information to enhance fact verification, like Tu et al. Tu et al. (2020), we adopt confidence aggregation.

Formally, let $H_k$ $\in$ $R^{2N\times d}$ denotes evidence representations at the last graph layer. The confidence score of j-th evidence ${\hat{y}}_j$ is obtained from its representation $H_k^j$:

The final inference representation for fact verification $R_v$ is the weighted sum of the evidence representations, where the weights are corresponding confidence scores:

and the fact verification result is obtained through a 3-way classification network:

The total loss consists of the binary cross entropy loss of evidence confidence, and the cross entropy loss of 3-way fact verification:

Here y is the evidence sentence label, 1 for groundtruth evidence sentences and 0 for non-evidence sentences. Besides, v is the fact verification label.

We conduct our experiments on FEVER 1.0 shared task Thorne et al. (2018), which consists of 185,455 annotated claims with 5,416,537 Wikipedia documents from the June 2017 dumps. We adopt the original dataset split of the task, which includes a training set, a development set and an online blind test set. The detailed information is shown in Table 1.

Moreover, for a claim, there exist several groups of evidences, and each group itself is enough to independently verify the claim. To further study the impact of multi-view contextual information, we conduct a refined split on the development set. Specifically, samples of the development set can be divided into 5 parts and the ratio of different parts are displayed in Table 2:

$\bullet$ Single. All evidence groups contain exactly one sentence.

$\bullet$ Single+. At least one evidence group contains only one sentence, and at least one group contains multi sentences.

$\bullet$ Multi. All evidence groups contain exactly two sentences.

$\bullet$ Multi+. All evidence groups contain multi sentences, and at least one group contains more than two sentences.

$\bullet$ NEI. The sample is labeled as NEI with no evidence groups annotated.

Our IMCI is implemented through Pytorch 1.2.0 and our experiments are conducted on a computation node with 4 NVIDIA Titan V GPU. Pre-trained BERT Devlin et al. (2019) encoder is employed for all experiments. We also try RoBERTa encoder Liu et al. (2019) for fact verification. For the claims, we set max length as 64, and claims longer than this will be truncated. For the encoders, we set max input sequence length as 512, and sequence longer than this will be split with stride window size of 128. We utilize BERTAdam optimizer with initial learning rate of 1e-5 and warmup ratio of 0.1. For sentence retrieval, we adopt mini batch size of 4 and gradient accumulation step of 8. In each iteration, we train 2 epochs and select top 5 sentences as candidate evidences. For fact verification, we adopt mini batch size of 1 and gradient accumulation step of 32. For dual evidence graph, we stack 3 graph layers, where the hidden dimension is the same as that of the encoder. In each condition, we randomly start 4 times, train 4 epochs, and choose model parameters with the best performance on the development set.

We adopt FEVER score as the dominant evaluation metric, which is the officially chief metric. FEVER score requests that fact verification label is correctly predicted, and at least one complete group of evidence sentences is found for SUPPORTS and REFUTES samples. The second important metric is label accuracy. For document retrieval and sentence retrieval, we take precision, recall as well as F1 into account. Here, we attach more importance to recall according to the task setting.

Main results on the blind test set are shown in Table 3. With multi-view contextual information joined, our ICMI framework obtains FEVER score of 70.10% and label accuracy of 73.04% with BERT${}_{base}$ encoder. The performance is comparable and even slightly promoted compared with the state-of-the-art one among all baselines with BERT${}_{base}$ encoder. Moreover, our model with RoBERTa${}_{base}$ encoder obtains FEVER score of 72.97% and label accuracy of 75.84%, and shows even higher performance than several baselines with large encoder. These indicate that our framework has made great progress to conduct more accurate fact extraction and verification.

Document retrieval results of different iterations on the development set are displayed in Table 4. With search engine power adopted through Wikipedia search, and statistical word frequency information joined through TF-IDF retrieval, coarse document retrieval obtains the highest recall of 92.77%. Besides, with dense semantic retrieval model guided and top one evidence hyperlinks joined, refined document retrieval shows even higher recall of 95.69%. Furthermore, refined document retrieval has much higher precision of 29.90% and F1 of 45.56%, respectively obtains 21.22% and 29.69% absolute increase than coarse retrieval. Therefore, our iterative fact extraction model has made great improvement on document retrieval.

Moreover, document retrieval results on different parts of the development set are displayed in Figure 4. Coarse document retrieval can handle Single and Single+ samples, where the recall are respectively as high as 97.56% and 98.02%. However, coarse document retrieval fails to handle multi-hop samples, where the recall of Multi and Multi+ samples are both pretty low, respectively 46.82% and 31.31%. Compared to coarse document retrieval, refined document retrieval shows comparable recalls but much higher precision and F1 score on Single and Single+ samples. Furthermore, refined document retrieval makes great progress on multi-hop samples. For Multi and Multi+ samples, refined document retrieval respectively achieves 32.60% and 33.34% absolute increase on recall. Besides, the precision and F1 score also get significantly improved. These results indicate the high efficiency of our iterative multi-view fact extraction model. However, although refined document retrieval has achieved significant improvement, the recall of multi-hop samples is still far lower than single-hop ones.

Sentence retrieval results on the development set are summarized in Table 5. Our IMCI framework obtains the highest recall of 92.86%, and significantly outperforms all baselines.

For sentence retrieval of iteration 1, upstream coarse document retrieval obtains an extremely low precision of 8.68% (in Table 4). Thus, for each claim, on average our sentence retrieval model is requested to distinguish top 5 sentences as candidate evidences from more than 250 sentences. In this condition, with intra-document contextual information joined, the model obtains pretty high recall of 90.30%, and shows comparable performance with state-of-the-art iterative sentence retrieval model HESMSubramanian and Lee (2020). Besides, top one candidate evidences show a pretty high precision of 86.11%. This shows the importance of intra-document context, and is the base of refined document retrieval. Besides, the high precision also guarantees that top one candidate evidences can be considered as inter-document context in sentence retrieval of iteration 2. With multi-view contextual information joined, our sentence retrieval model of iteration 2 obtains even higher recall of 91.98%. Moreover, full pipeline reranking makes the recall get far more increase to 92.86%. These show the great power of multi-view contextual information on fact extraction.

Moreover, sentence retrieval results on different parts of the development set are displayed in detail in Table 6. For Single and Single+ samples, the recall are respectively high at 96.33% and 95.90%, while the precision and F1 score are pretty low. However, the recall of Multi samples is pretty low at 64.41%, while that of Multi+ samples is far lower at 26.26%. Therefore, taking these and the document retrieval results in Figure 4 into consideration, it seems that fact extraction for multi-hop samples is still a difficult problem, although our model has made several progress.

Fact verification results on the development set are shown in Figure 5. With multi-view contextual information joined, our IMCI framework obtains the highest label accuracy of 75.83% and the highest FEVER score of 73.21%. When ignoring intra-document encoding, the label accuracy and FEVER score suffer severe decrease to 74.23% and 71.58%. This indicates the great importance of intra-document contextual information on fact verification. However, compared to intra-document encoding, inter-document encoding and dual evidence fusion graph have relatively weak influence on the performance. These two components mainly aims to handle multi-hop samples. However, multi-hop samples take pretty low ratio (about 6.02% in total in Table 2). Even worse, multi-hop samples have suffered serious performance damage on upstream fact verification task (in Table 5). Evidence confidence aggregation also makes some contribution, indicating the influence of evidence label information. Besides, we also study the influence of fact verification. It seems that progress on fact verification mainly contributes to FEVER score while shows weak influence on label accuracy.

Moreover, for our IMCI framework, the statistic information of prediction errors on fact verification is shown in Figure 6. The framework can correctly distinguish SUPPORTS and REFUTES examples, since SUPPORTS (REFUTES) and REFUTES (SUPPORTS) errors respectively take about 3.66% and 11.51%. This may indicate that the logical boundary between SUPPORTS and REFUTES is relatively clear. Besides, the framework hardly mistakes SUPPORTS examples for NEI examples. However, it may be difficult for the framework to distinguish REFUTES examples from NEI examples, as well as NEI examples from non-NEI examples, for REFUTES (NEI), NEI (SUPPORTS), and NEI (REFUTES) errors respectively take 26.59%, 26.75%, and 20.76%. The situation may be due to the pretty unbalanced label distribution of the training set (in Table 1). Besides, NEI may contain more complex logic semantic than the other two.