Domain Adaptation from Scratch

Natural Language Processing (NLP) algorithms have made impressive progress in recent years, achieving great success in various tasks and applications. However, these algorithms may require substantial labeled data to reach their full potential. Such data is often too costly and labor-intensive to obtain. Furthermore, modern systems are required to perform well on data from many domains, while for most tasks, labeled data is available for only a few. Consequently, NLP algorithms still struggle when applied to test examples from unseen domains (dilbert; pada; docogen).

A common strategy for annotating data is through crowdsourcing platforms, e.g., Amazon Mechanical Turk, where data can be annotated on a large-scale, as one can harness a large workforce at affordable prices. However, crowdsourcing is not an option for sensitive data (e.g., in medical and military applications). For instance, virtual assistants such as Alexa, Siri, and Google Home serve millions of users. Annotating data from virtual assistants’ recorded conversations would be optimal input for training applicable NLP models for their purposes. However, crowdsourcing is not a valid option for such sensitive data as it does not preserve users’ privacy. Hiring annotators internally is expensive and often hard to scale efficiently.

This paper examines different approaches to annotating and using data from non-sensitive sources (NSS) when target domain data can not be annotated. More precisely, we experiment with several methods (§3) to train a model for a sensitive target domain under a given annotation budget of examples from NNS. Various methods aim to ease the burden of data annotation. For example, domain adaptation (DA) algorithms are trained on source domain training data to be effectively applied to other target domains. Several setups have been explored in the DA literature, differing in the availability of target domain supervision. This paper focuses on unsupervised DA (UDA), i.e., where no labeled data is available from the target domain. Still, unlabeled data can be freely obtained from both source and target domains. Unlike previous works on UDA ziser-pblm; perl, trying to train the best model for a given labeled source data, we are interested in efficiently annotating the source data. Hence, we perform domain adaptation from scratch.

Another prominent approach for coping with limited annotation resources is Active Learning (AL, (DBLP:journals/jair/CohnGJ96)). AL algorithms automatically choose a subset of unlabeled samples from a massive unlabeled corpus, maximizing the value of the labeling process. Such algorithms have shown effectiveness for deep neural network (DNN) modeling across various NLP tasks (DBLP:conf/acl/DuongAEPCJ18; DBLP:conf/conll/PerisC18; DBLP:conf/emnlp/Ein-DorHGSDCDAK20). The typical assumption in the AL literature is that AL algorithms choose the most valuable subset to annotate under the assumption of being tested on data from the same distribution. This paper examines their usefulness in improving model performance when applied to unknown target domains.

As DA and AL are highly popular approaches to ease the problem of insufficient annotation budget, combining the two is natural rai2010domain; saha2011active. Most works in this intersection share a similar setup, in which labeled data from a source domain facilitates the annotation process in the target domain (li2013active; wu2017active). xiao2013online consider data annotation to be cheaper in the source domain compared to the target domain. We consider this work the most similar to ours since it recognizes the cost of annotating samples from the source domain for improving in the target domain. However, annotating samples from the target domain is unavailable in our setup.

We propose and evaluate various methods to tackle the challenging “DA from scratch” setup, focusing on two tasks: Binary sentiment classification and named entity recognition. Going beyond existing methods, we propose several methods for integrating domain knowledge into AL systems and demonstrate encouraging results: Combining the two is better than applying them separately. We hope to inspire future work in this direction.

The goal of a DA algorithms is to learn a model $m$ that generalizes well from a set of source domains $\{D_{S_i}{\}}_{i=1}^K$ to a target domain $D_T$, for a given task. We assume that unlabeled data from the source and target domains are available. We denote the corresponding unlabeled sets with $\{U_{S_i}{\}}_{i=1}^K$ and $U_T$ respectively. Initially, no labeled data exists, and since data from the target domain is sensitive, we cannot annotate it. Thus, we consider an approach for automatically choosing $L$ source domain examples to annotate in each iteration. We choose examples that would best benefit a model applied to test data that originates from the target domain.

Table 1 outlines our methods, which are divided into four branches: (1) Supervised learning with naive labeling (§3.1); (2) active learning methods (§3.2); (3) domain adaptation approaches (§3.3); and (4) methods that combine domain adaptation with AL (§3.4).²²2There are two folk wisdom definitions of AL. The first clings to sample selection while the second advocates interactivity. For simplicity, we follow the first definition. We implement all our algorithms with a pretrained BERT model (bert).

We focus on two common NLP tasks, (binary) sentiment analysis (SA) and named entity recognition (NER). For SA, following a large body of prior work (ziser-reichart-2018-deep; ziser19; perl), we include four product review domains (blitzer2007biographies) - Books (B), DVDs (D), Electronics (E) and Kitchen appliances (K); and an additional airline review dataset (A, Nguyen2015airline). For NER, we experiment with the OntoNotes 5.0 dataset (ontonotes), including six different domains: Broadcast Conversations (BC), Broadcast News (BN), Magazine (MZ), Newswire (NW), Telephone Conversation (TC), and Web (WB). We report the statistics of our experimental setups in §B.⁴⁴4URLs of the datasets, code, implementation details, and hyperparameter configurations are described in § B.

We choose a single target domain that is held during training and used only for testing. Accordingly, all other domains form our source domains and are used for training. We repeat the experiments in each task to use each domain as the target domain. We start each IS algorithm by randomly sampling $M$ examples and annotating them ($M=50$ and $M=250$ for SA and NER, respectively). Afterward, we choose $M$ examples to annotate in each iteration according to the method in use. We report accuracy scores for sentiment classification⁵⁵5We use balanced test sets for sentiment analysis. and macro-F1 scores for NER. All reported scores are averaged across five different seeds. To conclude the accumulated results across different iterations, we report the AUC score for each model.

Table 2 presents our main results. In Figure 1, we present the results of two representative setups for each task, with a breakdown to iterations. As our results indicate, methods that integrate DA and AL, which is a novel contribution of this paper, performs best on both tasks.

For both tasks, Figure 1 demonstrates the effectiveness our methods compared to Vanilla (AL, DA, and their combination) across (almost) all active learning iterations. In NER, the trend is clear, AL methods improve upon random annotation methods (e.g., Certainty compered to Vanilla), and DA approaches improve upon non-DA approaches (e.g., DTvs. Vanilla). Importantly, we observe a complementary effect when combining AL and DA, with Cert-TAPAD and Cert-DT outperforming all other methods, with an average improvement of $2.0\%$ and $1.2\%$ (AUC) compared to Certainty (their AL-based baseline) and an average improvement of $4.2\%$ and $4.6\%$ (AUC) compared to TAPAD and DT (their DA baselines). For SA, while UDA methods prove to be highly efficient (e.g., consider Vanilla vs. DT), AL gains are inconsistent. On the one hand, Smoothing and MC-Dropout improve Vanilla by $0.6\%$ and $0.8\%$, respectively, on average. On the other hand, Vanilla and Certainty are equivalent. Nevertheless, it is Selection and Cert-DT, which combine AL and DA, that perform best, with an average improvements of $7.3\%$ and $3.6\%$, respectively, on top of the vanilla baseline, Vanilla.

Finally, we notice a limitation of the Selection method, exhibiting dramatic performance drop in the NER setups. We analyze this behaviour in §A.

This paper presents a new challenging learning setup where annotating data from the desired target domain is not feasible, and annotating data from available source domains is costly. We experimented with various prominent approaches within this setting, mainly from the active learning and domain adaptation algorithm families. We presented novel ways of combining the two, yielding the best performance. We hope our findings will encourage further research on this setup, which we believe is common for real-world applications and should get more attention from the research community.