Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization

1) Reinforcement Learning in Continuous Spaces: RL is commonly studied following the MDP framework[33], which is defined as a tuple $M=⟨S,A,P,R,\gamma ⟩$, where $S$ is the set of states; $A$ is the set of actions; $P:S\times A\times S\to [0,1]$ is the environment transition probability function; $R:S\times A\times S\to R$ is the reward function, and $\gamma \in [0,1]$ is the discount factor. At each time step $t\in N$, the agent moves from $s_t$ to $s_{t+1}$ with probability $p(s_{t+1}|s_t,a_t)$ after it takes action $a_t$, and receives instant reward $r_t$.

Most RL algorithms rely on the mechanism of policy gradient to handle tasks with continuous state and action spaces[8]. In such cases, the policy is defined as a function $\pi :S\times A\to [0,1]$, mapping each state to a probability distribution of actions, with ${\sum }_{a\in A}\pi \left(a|s\right)=1$,$\forall s\in S$. If only the state space is continuous, the policy can use the Boltzmann distribution to select discrete actions, while when the action space is also continuous, the Gaussian distribution is commonly used. The goal of policy-based RL is to find an optimal policy ${\pi }^{\ast }$ with internal parameter $\theta \in \Theta$ that maximizes the expected long-term discount return

where, the expectation is over the complete trajectory $\tau =(s_0,a_0,r_0,s_1,a_1,r_1,\dots )$ generated following ${\pi }_{\theta }$ until the end of the agent’s lifetime.

In basic policy gradient methods (e.g., REINFORCE[40]), the action-selection distribution is usually parameterized by a deep neural network trained by taking the gradient ascent with the partial derivative of the objective (i.e., maximize the excepted return) with respect to the policy parameters. The policy gradient can be approximated expressed as

where $({\tau }^1,{\tau }^2,\dots ,{\tau }^N)$ is a batch of learning trajectories sampled from policy ${\pi }_{\theta }$ to estimate the return expectation.

2) CRL in Dynamic Environments: Following the convention in [36], in this article, we consider the dynamic environment as an infinite sequence of stationary tasks where each task corresponds to the specific environmental dynamics within its time period and the same dynamics may recur more than once across different time periods. The time period is assumed to be long enough for the agent to get sufficient experience samples to finish policy learning for the associated task. Suppose that there is a space of MDPs denoted as $M$, and a dynamic environment $D$ changing over time in $M$. The CRL agent interacts with $D=[M_1,M_2,\dots ,M_{t-1},M_t,\dots ]$, where each $M_t\in M$ is a specific MDP/task that is stationary in the $t^{th}$ time period, and the identify of each task $M_t$, $t\in [1,2,\dots ]$ is unknown to the agent.

To represent the policies for multiple tasks with a single model, we assume that the observations of the dynamic environment contain the discriminative information of the tasks. Under this assumption, the goal of CRL in dynamic environments in time period $t$ is to extend the acquired knowledge, accumulated from previously learned tasks $M_1\sim M_{t-1}$, to the current task $M_t$, to learn a single policy to achieve the maximum return on all learned tasks $M_1\sim M_t$

where $J_{M_i}$ is the expected return on task $M_i$. Note that the agent is expected to learn a sequence of tasks one by one strategically so that it can retain the previously acquired knowledge when learning new tasks. In other words, the policy $\pi$ with parameter ${\theta }_t^{\ast }$ in (3) is an approximate optimal policy over all learned tasks $M_1\sim M_t$. In this article, the overall performance of the learned policy across all tasks is the primary metric for judging the performance of CRL agents.

The CRP[1] is a discrete-time stochastic process that defines a prior distribution over the cluster structures. The term CRP arises from an analogy of seating a sequence of customers (equivalent to a stream of observations) in a Chinese restaurant with an infinite number of tables (equivalent to clusters) and each table has infinity capacity. Each customer sits randomly at an occupied table with probability proportional to the number of current customers at that table, or at an unoccupied table with probability proportional to a hyperparameter $\alpha$ ($\alpha >0$). The conditional probability for the $t^{th}$ customer sitting at the $k^{th}$ table is

where $z_t$ is the latent cluster label of the $t^{th}$ customer; $k$ is the cluster label and $z_{1:t-1}^{\ast }=\{z_1^{\ast },z_2^{\ast },\dots ,z_{t-1}^{\ast }\}$ represents the cluster labels assigned to the $t-1$ customers, respectively; $m_k^{(t-1)}$ counts the number of assignments to cluster $k$ up to time $t-1$ and $K_{t-1}$ denotes the number of occupied clusters after the $t^{th}$ customer is seated; $\alpha$ serves as the concentration parameter that controls the likelihood of new clusters.

In dynamic environments, it is important to automatically detect and identify the changes of the environment, which can enable specialized policy learning for different tasks, alleviating the catastrophic forgetting. In this article, to characterize the environmental dynamics, we inherit and extend the concept of “context” in [42] where a set of tasks with similar dynamics is regarded as the same context. Under this setting, the detection of environmental changes can be transformed into a context division procedure in the latent space.

However, compared with the single stationary environment investigated in [42], context division in dynamic environments brings significantly more challenges. In particular, the changes are usually infinite and highly uncertain and, without prior knowledge, it is hard to determine in advance the number of contexts that need to be instantiated.

To address this issue, we propose an incremental context detection module in DaCoRL, which can instantiate contexts when necessary in an incremental manner without any prior knowledge. Specifically, we formalize the task of context detection as an online Bayesian infinite Gaussian mixture clustering procedure. To do so, we define a set of environment features to represent the environmental dynamics, and then perform clustering on the features by assuming a CRP prior distribution over the clustering structure. In CRP, the instantiation of new contexts is controlled by the concentration parameter $\alpha$: the larger the value of $\alpha$, the more contexts instantiated in general. On the one hand, when $\alpha =0$, only a single context is instantiated during the learning process. On the other hand, when $\alpha \to \infty$, a new context is instantiated for each task encountered. The complete procedure of incremental context detection with CRP is described in Algorithm 1, which mostly entails the next two steps.

1) Environment Feature Construction (lines 1-2): With the assumption that the task information is contained in the observation space, we can implicitly detect the changes of environmental dynamics by tracking the variation of the observation distribution. In practical implementations, we construct the features for a specific task from the collected observations. Specifically, before the start of RL training in each time period, the agent is required to explore the current environment using a random policy

where $A\left(s\right)$ is the set of available actions in the state $s$ and $Uniform(\cdot )$ is the uniform distribution function. Subsequently, we construct the feature vector ($x_t$) for $M_t$ so that $x_t$ can represent the dynamics of the current environment from the observations $T_{\epsilon }=\{s_0^i,s_1^i,s_2^i,\dots {\}}_{i=1}^m$, where $m$ is the number of trajectories and $i$ denotes the trajectory index.

Here, we approximate the set of collected observations as a Gaussian distribution and use its mean vector as the feature of the current environment. Naturally, we can obtain $x_t$ directly from the original observation space

for the vector inputs, or from the embedding space

for the visual inputs, where $N$ is the number of states contained in $T_{\epsilon }$ and $\varphi \left(s\right)$ can be represented by a random encoder[31] or a standard GAN model[13].

2) Context Detection via Online Incremental Clustering (lines 3-12): To enable incremental context detection in dynamic environments, based on the constructed environment features, we employ online Bayesian inference to update the context models in a fully online fashion, avoiding the necessity for storing previously seen samples. The key step is to estimate the context parameters $\{{\phi }_k{\}}_{k=1}^{K_t}$, which can build up a mapping between specific tasks and context variables in the latent space. In our implementations, the context model (predictive likelihood function) represents environment features using diagonal Gaussian distributions

where ${\mu }_k$ is the mean of the Gaussian used to denote context $k$ and ${\sigma }^2$ is a constant indicating the variance. Under this setting, the context centroids are just the mean vectors of the Gaussian distributions to be estimated: ${\phi }_k=\left\{{\mu }_k\right\}$.

For a sequence of tasks $[M_1,M_2,...,M_{t-1},M_t]$, the first task is assigned to the first context by default. For $M_t$ in the $t^{th}$ time period, we instantiate a new potential context $K_{t-1}+1$, and initialize the new context model parameter ${\phi }_{K_{t-1}+1}^{(t-1)}$ as the feature vector $x_t$, regardless of whether $M_t$ has been encountered before (line 3). After that, the posterior probabilities of the task-to-context assignment over all $K_{t-1}+1$ contexts are estimated (lines 4-5) by

By combining the definition of the predictive likelihood in (8) and the CRP prior distribution in (4), the posterior distribution can be rewritten as

which is abbreviated to $p(z_t=k|z_{1:t-1}^{\ast },x_t)$ in subsequent derivations for simplicity.

With the estimated $p(z_t=k|z_{1:t-1}^{\ast },x_t)$ for each context, we can determine whether to keep the new context for $M_t$ (lines 6-10). If the posterior probability of the potential context is greater than those of the $K_{t-1}$ existing contexts, this new context is instantiated for $M_t$.

Next, based on the inferred posterior probabilities, we update context parameters by optimizing the expected log-likelihood

where $M_t\sim p(z_t=k|z_{1:t-1}^{\ast },x_t)$. Suppose that all context models with the prior parameters ${\phi }_k^{(t-1)}$ have been optimized up to the $(t-1{)}^{th}$ time period. The estimation of $\{{\phi }_k^{\left(t\right)}{\}}_{k=1}^{K_t}$ can be updated based on the gradient (line 11)

where ${\eta }_{k,t}=\frac{1}{m_{k,t-1}+p(z_t=k|z_{1:t-1}^{\ast },x_t)}$ is the learning rate for the $k^{th}$ context in time period $t$. The gradient term can be derived as $p(z_t=k|z_{1:t-1}^{\ast },x_t){\nabla }_{{\phi }_k}\log p(x_t|z_t=k,{\phi }_k^{(t-1)})$.

Based on the updated context parameters, the identity $z_t^{\ast }$ of $M_t$ can be finally obtained by computing an MAP estimate on the predictive likelihood (line 12), selecting the context model that best fits the current environment.

To minimize the interference among contexts in CRL, we opt for an expandable neural network as the policy network in DaCoRL, in which an output head is added synchronously with the newly instantiated context. Each output head specializes on a specific context, and the representation layers are shared among different contexts. This adaptive and expandable network structure can parameterize a dedicated policy for each context without any unnecessary redundancy. In Fig. 2(a), the set of weights of the expandable multihead neural network is denoted by $\theta =\{{\theta }_S,{\theta }_{H,1},{\theta }_{H,2},\dots ,{\theta }_{H,K_t},\dots \}$, where ${\theta }_S$ is a set of weights for shared representation layers, and ${\theta }_{H,k}$, $k\in \{1,2,\dots \}$ are the parameters of the $k^{th}$ output head.

The policy network is initialized as a canonical single-head neural network for the forthcoming learning in the first context. When a new context is instantiated, the neural network is adaptively expanded by adding an output head whose structure is consistent with that of the existing output heads. For the newly added output head $K_t$, we propose the following three practical implementations for parameter initialization.

• Random Initialization: The weights of the newly added output head are initialized to random values. In this case, the policy of the newly instantiated context inherits the representation module of the learned policies, while its output layer is trained from scratch.

• Random Trained Head Initialization: It randomly selects one of the trained output heads and then initializes the weights of the newly added output head to those of the selected one

  $${\theta }_{H,K_t}={\theta }_{H,k},k=Uniform\left(\right\{1,2,\dots ,K_{t-1}\left\}\right).$$

  (13)

  This method enables the new policy to inherit knowledge from the learned context. However, it is likely that the cloned policy may hinder the CRL agent’s ability to properly explore the task space corresponding to the current context, especially when there are significant differences between the two contexts.

• Nearest Trained Head Initialization: It initializes the newly added output head to a specific trained one whose associated context is nearest to the current instantiated context in the latent context space.

  $${\theta }_{H,K_t}={\theta }_{H,k^{\ast }},k^{\ast }=\underset{k}{\arg min}dist({\phi }_k,{\phi }_{K_t})$$

  (14)

  where $k\in \{1,2,\dots ,K_{t-1}\}$ and $dist({\phi }_k,{\phi }_{K_t})$ denotes the distance between contexts $k$ and $K_t$.

Intuitively, the third implementation is most likely to encourage forward transfer. The experimental results and analysis in Section V-F provide further elaboration on this point.

In the policy optimization stage, we integrate the above components with policy-based RL algorithms to achieve efficient CRL in dynamic environments. Furthermore, we employ the knowledge distillation technique [42] to mitigate the interference in the learning in different contexts (corresponding to different output heads in the policy network) caused by the shared low-level representation. This technique can effectively lessen the catastrophic interference caused by distribution drifts among contexts as well as the minor interference due to the differences among tasks within the same context. Taking REINFORCE as the underlying policy-based RL algorithm as an example, the optimization objective of our proposed DaCoRL is derived as follows.

First of all, we rewrite the original loss function of REINFORCE in (1) with the context label variable $z_t^{\ast }$ as

where ${\theta }_{z_t^{\ast }}=\{{\theta }_S,{\theta }_{H,z_t^{\ast }}\}$ and ${\pi }_{{\theta }_{z_t^{\ast }}}$ is the policy corresponding to the context associated with the current task $M_t$.

As a representative of model compression, knowledge distillation can work well for encouraging the outputs of one network to approximate the outputs of another [14, 35]. In DaCoRL, we use it as a regularization term in the probability distribution estimation of actions to preserve the previously learned policies. To construct the distillation regularization term, we regard the policy network from in the last time period as the teacher network, expressed as ${\pi }_{{\theta }^{-}}$, and the current policy network to be trained as the student network, expressed as ${\pi }_{\theta }$. We use the Kullback-Leibler (KL) divergence to constrain the difference between the policies corresponding to the two networks. Thus, the distillation loss of the output head for context $k$ is defined as

where ${\theta }_k=\{{\theta }_S,{\theta }_{H,k}\}$. In the $t^{th}$ time period, considering all $K_t$ output heads in the policy network, the distillation loss term sums up as

where $\theta =\{{\theta }_S,\{{\theta }_{H,k}{\}}_{k=1}^{K_t}\}$ denotes the set of all weights of the policy network.

Finally, to optimize a policy network that can guide the agent to make proper decisions in dynamic environments without being adversely affected by catastrophic forgetting, we combine (15) and (17) to form a joint optimization scheme. Namely, we solve the CRL problem in dynamic environments by the following optimization objective

where $\lambda \in [0,1]$ is a coefficient to control the tradeoff between learning the new policy and preserving the learned policies.

The complete procedure of DaCoRL is summarized in Algorithm 2 where the agent interacts with a dynamic environment $D=[M_1,\dots ,M_t,\dots ,M_T]$. In the first time period $t=1$, the task $M_1$ is instantiated as the first context with parameter ${\phi }_1^{\left(1\right)}$ (line 3). We initialize the CRP model with $K_1=1$, $m_1^{\left(1\right)}=1$ (line 4) and train the single-head policy network from scratch using the canonical policy gradient method (line 5). In the $t^{th}$ ($t\ge 2$) time period, we first apply the incremental context detection module to identify the context to which the current task belongs (line 7), and update the CRP model based on the identity of $M_t$ for future context detection (line 8). Then, we employ an expandable multihead neural network for policy optimization. Specifically, when a new context is instantiated, the policy network is synchronously expanded by adding an output head initialized with the nearest trained head (lines 9-12). Next, the knowledge distillation regularization term is integrated into the loss of the original RL algorithm to reduce the catastrophic forgetting of learned tasks, and the parameters of the policy network are updated till convergence (lines 13-14). Finally, $K_T$ contexts with parameters $\{{\phi }_k{\}}_{k=1}^{K_T}$ and the optimal policy ${\pi }_{{\theta }^{\ast }}$ are obtained for all learned tasks.

1) Robot Navigation[38, 36]: It contains three types of dynamic environments with parametric variation across tasks. In Fig. 3, Types I, II, and III indicate that the dynamic environments are created in terms of parametric variation in the goal position (changes in the reward function), the puddles positions (changes in the state transition function), and both the goal and puddles positions (changes in both the reward function and state transition functions), respectively. In each navigation task, a robot agent needs to move to a goal position within a unit square. The state consists of the agent’s current 2-D position, and the action corresponds to the 2-D velocity commands in the range of $[-0.1,0.1]$. The reward is equal to the negative squared distance to the goal position minus a small control cost that is proportional to the action’s scale. Each learning episode always starts from a given position and terminates when the agent is within $0.01$ from the goal or when the episode length is greater than $100$. We choose these commonly used domains as they are well-understood, suitable for highlighting the mechanism and verifying the effectiveness of our proposed method in a straightforward manner.

2) MuJoCo Locomotion[36]: It contains three locomotion tasks with parametric variation and growing dimensions of state-action spaces, as shown in Fig. 4. These continuous control tasks require a one-legged hopper, a planar cheetah or a 3-D quadruped ant robot to run at a particular velocity along the positive $x$-direction. The reward is an alive bonus plus a regular part that is negatively correlated with the absolute value between the current velocity of the agent and a preset target velocity. The dynamic environment is designed to apply parametric variations in the target velocity within a range: $[0.0,1.0]$ for Hopper, $[0.0,2.0]$ for HalfCheetah, and $[0.0,0.5]$ for Ant. Each learning episode always starts from a given physical status of the agent and terminates when the agent falls down or when the episode length is greater than $100$. We choose these domains to further evaluate the efficiency of our method on more sophisticated domains.

In our experiments, to constitute the continual learning process, we uniformly generate four Gaussian clusters (representing the expected contexts in this article) for each type of the dynamic environment in its parametric variation space. Each of the first three clusters consists of 12 samples and the fourth cluster contains 14 samples, with each sample containing the values of the variable parameters of a specific task. We sequentially arrange these $T=50$ tasks in a random order, resulting in a dynamic environment $D=[M_1,M_2,\dots ,M_T]$. For DaCoRL (Oracle), the supervised version of DaCoRL mentioned in the next section, we make available the task-to-context assignments based on the generation process of dynamic environments. By contrast, the correspondence between tasks and contexts is unknown and needs to be identified by DaCoRL itself. In addition, by reference to the benchmarks in the scenario of learning multiple tasks with a single model in environments with parametric variation [41], we construct observation spaces with extra dimensions of variable parameters for the above dynamic environments to meet the assumption in this article that the observations contain the discriminative information of the tasks.

We evaluate our method in comparison to the following four state-of-the-art baseline methods and one supervised version of our proposed DaCoRL in dynamic environments.

• Naive: It refers to canonical RL methods (e.g., REINFORCE[40], Proximal Policy Optimization (PPO)[30]) that simply train a policy model during the learning process, without paying attention to any possible environmental changes or forgetting.

• CRLUnsup[22]: It detects environmental changes by observing the ability of the agent to perform the task (i.e., the difference between the actual reward and the expected one). When this difference goes below a certain threshold, it triggers the memory consolidation procedure employing Elastic Weight Consolidation (EWC) [20].

• CDKD: It is adapted from the IQ [42] framework that can alleviate catastrophic interference for value-based RL in stationary environments. In this article, we extend IQ to policy-based RL and perform context division and knowledge distillation (renamed as CDKD), and set the true number of contexts in advance so that it can conduct continual learning in dynamic environments.

• LLIRL[36]: It develops and maintains a library that contains an infinite mixture of parameterized environment models for lifelong adaptation in dynamic environments, which focuses on building upon the prior knowledge accumulated during previous learning to optimize the learning parameters to achieve the maximum return in the current environment, without considering the forgetting of previously learned policies.

• DaCoRL (Oracle): This approach can be regarded as a supervised version of our proposed DaCoRL, and the key difference is that the agent is informed in each time period of the specific task-to-context mapping (i.e., all context identifications are made available).

In the experiments, we use policy search with nonlinear function approximation to handle continuous control tasks in dynamic environments. For the robot navigation tasks, we perform gradient updates using the vanilla policy gradient RL algorithm REINFORCE. In addition, we employ PPO as the base RL algorithm to learn in the more challenging MuJoCo locomotion tasks.

We adopt a similar network architecture for all tasks of robot navigation and MuJoCo locomotion. The policy of DaCoRL is approximated by a feed-forward neural network that contains a fully connected hidden layer (with $200$ units) used as the feature extractor and an expandable output head module used as the action distribution predictor, where each output head consists of a $200$-unit fully connected hidden layer and a fully connected output layer. The hidden layers are connected by ReLU nonlinearity, following the network configuration for these tasks in [36]. For a fair comparison, the network architecture of CDKD is set to the same as that of DaCoRL and the number of contexts predetermined for CDKD is also set to be consistent with that automatically detected by DaCoRL. For Naive, CRLUnsup, and LLIRL, each policy network consists of two $200$-unit hidden layers connected by ReLU nonlinearity and a fully connected output layer.

During the learning process, we train the model for $1$k policy iterations on each task, and evaluate the policy performance by testing the current policy on all tasks in the dynamic environment every $100$ iterations. All results reported are the average performance over five independent runs.

Following the convention in previous studies[38, 36], we define two performance metrics to systematically evaluate our proposed method. The first one is the average return over a batch of test episodes on all tasks in the dynamic environment, which is used to evaluate the overall performance of the model on all tasks after a fixed number of policy iterations during training in real time. It is defined as

where $T$ is the number of tasks in the dynamic environment; $m$ is the number of episodes tested for each task; $R\left({\tau }_{ij}\right)$ is the cumulative reward obtained in the $j^{th}$ test episode of task $i$. The other is the average return over all test episodes, which evaluates the average performance of the model over the entire training process. It is defined as

where $J$ is the number of $R_{ave}$ evaluations in the learning process, and $R_{ave}^{\left(i\right)}$ is $R_{ave}$ in the $i^{th}$ evaluation.

Remark 1: For task $M$, the test procedure of DaCoRL is conducted in two steps: 1) Policy selection. It firstly constructs the feature vector $x$ of $M$ following the procedure in lines 1-2 of Algorithm 1, and the identity $z^{\ast }$ of $M$ can be obtained by the MAP estimate of the predictive likelihood of the $K_T$ contexts with parameters $\{{\phi }_k{\}}_{k=1}^{K_T}$ obtained by training. The policy corresponding to the output head $z^{\ast }$ of the trained neural network is the final policy selected for $M$. 2) Policy execution. The agent applies the selected policy on $M$ to evaluate its performance in terms of the cumulative reward in each episode.

To investigate the effectiveness of our proposed method (Q1), we present the results of DaCoRL and all baselines in three types of dynamic environments of the robot navigation tasks. Fig. 5 shows the average episodic returns over all tasks during training according to (19), and Table II reports the numerical results in terms of the average returns over $1000/100\ast 50$ tests throughout the whole training process according to (20). For DaCoRL, the numbers of instantiated contexts are $K_T=5$, $K_T=6$, and $K_T=4$ for the three types of navigation tasks, respectively. In Fig. 5, it is clear that DaCoRL is significantly superior to all baselines in terms of the average test return and the stability. Naive shows the worst forgetting and performance fluctuations since it does not employ any mechanism to overcome context drifts in dynamic environments. CRLUnsup is slightly better than Naive, possibly due to the EWC technique. However, in Fig. 5(b), it still suffers from wild fluctuations in performance, as the changes of environment cannot be detected just by looking at the cumulative reward curve, and learning the new task can overwrite the learned policy without triggering the memory consolidation procedure. CDKD and LLIRL perform clearly better than the above two baselines. Nevertheless, the fixed policy network structure may lead to an over-constrained optimization objective at the early stage of CDKD training, and although LLIRL employs separate neural networks to learn tasks from different context clusters, it does not consider the interference among tasks within the same cluster.

By contrast, DaCoRL achieves superior performance compared with all baselines. Due to the effective context division of environments in the latent space, along with an expandable multihead policy network and the knowledge distillation technique, DaCoRL can achieve competent continual learning of sequential tasks in dynamic environments. In particular, DaCoRL is on a par with [see Fig. 5(c)] or even outperforms [see Figs. 5(a) and 5(b)] DaCoRL (Oracle) where the task-to-context assignments are provided in advance. This results reveal that the incremental context detection module may produce more rational and valuable context division for continual learning than the supervised version with known task-to-context assignments. Furthermore, the average results in Table II indicate that DaCoRL receives significantly larger or comparable average returns over all test episodes during the entire learning process than all baselines. Meanwhile, the statistical results also indicate that DaCoRL generally features smaller standard deviations in performance than the baselines.

To further demonstrate the scalability and flexibility of our method, we evaluate DaCoRL and all baselines on MuJoCo locomotion tasks. The average episodic returns over all tasks during training according to (19) are shown in Fig. 6 and the corresponding average returns over all tests throughout the whole learning process according to (20) are summarized in Table III. It shows that DaCoRL consistently exhibits better and more stable performance than all baseline methods, even in these complex dynamic environments. More specifically, DaCoRL instantiates totally four contexts ($K_T=4$) for each dynamic environment, which is consistent with the given contexts in DaCoRL (Oracle). Since the learning curves of these two methods are largely coincident in all tasks in Fig. 6, it confirms that DaCoRL can accurately identify environmental context changes in a fully self-adaptive manner and can achieve comparable performance with the case where the task-to-context assignments are known in advance.

1) Influence of the Initialization of Output Heads: To address Q2, we evaluate DaCoRL in the robot navigation tasks with different initialization strategies proposed in Section IV-B. The average returns over all tasks during the first $100$ policy iterations are shown in Fig. 7. Here, we shorten the three initialization implementations to “Random”, “Random Head”, “Nearest Head”, respectively, to ensure the readability.

From Fig. 7, it is clear that the initialization by Nearest Head can enable DaCoRL to attain a better initial policy and more positive forward transfer to the new task, which is consistent with our analysis in Section IV-B. Meanwhile, Random Head exhibits obvious positive forward transfer on Type I and Type II environments and negative forward transfer on the Type III environment, compared with Random initialization. Referring to [37, 38], a possible explanation is that, in the Type III environment, Random Head often chooses an initial policy that hinders the exploration of the new task such that the agent needs to spend more time in the early training stage to counter the old policy, resulting in slow performance improvement.

2) Influence of the Number of Instantiated Contexts ($K_T$): Intuitively, instantiating a separate context for each task contained in the dynamic environment (i.e., $K_T=T$, each output head of the policy network corresponds to the policy for an individual task) is likely to result in an optimal policy for the agent. However, it is hard to operate in practice, especially when the dynamic environment comprises a large number of tasks due to the complicated network structure and challenging model training process. Thus, it is necessary to investigate the influence of the number of instantiated contexts on the performance of DaCoRL (Q3).