Learning to SLAM on the Fly in Unknown Environments: A Continual Learning Approach for Drones in Visually Ambiguous Scenes

A video showing our continual learning SLAM pipeline is provided at http://tinyurl.com/ycyc5upc.

Simultaneous Localisation and Mapping (SLAM) is a fundamental task for autonomous agents such as surveillance and rescue drones [1]. State-of-the-art SLAM systems either rely on handcrafted feature descriptors [2, 3] or on learning-based representations [4, 5] for template matching and loop closure detection. Compared to handcrafted descriptors, learning-based systems perform better in environments with less features or with high amounts of visual ambiguities (i.e., different scenes that look similar) [5, 6].

Still, most learning-based SLAM systems use traditional deep neural networks (DNNs), where a dataset of the target environment must be available a priori for an offline DNN training phase [5]. Hence, those learning-based systems do not always generalize well to environments unseen during training [4]. This has recently motivated the study of continual learning (CL) for SLAM [4].

In the past years, CL has gained much attention and has been mostly investigated in the context of deep learning,

More recently, Simultaneous Localisation and Mapping (SLAM) has also been proposed as an interesting application for CL [4]. Indeed, when adopting a learning-based SLAM system, a robot trained offline on a dataset acquired beforehand could perform extremely poorly in environments that were not captured by the dataset [4], leading to unreliable and unsafe navigation. Of course, this situation is paramount in the context of search and rescue, which requires robust navigation in extreme and harsh environments [1].

Therefore, to enable the deployment of learning-based SLAM systems in unseen environments, this paper proposes continual Dictionary Learning and Sparse Coding (DLSC), a fast and robust CL system for SLAM, and a low-complexity yet high-performance alternative to DNN approaches. The contributions of this paper are:

1. We propose a new Quadratic Bayesian Surprise (QBS) for modulating DLSC, enabling continual learning.

2. We experimentally demonstrate our continual learning DLSC-QBS method for performing SLAM in the important context of environments not captured by any dataset beforehand, without any offline pre-training.

This paper is organized as follows. First, related work is reviewed in Section II. Next, necessary background is given in Section III. Our methods are provided in Section IV and results are presented in Section V. Finally, conclusions are provided in Section VI.

A number of SLAM architectures have been proposed in the past decades, exploring the use of handcrafted features and learned representations [2, 3, 4, 5].

Related to this work, the popular RatSLAM architecture has been proposed as a bio-inspired system modelled following the navigational processes taking place in the rat hippocampus [2]. In contrast to our proposed system, RatSLAM does not use learned representations, but opts for a simple template matching based on raw RGB data in order to detect loop closures. This simple template matching was later shown to be inefficient in environments with lots of ambiguous views (as in the case of our work) [5, 6].

More recently, the LatentSLAM architecture has been proposed in [5], outperforming RatSLAM through the use of a DNN encoder outputting latent codes for template matching, trained offline as a variational autoencoder. In contrast to LatentSLAM, our CL system does not require any offline training phase, enabling its deployment in unseen environments (not captured by a dataset available a priori).

In this work, we also deviate from most learning-based SLAM systems by using a lower-complexity, unsupervised DLSC learning pipeline [10] instead of conventional DNNs [5, 4]. Similar to LatentSLAM [5], we feed the feature descriptors produced by DLSC to the loop closure detection and map correction back-end of the RatSLAM system [2].

In Section V, we benchmark our system against both RatSLAM and LatentSLAM, and against the use of hand-crafted ORB feature matching [9], extensively used in state-of-the-art SLAM systems [3].

In order to cast (1) into the continual learning case, we use the DLSC reprojection error $||\Phi {\overline{c}}_k-{\overline{s}}_k|{|}_2^2$ in (1) to reduce the coding problem (inference of ${\overline{c}}_k$ by LASSO proximal descent) into a 2-class likelihood test. Indeed, the DLSC reprojection is linked to the Gaussian likelihood $p\left({\overline{s}}_k|\Phi \right)$:

This likelihood can be used to perform an inlier-outlier test:

The test (6) can be seen as a binary classification problem inferring whether the current input ${\overline{s}}_k$ corresponds to the model learned in the dictionary $\Phi$ ($H_0$) or not ($H_1$). This classification is done by testing $p\left({\overline{s}}_k|\Phi \right)$ against a threshold $\theta$. Therefore, we identify two dual problems: A) the original problem that we are seeking to solve (i.e., learning $\Phi$ in a continual manner in order to infer $\overline{c}$) and B) its associated dual problem as learning $\Phi$ in a continual manner in order to classify $\overline{s}$. Hence, by casting the dual problem B) to the continual learning case, we implicitly cast the original problem A) to the continual learning case as well.

In order to transpose B) to the continual case, we must provide robustness to the severe data imbalance caused by the non-stationary stream ${\overline{s}}_k$ (due to the fact that ${\overline{s}}_k$ does not originate from a shuffled dataset where realisations are assumed to be i.i.d, see Section III-B). Therefore, we propose to solve the binary classification problem B) using the well-known Support Vector Classification (SVC) framework [12]:

where $k^{\ast }$ is the index of the current time step in the data stream, $\theta$ is the threshold used in (6), and ${\xi }_j$ are the slack variables defining the margin of the separation hyperplane.

The SVC formulation in (7) is indeed a natural choice when dealing with class imbalance since in an SVC, all data points are not contributing to the synthesis of the decision hyperplane [12] (the contribution is only due to a few support vectors). This prevents SVC over-fitting on the redundant data found in our heavily imbalanced video streams.

We can then cast (7) into its hinge loss formulation [12]:

where the hinge loss $max(0,x)$ leads to the sparsity property of SVCs, which in turn leads to a large number of input vectors ${\overline{s}}_k$ not playing any role in the synthesis of the SVC hyperplane. This provides the robustness to data redundancy.

In our particular case, we seek to continuously maintain a dictionary $\Phi$ such that all incoming data points are well modelled by this dictionary. Therefore, the generic 2-class formulation in (8) reduces to (9) in our case since all data points must be associated to the $H_0$ class to be inliers [13]:

Using (9) in a probabilistic context, we can define the Bayesian surprise factor $S_1$ associated with the incoming data point ${\overline{s}}_{k^{\ast }+1}$ as the posterior to prior ratio [14]:

where $S_1$ is proportional to the negative log-likelihood via the Bayes theorem. In addition, the formulation in (10) assumes that the negative log-likelihood $-\log p\left({\overline{s}}_{k^{\ast }+1}|{\Phi }_{k^{\ast }}\right)$ is proportional to the hinge loss energy term in (8) under the SVC formulation (9). The surprise factor $S_1$ is null when the hinge loss associated with the incoming data point ${\overline{s}}_{k^{\ast }+1}$ is zero, denoting that the data point is already well modelled by the dictionary $\Phi$ and hence, does not lead to any surprise.

Still, using $S_1$ in (10) is inconvenient since it requires the knowledge of the likelihood threshold $\theta$. This can be an even greater problem since an optimal choice of $\theta$ could slowly vary over time due to changes in the environment. Under the generic assumption of a high-enough camera frame rate (4), we now show that having an explicit knowledge of $\theta$ can be eliminated by introducing a second-order surprise factor $S_2$:

which is related to the discrete-time second derivative ($\frac{{\delta }^2}{\delta k^2}$) of the posterior density evolution through time. We thus refer to $S_2$ as a Quadratic Bayesian Surprise (QBS) factor.

Intuitively, $S_2$ can be understood as the curvature of the posterior density evolution through time, indicating how much a newly received input vector ${\overline{s}}_{k^{\ast }+1}$ will accelerate or decelerate the change in posterior $p\left({\Phi }_k|{\overline{s}}_k\right)$.

Dropping proportionality constants and using (10) in (11):

where $e_{k^{\ast }+1}$ and $e_{k^{\ast }}$ are the current and past reprojection errors. Four possible outcomes a,b,c,d are identified in (12). Under the high frame rate assumption (4), the probability of outcomes b and c in (12) will be very low since (4) induces that if $e_{k^{\ast }}-\log \theta \le 0$ then $e_{k^{\ast }+1}-\log \theta \le 0$ must likely be the case as well, and vice versa. In addition, outcomes a and c in (12) can be merged under the assumption (4):

while still observing the conditions $e_{k^{\ast }+1}-\log \theta \le 0,e_{k^{\ast }}-\log \theta \le 0$ of a) in (12). Therefore, the QBS in (11) can simply be computed as follows:

In practice, we estimate the QBS (14) by low-pass filtering $S_2$ using a moving average of width $5$ to attenuate noise.

We can now use the QBS $S_2$ to modulate SGD learning in an online manner using the following dictionary update procedure (with learning rate ${\eta }_d$) in Algorithm 1 (lines 6):

Therefore, by ignoring the effect of redundant data associated to a negative QBS factor, we prevent the over-fitting of the dictionary learning to the local contexts in the video streams. In turn, this leads to the learning of a more diverse set of features in $\Phi$ and consequently, to lower reprojection errors during the replay of the input sequence (see Fig. 4).

In addition, we observed during our experiments a useful stability property induced by the use of the QBS in our continual dictionary learning scenario. Indeed, it is known that DLSC problems can suffer from instability [15] and we have observed run-away behaviors as well for cases where the QBS was not being used. In contrast, we did not observe any instability issue when using our QBS-modulated learning rule (15) in Algorithm 1, suggesting a regularizing effect (see Fig. 4 b, where using the QBS induces stability).

We evaluate the Mean Absolute Error (MAE) of our DLSC-QBS SLAM approach (see Fig. 1) against the ground truth localisation and mapping, acquired by UWB indoor positioning beacons [8]. As DLSC parameters, we use ${\eta }_c=5\times {10}^{-3}$, ${\eta }_d=1.4\times {10}^{-3}$, ${\lambda }_1=0.2$, $N_c=10$, $N_d=1$, $M=64$ in Algorithm 1. These parameters were tuned empirically in order to optimise SLAM performance.

Three different flight sequences are used for performance assessment, with increasing difficulty:

• Fligh 1: Flying between wall and shelves, which makes the continual learning and loop closure detection easier since the visited environment is more diverse and hence, easier to disambiguate (blue path in Fig. 3).

• Fligh 2: Flying between shelves only, which leads to a larger amount of similar and ambiguous views compared to flight 1 (red path in Fig. 3)

• Fligh 3: Combining flights 1 and 2 (both red and blue paths in Fig. 3 are visited by the drone).

Since the SLAM result and the ground truth do not share the same coordinate system [2], the ground truth coordinates are projected to the SLAM domain via translation and rotation. The translation vector and rotation angle are found via grid search, by minimizing the localisation MAE.

The localisation MAE is computed as the average error between the SLAM localisation $(x_k,y_k)$ and the ground truth $(x_k^{\text{gt}},y_k^{\text{gt}})$ on the flight sequence of length $T_{end}$:

In addition, the mapping MAE is computed as the a posteriori deviation between the map obtained at the end of the SLAM process and all the ground truth points:

Table I quantitatively compares the performance of our approach against i) the original RatSLAM [2]; ii) a VGG-11 pre-trained on ImageNet [16] as feature encoder to RatSLAM; iii) the LatentSLAM, trained specifically for our environment following [5], on a set of 3998 frames acquired independently from flights 1-3; and iv) ORB features for template matching [9] following the Lowe’s Ratio Test [17].

For the ORB features, the similarity between two frames $i,j$ is computed as $s_{ij}=1-N_m/{max}_m$ where $N_m$ is the number of matches and ${max}_m$ the maximum number of possible matches. For VGG-11, LatentSLAM and DLSC-QBS, the similarity is computed as $s_{ij}=\frac{{\overline{c}}_i^T{\overline{c}}_j}{||{\overline{c}}_i|{|}_2||{\overline{c}}_j|{|}_2}$ with ${\overline{c}}_{i,j}$ denoting the feature descriptor obtained by each method.

The RatSLAM back-end integrates the raw odometry and detects loop closures by sampling latent codes from the data stream each $100$ms and by comparing the similarities $s_{ij}$ against a threshold $\mu$ tuned for minimum MAE (see Fig. 1).

Fig. 5, 6 and 7 show the SLAM trajectories obtained by each method, compared to our proposed DLSC-QBS system. As expected by prior work [5, 6], the original RatSLAM system does not perform well in our ambiguous warehouse environment since its loop closure detection approach, based on the matching of raw image data, is too sensitive to the ambiguities between different views (see Fig. 3).

The pre-trained VGG-11 (deep architecture with 11 layers) used as feature descriptor performs better than the original RatSLAM for Flight 3 (see Fig. 7), but does not perform well overall. This confirms the observations in [5] where finding a good template matching threshold $\mu$ was shown to be hard with off-the-shelf DNNs.

Latent-SLAM is among the top performers in Table I, but requires the offline training of an 11-layer DNN on a dataset from the specific environment where SLAM must be performed. This makes LatentSLAM not suited for navigation in unknown environments that are not captured in datasets available beforehand, which was our goal in this work.

The use of handcrafted ORB features, extensively used for SLAM [3], performs well in between the storage aisles (e.g., view 2 in Fig. 3) but can also perform poorly at the end of the aisles, where lighting conditions suddenly change, leading to unreliable feature matching. Still, the SLAM back-end is able to recover the track in most cases (see Fig. 5, 6) but can also fail when, at the same time, the drift in raw odometry becomes too unreliable (see Fig. 7). This leads to large errors during flight 3 in Table I.

On the other hand, our continual learning DLSC-QBS method either outperforms or either reports close MAE performance to both the pre-trained LatentSLAM and the use of ORB features, while a) having a complexity similar to a 1-hidden-layer network; b) not requiring any pre-training and c) not suffering from track loss issues encountered with ORB features when lighting conditions suddenly change as the drone exits the aisles (see Fig. 7). Therefore, our approach might be a promising avenue for safety-critical navigation in unknown places where a dataset is not available beforehand.

This paper has presented what is, to the best of our knowledge, one of the first continual learning SLAM systems. Our method has been experimentally validated by performing SLAM with a drone in a challenging and visually ambiguous warehouse environment, without any model pre-training, while reporting competitive performance compared to prior systems. We hope that this work will contribute towards safer indoor drones that can adapt to new environments on the fly.

We thank Prof. J. Suykens for discussing the QBS formulation of Section IV, and Dr. L. Keuninckx for his support.