Active Gaze Control for Foveal Scene Exploration

The main goal of this work is to implement a methodology optimizing the exploration of a scene using foveal vision, to gather as much information as possible about the objects present in the scene in the least number of gaze shifts.

We assume the observing agent to be stationary, analogous to a human not moving the head nor the body, yet to be able to, without any uncertainty, i) shift its gaze to certain parts of the visual field and ii) sense the current gaze direction. The transformation from world to retinal coordinates is defined as a simple translation. The agent’s gaze being fixed at world coordinates $(x_t,y_t)$, an observation at time step $t$ in world coordinates $(x,y)$ corresponds to an observation in local image coordinates $(u_t,v_t)=(x-x_t,y-y_t)$, and the distance $d_t$ to the focal point is given by $d_t=\parallel (u_t,v_t)\parallel$ — the subscript of $d_t$ will be omitted for simplicity, unless required to avoid ambiguity. In Fig. 2, a schematic illustration of the spatial scene representation considered is shown.

After the fixation of the gaze at time step $t$, the image is foveated. The foveation method takes into account the current focal point (center of the fovea) and foveates the full image such that the foveal region keeps the spectral content of the original image and the regions in the periphery are low-pass filters with decreasing bandwidth as the distance from the fovea increases. The used foveation method is described in [1, 9] and code is available online¹¹1github.com/vislab-tecnico-lisboa/laplacian-foveation.

The foveated image then serves as input to an objector detector, which computes a set of $L_t$ detections, denoted as $I_t=\left\{I_t^{l_t}\right\},l_t=1,\dots ,L_t$, where $I_t^{l_t}=(B_t^{l_t},S_t^{l_t})$ is a 2-tuple comprised of the location and size (bounding box) of the $l_t$-th detection, $B_t^{l_t}$, and the respective class confidence scores, $S_t^{l_t}=(s_{t,0}^{l_t},s_{t,1}^{l_t},\dots ,s_{t,K}^{l_t})\in {\Delta }^K$. The set of possible object classes is $C=\{0,1,\dots ,K\}$, where $K$ is the number of object classes and $0$ denotes the background class, i.e., a class representing the nonexistence of an object. In Fig. 3, object detections in a foveated image are depicted: the names and percentages correspond to the class of each detection having the highest confidence score.

Nevertheless, off-the-shelf object detectors do not provide classification scores appropriate for foveated images. As they are trained on regular images, for objects undistorted by foveal vision (objects in the foveal region) the detector output is expected to be similar to that of a regular image; however, for objects in the periphery of the fovea, the detector is expected to output classification scores with increased uncertainty (increased entropy) due to the additional class ambiguity in the blurred part of the visual field. Thus, a Foveal Observation Model learns how the class confidence scores of the detections vary with the position relative to the focal point, allowing to calibrate such scores and providing uncertainty measures of the detections. This approach sidesteps the retraining of the object detector’s neural network, which can be excessively time consuming.

On the other hand, a coarse detection in a foveated image corresponding to the peripheral view of an object will inform the agent of the potential semantic classes that may be in that location and should prompt the gaze to be shifted to that location if the information about the object is still scant, i.e., so far there has been no gaze fixation close to the object. A semantic map would then be improved by a gaze fixation closer to such spacial location. Then, to perform scene exploration efficiently, the detector output, after calibration by the observation model, is merged with previous detection information to update a body-centered semantic map with the knowledge about the scene.

Active Perception can then exploit this updated knowledge to determine the next best viewpoint, i.e., the focal point producing new observations expected to decrease the overall uncertainty about the scene. The gaze is then shifted towards this point, and the image is again foveated to reflect this new choice of focal point. This process of image foveation, object detection, map information update and next best gaze fixation determination is then repeated until a suitable stopping criterion is met.

The Foveal Observation Model should account for the increased uncertainty of each detection due to the foveation process. The detector output scores, $S_t^{l_t}$, when normalized to unit sum, can be seen as a realization of a Dirichlet distribution. Thus, for a given labeled dataset, the parameters of the Dirichlet distributions that characterize the objector detector response to each object class $k\in C$ at each distance $d$ to the fovea, ${\alpha }_{k,d}=\left({\alpha }_{k,d,1},{\alpha }_{k,d,2},\dots ,{\alpha }_{k,d,K}\right)$, can be estimated. The Foveal Observation Model thus consists of sets of Dirichlet distribution parameters — one set of parameter values per object class $k$ and per distance level $d$ (7 different distance levels were considered), and it is trained to learn the probability distribution $p(S|C=k,d)$.

The normalized scores vector $S_t^{l_t}$ corresponding to each new detection $I_t^{l_t}$ being modeled as realizations of the previously trained Dirichlet distributions: $S\sim \text{Dir}({\alpha }_{k,d}$), for each class $k=1,...,K$, a new set of calibrated classification scores $S^{\prime }$ can be computed that more accurately reflect the probabilities of detections in the foveal images:

where $D={\sum }_{k=1}^{K}p(S_t^{l_t}|C=k,d)$ is a normalization factor such that ${\sum }_{k=1}^K{s}_k^{\prime }=1$. The calibrated classification scores $S^{\prime }$ are expected to have less entropy than the original $S$.

The fusion problem consists in, given a set of classification scores for a single pattern, calculating an overall classification score $p$ or estimating a distribution of classification scores. As the agent explores the scene by shifting the gaze to different parts of the visual field, the information of subsequent detections is collected. This information is used to update a map $M$. The $(K+1)$-channel map $M_t(x,y)$ is defined as:

where $p_{t,k}^{x,y}=P(C^{x,y}=k|I_{1:t},x_{1:t},y_{1:t})$ specifies the class posterior probability of class $k\in C$. Thus, $p_t^{x,y}$ constitutes the estimate, at time step $t$, of the class posterior probabilities of an object located at coordinates $(x,y)$.

Given a sequence of classifier scores $S_{1:t}$ for time steps 1 up to $t$, world coordinates $(x,y)$ and distances $d_{1:t}$ to the fovea, a simple product rule (Naïve Bayes) can be used to update the map information, according to

where $Z$ and $Z_t$ are unit-sum normalizing constants.

A categorical distribution is unable to deal with the uncertainty in the estimated parameters $p$ (variance), only with the first moment (expected value). Such uncertainty can, however, be captured by modeling the parameters $p$ as realizations of a Dirichlet distribution, $p\sim \text{Dir}\left(\beta \right)$, $\beta =({\beta }_0,\dots ,{\beta }_K)$. This leads to the definition of a map $B_t(x,y)$ containing the estimates at time step $t$ and for world coordinates $(x,y)$ of the parameters:

These parameters are able to encode the uncertainty — through the second moment or variance — of the class posterior scores $p_{t,k}^{x,y}$ and allow for practical sequential update rules. The class posterior probabilities of $M_t(x,y)$ can be computed from the parameters of $B_t(x,y)$ through

Several strategies for the sequential update of the map $M_t$ exist. In [7], Kaplan et al. proposes several update rules for the fusion of classifier scores: the product rule (equivalent to Naïve Bayes presented), the sum rule, and a novel approach that we denote by Kaplan’s rule.

The application of the sum rule consists in simply adding each of the classifier scores to the parameter of the corresponding class $k$:

Differently from the product rule, in the sum rule the observations are summed instead of multiplied, providing robustness to outliers. On the other hand, the sum rule does not yield a posterior Dirichlet distribution.

In Kaplan’s rule [7], a Dirichlet distribution is fitted to the actual posterior through a moment matching approach. Nevertheless, using this rule with the raw output of the detector would ignore the knowledge about the influenced of the objects location relative to the center of the fovea (Foveal Observation Model). Thus, in order to assess a potential improvement in exploring the scene, a modified version of Kaplan’s rule can consider the calibrated scores $S_t^{\prime }$ of (1) instead of the original detector scores $S_t$:

In order to perform scene exploration in the least number of gazes, and based on the updated knowledge of the map, an Active Perception method chooses the point $(x_{t+1},y_{t+1})$ to which to shift the gaze in the next time step $t+1$, minimizing some measure of uncertainty $F$:

This requires predicting the information accumulated up to time step $t+1$, $I_{1:t+1}$, that would be obtained if the gaze direction were changed to coordinates $(x^{\prime },y^{\prime })=(x_{t+1}^{\ast },y_{t+1}^{\ast })$. As the knowledge about the scene at time $t$ is represented by the semantic map $B_t(x,y)$, we need to compute:

To perform the computation of (8) we need to predict the detector classification scores for each world coordinate $(x,y)$, given that the gaze is shifted to $(x^{\prime },y^{\prime })$. Let $d^{\prime }=||(x-x^{\prime },y-y^{\prime })||$. Then, $p(S^{x,y}|x^{\prime },y^{\prime },B_{t,k}^{x,y})$ can be written as:

where the last step used the fact that $S^{x,y}$ is conditionally independent of $B_{t,k}^{x,y}$ given the object class $k$, and that the object class $k$ at coordinates $(x,y)$ does not depend on the placement of the fovea at time $t+1$. Finally, since $P\left(k|B_{t,k}^{x,y}\right)={\beta }_{t,k}^{x,y}/{\sum }_{i=0}^K{\beta }_{t,i}^{x,y}$:

Given the linearity of the expectation operator:

The predicted classifier scores can then be used to run the data fusion processes (2,5,6) and compute (8).

A measure of uncertainty is associated to the class posterior distribution at each map coordinate, represented as $U(x,y)$. Several measures of uncertainty are considered: the Kullback-Leibler (KL) divergence or relative entropy, the entropy, and the difference between the scores of the two classes with highest confidence.

The KL divergence is used to measure the information gain between the current Dirichlet distribution of parameters $\beta$ and an initial Dirichlet uniform distribution (${\beta }_{0,k}=1,k=0,\dots ,K$):

The entropy of a probability distribution measures the degree of randomness in the random variable. The entropy is used as a measure of uncertainty of the Dirichlet distributions of the map:

The last uncertainty measure is defined as the difference between the two classes of largest expected value. Let $p_{t,i}^{x,y}$ and $p_{t,j}^{x,y}$ be, respectively, the largest and second largest expected class posterior probabilities at map coordinates $(x,y)$. Then, we minimize

The predictions for the map at time $t+1$ being given by (8), the expected values of the uncertainty for all possible next gaze locations $(x^{\prime },y^{\prime })$ are computed through:

Finally, we minimize over all possible next gaze directions $(x^{\prime },y^{\prime })$ the expected uncertainty accumulated over all map coordinates $(x,y)$:

A set of 50 images was randomly chosen from the COCO 2017 validation dataset (80 object classes) as scenes in which to perform visual exploration. The employed Deep Object Detector [11] is pre-trained on the COCO 2017 training dataset. To validate our approach, we have incrementally implemented the pipeline illustrated in Fig. 4. The incremental implementation corresponds to the successive evaluation of each of the three key methods of the approach:

1. Foveal Observation Model: the parameters of each Dirichlet distribution are learned by employing an iterative method [10]; to analyse the effect of class classification scores calibration by the Foveal Observation Model in a simple classification task, the 50 images were foveated at randomly chosen focal points; the match between the detected bounding boxes and the ground-truth bounding boxes was considered valid for an Intersection over Union (IoU) with the ground-truth information greater than 30%; then, for all matches, the classifications induced by the classifier scores (the class corresponding the the maximum classification score) were compared with the ground-truth, with and without the calibration performed by the Foveal Observation model; the entropy of the classification scores was also compared, in both calibration cases.

2. Data Fusion techniques: for each map cell $(x,y)$, there might be none to several detections at a given time step $t$; Let $I_t^{x,y}$ be the set of detections at instant $t$ whose bounding boxes intersect the map cell $(x,y)$; at each time step $t$, the map cell $(x,y)$ update is repeated for every detection belonging to the set $I_t^{x,y}$; the calibrated classification scores by the Foveal Observation Model (1) serve as input to the four fusion techniques: Naïve Bayes update (2), Kaplan Update, Modified Kaplan’s rule (6) and sum rule (5); the next gaze direction was determined randomly instead of through Active Perception methods, in order to evaluate the raw performance of the fusion methods without considering gaze selection; the mean of the accuracy metrics across images was computed.

3. Active Perception techniques: the performance of the different acquisition functions was compared; each of the 50 images has been foveated 10 times, with the foveation points having been chosen by the various active perception techniques; the initial focal point was randomly chosen for each image, being the same for every technique; the detections at each iteration were used to update the information of the map using the Modified Kaplan update (6).