Semi-Supervised Disentanglement of
Tactile Contact Geometry from Sliding-Induced Shear

The sense of touch is ubiquitous in everyday life. It allows us to interact, explore and manipulate the environment around us using physical contact. The physical contact allows us to measure the environment directly, which is not possible with other sensory modalities like vision. However, this physical contact also results in entangling of relevant information like contact geometry with the manner in which the sensor-stimuli contact is established. For example, when sliding or rubbing fingers across tactile stimuli, soft tactile sensors are inevitably distorted by motion-dependent shear, making their response history-dependent and generalization hard across tactile tasks. This necessitates the removal of motion-induced global shear while preserving the sensor distortions due to the contact-geometry or any other attribute of interest. This issue plagues all shear-sensitive tactile sensors, in particular camera-based soft tactile sensors that use the displacement of markers to encode stimuli attributes [1, 2, 3, 4], complicating tactile-dependent tasks like shape modelling and exploration.

Recently, this shear problem was tackled in [5, 6] for sliding exploration of complex 2D or 3D objects, using a supervised deep learning model to learn mappings between shear-distorted data and the target pose. However, this approach to learn insensitivity to shear requires training separate models for each tactile task, such as contact reconstruction or shape exploration. A study by the authors of this paper instead used a supervised deep learning model to remove shear distortion from the original sheared tactile image so that quantities of interest (e.g. pose) can be predicted from unsheared tactile data in a task-agnostic manner, which is more computational and data efficient [7]. However, because these approaches are supervised, they suffer from costly burden of requiring annotated data for learning. This limitation can be overcome by the use of self- and semi-supervised approaches that can learn rich representations from uncurated/unlabelled datasets, and are thus more labeled sample-efficient. This mode of learning is closer to human learning, as we likewise learn with only small amounts of labelled data and a vast resource of self-supervised experiences.

In this work, we propose a semi-supervised learning approach to remove shear from shear-distorted tactile images that uses only a small subset of annotated data (10%) to achieve comparable performance to previous fully-supervised approaches [7]. Self supervision with tactile data differs from computer vision because it lacks attributes like colour, texture and the large-scale structure of natural image data that have been exploited in computer vision to generate supervisory signals without external supervision  [8, 9, 10]. Moreover, the present work relies only on tactile data without recourse to techniques that correlate data across multiple sensory modalities to generate training signals without external supervision [11, 12, 13] (see Background). In the presence of these limitations, we achieved self-supervision by first disentangling sliding data into its contact-only and shear components, followed by recombining the shear component with paired and novel canonical (non-sheared or vertical tap) data to generate training signals without external supervision.

Our experiments demonstrated that the proposed model could successfully recover contact geometry from sheared images, allow continuous shape exploration and also allow object reconstruction across multiple planar objects. We also found that our proposed semi-supervised model achieved near identical performance to its fully supervised counterpart [7] while using only 10% of labelled samples for training.

Soft optical tactile sensors like the TacTip [1] and other marker-based sensors [2, 4] are vulnerable to sliding-induced global shear that distorts the geometry of contact. Local shear caused by the stimulus imprint on the sensor, represented in the lateral displacement of the markers, is superimposed upon a global shear caused by contact motion such as sliding. This distortion, by masking contact geometry, complicates challenging tactile tasks like shape reconstruction and tactile exploration, which require spatial information about the local geometry of an object free from motion-induced distortion.

Recent works that use TacTip sensor (also used here) addressed this shear problem by using a shear-insensitive pose-prediction deep network to slide around complex planar shapes [5] and complex 3D objects [6, 14]. Shear-insensitive training was achieved by mapping the shear-distorted samples directly to target pose outputs. While effective, this approach necessitated separate training for each new tactile task, for example surface- and contour-following [14], greatly complicating the hyper-parameter tuning for accurate network performance [6], and could not be applied to shape reconstruction. To overcome these drawbacks, the authors of the present paper instead focused on learning a mapping between shear-distorted tactile images and their unsheared (vertical discrete tap) counterparts with a fully-supervised convolutional neural network [7]. Once learned, this mapping can be reused to remove shear for any number of downstream tasks, making this approach computational and data efficient. However, the supervised nature of that approach necessitated complex labelled-data collection. Here, we improve upon that work by proposing a semi-supervised model that achieves comparable performance to the previous supervised baseline, by training first in self-supervised phase followed by a brief supervised phase (with a tenth of the labelled data from [7]).

Tactile data differs from natural images in being low-dimensional due to absence of attributes like color and texture, alongside lacking the large-scale global structure of natural image data. This is primarily due to the localised nature of tactile sensing, unlike vision or audition which are global sensing modalities over a scene. This increases the complexity of generating a supervisory signal in the absence of external supervision; for example, in computer vision, supervisory signals in prior studies have been generated by: 1) learning to predict withheld data dimensions at the output, such as in [8] two sub-networks are trained that are each tasked with predicting one subset of the data channels (color or lightness channel) from another (lightness or color channel); 2) image completion conditioned on the immediate surroundings, as in  [10] in which the input image has a missing part that needs to be predicted at the output based on the surroundings; and 3) predicting the correct arrangement of image subsections, as in  [9] in which the input image is first split into $N$ rectangular tiles whose locations are shuffled to break spatial structure, and the network is then tasked to predict the original image from a shuffled input image. These methods cannot be readily extended to tactile data due to its low-dimensional structure and lack of global structure as discussed previously.

Another approach to generate supervisory signals in absence of external supervision is by correlating data from multiple sensory modalities like audio and vision [11] or touch and vision [12, 13]. In this work, however we rely on the information encoded by the tactile sensor alone.

To generate a supervisory signal, like in  [7], the proposed model learns to disentangle the sensor response components due to contact geometry and motion-induced shear. Our approach, like [7], is inspired from similar works in computer vision where disentanglement of object attributes and factors of variation are considered important for learning robust representation that improve generalizability on out-of-distribution (OOD) samples or novel scenarios [16]. Several works in computer vision learn disentangled representations via varitional autoencoders (VAE) [17, 18] and generative adversarial networks (GAN) [19, 20, 21] for downstream applications such as style transfer between images[22, 19]. Our approach is akin to style transfer where the input image is decomposed into its content (geometry) and style (color, intensity, texture) components. Once the content and style components are disentangled, they can be used to transfer style to novel content. In our work, content is akin to sensor response due to contact geometry and style to response due to motion-induced shear. We are aware of no other prior work exploring semi-supervised learning of disentangled representations for robust representation learning in tactile sensing.

This work proposed a semi-supervised approach that preserved the sensor deformations due to stimulus contact geometry while removing the shear-distortion caused by motion-dependent shear. We showed that the proposed approach achieves comparable performance to its fully supervised counterpart [7] with an order-of-magnitude less annotated data, simplifying data collection considerably.

We validated our approach, similar to [7] by: 1) demonstrating a good match (92.5%) between the model-generated unsheared images (from sliding contacts) and their non-sheared counterpart taken from vertical tapping, using the structural similarity index measure; 2) reconstructing an approximate stimulus contact geometry from model-generated unsheared images that was previously masked by sliding-induced shear (Fig. 3, top row); 3) predicting local object pose from a pose prediction network [7] trained on non-sheared (tapping) data (Table I). This pose prediction was then used for sliding exploration of multiple planar objects (Fig. 3, second row), and so result 4) combined 2) and 3) to reconstruct full object shapes for several planar objects (Fig. 3, third row).

One limitation of our study is that it considered only planar objects. We expect our approach will extend to 3D objects, like recent work that has successfully implemented tactile servoing to slide over complex 3D surfaces (3 pose components) and edges (5 pose components) [5, 6]. An extension would be to sliding shape exploration and full 3D object reconstruction, which are open problem under active investigation [25, 26]. This would open interesting avenues for future research, such as: 1) the fusion of visual and tactile sensory modalities for robust object representations and 2) tactile object recognition by exploiting prior visual experience of the object. An additional limitation is the requirement of annotated data, although much less than the fully supervised counterpart [7]. An unsolved problem for future work is to completely eliminate external supervision. Finally, our training was limited to translational shear even though servo control introduces rotational shear from the angular motion while sliding over objects. That the methods worked well both for sliding exploration and contact geometry reconstruction in the absence of trying to compensate for rotational shear shows that some shear removal carries over from translation to rotational shear. Our expectation is that bringing rotational shear into the training will become important when extending the methods to 3D objects.

Overall, we expect that our methods should apply to various other exploration and manipulation tasks using soft optical tactile sensors adversely affected by motion-induced shear.