MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment

In the early years of PCQA development, some simple point-based FR-PCQA methods are proposed by MPEG, such as p2point [23] and p2plane [34]. To further deal with colored point clouds, point-based PSNR-yuv [35] is carried out. Since the point-level difference is difficult to reflect complex distortions, many well-designed FR-PCQA metrics are proposed to employ structural features and have achieved considerable performance, which includes PCQM [25], GraphSIM [46], PointSSIM [2], etc.

To cover more range of practical applications, some NR-PCQA methods have been proposed as well. Chetouani $et$ $al.$ [9] extract patch-wise hand-crafted features and use classical CNN models for quality regression. PQA-net [21] utilizes multi-view projection for feature extraction. Zhang $et$ $al.$ [51] use several statistical distributions to estimate quality-aware parameters from the geometry and color attributes’ distributions. Fan $et$ $al.$ [13] infer the visual quality of point clouds via the captured video sequences. Liu $et$ $al.$ [22] employ an end-to-end sparse CNN for quality prediction. Yang $et$ $al.$ [45] further transfer the quality information from natural images to help understand the point cloud rendering images’ quality via domain adaptation. The mentioned methods are all based on single-modal information, thus failing to jointly incorporate the multi-modal quality information.

Many works [4, 31, 8] have proven that multi-modal learning is capable of strengthening feature representation by actively relating the compositional information across different modalities such as image, text, and audio. Afterwards, various works utilize both point clouds and image data to improve the understanding of 3D vision, which are mostly targeted at 3D detection. Namely, AVOD [19] and MV3D [5] make region proposals by mapping the LiDAR point clouds to bird’s eye view. Then Qi $et$ $al.$ [29] and Wang $et$ $al.$ [39] attempt to localize the points by proposing 2D regions with 2D CNN object detector and then transforming the corresponding 2D pixels to 3D space. Pointpainting [37] projects the points into the image segmentation mask and detects 3D objects via LiDAR-based detector. Similarly, PI-RCNN [42] employs semantic feature maps and makes predictions through point-wise fusion. 3D-CVF [47] transfers the camera-view features using auto-calibrated projection and fuses the multi-modal features with gated fusion networks. More recently, some works [1, 50] try to make use of multi-modal information in a self-supervised manner and transfer the learned representation to the downstream tasks.

Suppose we have a colored point cloud $P=\{g_{\left(i\right)},c_{\left(i\right)}{\}}_{i=1}^N$, where $g_{\left(i\right)}\in R^{1\times 3}$ indicates the geometry coordinates, $c_{\left(i\right)}\in R^{1\times 3}$ represents the attached RGB color information, and $N$ stands for the number of points. The point cloud modality $\hat{P}$ is obtained by normalizing the original geometry coordinates and the image modality $I$ is generated by rendering the colored point cloud $P$ into 2D projections. Note that $\hat{P}$ contains no color information.

It is natural to transfer mainstream 3D object detectors such as PointNet++ [30] and DGCNN [38] to visual quality representation of point clouds. However, different from the point clouds used for classification and segmentation, the high-quality point clouds usually contain millions of dense points, which makes it difficult to extract features directly from the source points unless utilizing down-sampling. However, the common sampling methods are aimed at preserving semantic information whereas inevitably damaging the geometry patterns that are crucial for quality assessment. To avoid the geometry error caused by the down-sampling and preserve the smoothness and roughness of local patterns, we propose to split the point cloud into several local sub-models for geometric structure feature representation. Specifically, given a normalized point cloud $\hat{P}$, we employ the farthest point sampling (FPS) to obtain $N_{\delta }$ anchor points $\{{\delta }_m{\}}_{m=1}^{N_{\delta }}$. For each anchor point, we utilize K nearest neighbor (KNN) algorithm to find $N_s$ neighboring points around the anchor point and form such points into a sub-model:

where $S$ is the set of sub-models, $KNN(\cdot )$ denotes the KNN operation and ${\delta }_m$ indicates the $m$-th farthest sampling point. We use $N_{\delta }\times N_s>N$ to ensure that the sub-models cover sufficient number of points in the source point cloud. An illustration of the patch-up process is exhibited in Fig. 3. It can be seen that the local sub-models are capable to preserve the local patterns. We randomly select $N_P$ sub-models from the generated $N_{\delta }$ sub-models for geometric feature extraction. Then a point cloud feature encoder ${\theta }_P(\cdot )$ is employed to map the selected sub-models to quality-aware embedding space:

where $F_P^l\in R^{1\times C_P}$ indicates the quality-aware embedding for the $l$-th selected sub-model $S_l$, $C_P$ represents the number of output channels of the point cloud encoder ${\theta }_P(\cdot )$, and ${\tilde{F}}_P\in R^{1\times C_P}$ is the pooled results after average fusion.

Inspired by the success of PCQA methods via utilizing the image modality [21, 44], we propose to use a 2D image encoder for feature extraction. $N_I$ image projections are rendered from the distorted colored point clouds from random viewpoints but with a fixed viewing distance to keep the texture consistent. Considering that the perceived texture visual quality varies at different scales [40, 33], we propose to use hierarchical 2D CNN backbones to make use of multi-scale quality-aware features to better model the human perception of texture quality. Assume that the 2D CNN backbone has $j$ layers, the hierarchical features can be described as:

where ${\theta }_I(\cdot )$ represents the hierarchical image encoder, $\oplus (\cdot )$ indicates the concatenation operation, $GAP(\cdot )$ represents the global average pooling operation, $L_k\left(x\right)$ stands for the feature maps obtained from $k$-th layer, and ${\alpha }_k\left(x\right)$ denotes the average pooled features from $L_k\left(x\right)$. Then we embed the rendered 2D images into quality-aware space with the hierarchical 2D image encoder:

where $F_I^t\in R^{1\times C_I}$ denotes the quality-aware embedding for the $t$-th image projection $I_t$, $C_I$ represents the number of output channels of the 2D image encoder ${\theta }_I(\cdot )$, and ${\tilde{F}}_I\in R^{1\times C_I}$ is the pooled results after average fusion.

As shown in Fig 1, the single modality may be incomplete to cover sufficient information for quality assessment, thus we propose a symmetric attention transformer block to investigate the interaction between the visual quality features gathered from the point cloud and image modalities. Given the intra-modal features ${\tilde{F}}_P\in R^{1\times C_P}$ and ${\tilde{F}}_I\in R^{1\times C_I}$ from the point clouds and images respectively, we adjust them to the same dimension with linear projection:

where ${\hat{F}}_P\in R^{1\times C^{\prime }}$ and ${\hat{F}}_I\in R^{1\times C^{\prime }}$ are adjusted features, $W_P$ and $W_I$ are learnable linear mappings, and $C^{\prime }$ is the number of adjusted channels. To further explore the discriminate components among the modalities, the multi-head attention module is utilized:

where $\Gamma (\cdot )$ indicates the multi-head attention operation, $\beta (\cdot )$ represents the attention function, $h_{\mu }$ is the $\mu$-th head, and $W$, $W_Q$, $W_K$, $W_V$ are learnable linear mappings. As illustrated in Fig. 4, both point-cloud-based and image-based quality-aware features participate in the attention learning of the other modality. The final quality embedding can be concatenated by the intra-modal features and the guided multi-modal features obtained by the symmetric cross-modality attention module:

where $\oplus (\cdot )$ indicates the concatenation operation, $\Psi (\cdot )$ stands for the symmetric cross-modality attention operation, and ${\hat{F}}_Q$ represents the final quality-aware features ($\Psi ({\hat{F}}_P,{\hat{F}}_I)\in R^{1\times 2C^{\prime }}$, ${\hat{F}}_Q\in R^{1\times 4C^{\prime }}$).

Following common practice, we simply use two-fold fully-connected layers to regress the quality features ${\hat{F}}_Q$ into predicted quality scores. For the quality assessment tasks, we not only focus on the accuracy of the predicted quality values but also lay importance on the quality rankings [14, 16]. Therefore, the loss function employed in this paper includes two parts: Mean Squared Error (MSE) and rank error. The MSE is utilized to keep the predicted values close to the quality labels, which can be derived as:

where $q_{\eta }$ is the predicted quality scores, $q_{\eta }^{\prime }$ is the quality labels of the point cloud, and $n$ is the size of the mini-batch. The rank loss can better assist the model to distinguish the quality difference when the point clouds have close quality labels. To this end, we use the differentiable rank function described in [41, 32] to approximate the rank loss:

where $i$ and $j$ are the corresponding indexes for two point clouds in a mini-batch and the rank loss can be derived as:

Then the loss function can be calculated as the weighted sum of MSE loss and rank loss:

where ${\lambda }_1$ and ${\lambda }_2$ are used to control the proportion of the MSE loss and the rank loss.

To test the performance of the proposed method, we employ the subjective point cloud assessment database (SJTU-PCQA) [44] and the Waterloo point cloud assessment database (WPC) proposed by [20] for validation. The SJTU-PCQA database includes 9 reference point clouds and each point cloud is corrupted with seven types of distortions (compression, color noise, geometric shift, down-sampling, and three distortion combinations) under six strengths, which generates 378 = 9$\times$7$\times$6 distorted point clouds in total. The WPC database contains 20 reference point clouds and augmented each point cloud with four types of distortions (down-sampling, Gaussian white noise, Geometry-based Point Cloud Compression (G-PCC), and Video-based Point Cloud Compression (V-PCC)), which generates 740 = 20$\times$37 distorted point clouds.

The Adam optimizer [18] is utilized with weight decay 1e-4 and the initial learning rate is set as 5e-5. The model is trained for 50 epochs by default. We set the point cloud sub-model size $N_s$ as 2048 and set $N_{\delta }=N/N_s+1$. The projected images with the resolution of 1920$\times$1080$\times$3 are randomly cropped into image patches at the resolution of 224$\times$224$\times$3 as the inputs. During the experiment, 6 sub-models and 4 projected images are selected at random for each point cloud. The PointNet++ [30] is utilized as the point cloud encoder and the hierarchical ResNet50 [17] is used as the image encoder, where the ResNet50 is initialized with the pre-trained model on the ImageNet database [11]. The multi-head attention module employs 8 heads and the feed-forward dimension is set as 2048. The weights ${\lambda }_1$ and ${\lambda }_2$ for $L_{MSE}$ and $L_{rank}$ are both set as 1.

Following the practices in [9, 13], the k-fold cross validation strategy is employed for the experiment to accurately estimate the performance of the proposed method. Specifically, for the SJTU-PCQA database, 1 group of point cloud is chosen as the testing set and the remaining 8 groups of point cloud are taken as the training set. Such procedure is repeated 9 times with each of the point cloud groups used exactly once as the testing set. Similarly, we randomly split the WPC database into 5 folds and each fold consists of 4 groups of point clouds. Then a similar experiment procedure as discussed above is applied. In all, 9-fold cross validation is conducted on the SJTU-PCQA database and 5-fold cross validation is conducted on the WPC database. The average performance is recorded as the final results. It’s worth noting that there is no content overlap between the training and testing sets. We strictly retrain the available baselines with the same database split set up to keep the comparison fair. What’s more, for the FR-PCQA and RR-PCQA methods that require no training, we simply validate them on the same testing sets and record the average performance.

15 state-of-the-art quality assessment methods are selected for comparison, which consist of 8 FR-PCQA methods, 1 RR-PCQA method, and 6 NR-PCQA methods. The FR-PCQA methods include MSE-p2point (MSE-p2po) [23], Hausdorff-p2point (HD-p2po) [23], MSE-p2plane (MSE-p2pl) [34], Hausdorff-p2plane (HD-p2pl) [34], PSNR-yuv [35], PCQM [25], GraphSIM [46], and PointSSIM [2]. The RR-PCQA method includes PCMRR [36]. The NR-PCQA methods include BRISQUE [26], NIQE [27], IL-NIQE [49], ResSCNN [22], PQA-net [21], and 3D-NSS [51]. Note that BRISQUE, NIQE, IL-NIQE are image-based quality assessment metrics and are validated on the same projected images. Furthermore, to deal with the scale differences between the predicted quality scores and the quality labels, a five-parameter logistic function is applied to map the predicted scores to subjective ratings, as suggested by [14, 3].

Four mainstream evaluation criteria in the quality assessment field are utilized to compare the correlation between the predicted scores and MOSs, which include Spearman Rank Correlation Coefficient (SRCC), Kendall’s Rank Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), Root Mean Squared Error (RMSE). An excellent quality assessment model should obtain values of SRCC, KRCC, PLCC close to 1 and RMSE to 0.

As described above, MM-PCQA jointly employs the features from the point cloud and image modalities and we hold the hypothesis that multi-modal learning helps the model to gain better quality representation than single-modality. Therefore, in this section, we conduct ablation studies to validate our hypothesis. The performance results are presented in Table II. The performance of both single-modal-based model is inferior to the multi-modal-based model, which suggests that both point cloud and image features make contributions to the final results. After using SCMA module, the performance is further boosted, which validates the effectiveness of cross-modality attention.

To prove that the patch-up strategy is more suitable for PCQA, we present the performance of using farthest point sampling (FPS) strategy as well. To make the comparison even, 12,288 = 6$\times$2048 points (the same number of points as contained in the default 6 sub-models) are sampled for each point cloud and the results are shown in Table III. It can be seen that the patch-up strategy greatly improves the performance when only point cloud features are used. The reason is that the sampled points are not able to preserve the local patterns that are vital for quality assessment. Moreover, when the point cloud contains more points (even up to millions), the sampled points may not even maintain the main shape of the object unless greatly increasing the number of sampling points.

The cross-database evaluation is further conducted to test the generalization ability of the proposed MM-PCQA and the experimental results are exhibited in Table IV. From the table, we can find that the proposed MM-PCQA better generalizes learned feature representation from the WPC database and achieves much better performance than the most competitive NR-PCQA methods when validating on the SJTU-PCQA database. Although all the NR-PCQA models trained on the SJTU-PCQA database obtain poor performance, MM-PCQA still yields the best performance.

We try to investigate the contributions of the point cloud and image branch by varying the number of the 3D sub-models and input 2D projections. The performance results are exhibited in Table V and Table VI . We can see that employing 6 sub-models and 4 projections yields the best performance for the point cloud and image branch respectively. With the number increasing, the features extracted from sub-models and projections may get redundant, thus resulting in the performance drop.

From the above analysis, the major limitation of the proposed model lies in the cross-database (SJTU$\to$WPC) evaluation, mainly due to the following reasons: a) The number of training samples for the SJTU-PCQA database is half of the WPC database (378$\to$740), which means the SJTU-PCQA database contains less diverse content and distortion types. b) The distortion levels of the SJTU-PCQA database are relatively coarse-grained. Thus the quality representation learned from the SJTU-PCQA database is not effective on the WPC database which contains stimuli with fine-grained distortion levels. In all, due to the limited scale of current PCQA databases, the generalization performance of MM-PCQA is still unsatisfactory.