Blind Quality Assessment of 3D Dense Point Clouds with Structure Guided Resampling

Keypoint resampling. Suppose that a distorted 3D point cloud P contains $M$ points and each point has three geometry coordinates (e.g., $x,y,z$) as well as three color attributes (e.g., $r,g,b$), we can denote the 3D point cloud data as follows:

where ${\text{p}}_m=(x_m,y_m,z_m,r_m,g_m,b_m)$ is a point in the point cloud. In this representation, the geometry coordinates and color attributes can be separated as ${\text{g}}_m=(x_m,y_m,z_m)$ and ${\text{c}}_m=(r_m,g_m,b_m)$, respectively. In addition, another attribute called normal vectors can be estimated from the point cloud.

Generally, each 3D point cloud data has numerous points. For example, the 3D point cloud in Fig. 1 (b) has $M=2,486,566$ points. Moreover, these 3D points are irregular and unstructured. Nevertheless, the HVS inclines to perceive the structures of objects or environments. For traditional images and videos, such structure information has been widely used to design many objective quality measures [41, 43, 42]. Inspired by GraphSIM [46], we opt to extract the keypoints by graph-based resampling to realize the structure extraction of 3D point clouds. Specifically, considering that the resampled points should be high-frequency parts such as edges, contours, etc, GraphSIM uses a Haar-like high pass filter to extract point cloud skeleton based on original geometry coordinates. However, this method is too sensitive to the outlier, which may lead to unstable results. Therefore, we first compute the normal vectors as and treat them as the graph signal to guide keypoint resampling.

As defined, a graph filter represents a signal system that inputs a graph signal and outputs a tuned graph signal [8]. Mathematically, a linear and shift-invariant graph filter is a polynomial conversion of the graph shift operator $A\in R^{M\times M}$ computed by:

where $h_k$ denotes the $k-th$ filter coefficients and $K$ is the length of the graph filter. I represents the identity matrix. With the predefined graph filter, for an input graph signal $\Phi \in R^{M\times 1}$, the filtered output graph signal is a matrix-vector product which can be converted by eigendecomposition as:

where the eigenvectors of $A$ constitute the columns of matrix V. The eigenvalue matrix $\Lambda \in R^{M\times M}$ is the diagonal matrix containing the corresponding eigenvalues of $A$. Here, the eigenvalues are frequencies on the formed graph, which can be used to sort the 3D points.

Given a distorted 3D point cloud P, from the above analysis, we resample it to extract graph keypoints by:

where $\hat{\text{P}}\in R^{\Theta \times 6}$, and $h_H(\cdot )$ is a high-pass graph filter with length equaling to $K$, and the graph signal $\Phi$ is normal vector. Moreover, $\Theta$ represents the number of keypoints after the resampling operation. Similar to [49], we set $K$ and $\Theta$ as $4$ and $M/10,000$, respectively.

As shown in Fig. 4, we give examples of the keypoint resampling results for 3D point clouds which are distorted by various Octree-based compression levels. From (a), (c) to (e), the distortion levels increase, which can be easily observed by the appearances of distorted point clouds. We can see that the keypoints generally represent the structures of point clouds. In addition, with the increase of distortion levels, the number of resampled points decreases. Therefore, the keypoint resampling can be a promising quality indicator for subsequent local region construction and quality-aware feature extraction. This is mainly benefit from the efficient structure information of normal attribute.

Local region construction. After the process of keypoint resampling, we exploit the generated keypoints to construct local regions. For each keypoint $\widehat{{\text{p}}_{\theta }}$ in $\hat{\text{P}}$, we cluster its neighbors based on the Euclidean distance of the corresponding geometry coordinates as follows:

where $\tilde{\text{P}}$ denotes the groups of clustered neighbors of the keypoint $\widehat{{\text{p}}_{\theta }}$. $\tilde{g}$ and $\widehat{g_{\theta }}$ are the geometry components of $\tilde{\text{P}}$ and $\widehat{{\text{p}}_{\theta }}$, respectively. In addition, $\alpha$ is used to cluster neighbors, which is $1/20$ of minimum range among $x$, $y$, and $z$ coordinates.

In Fig. 5, we show the structure guided local region construction of distorted 3D point cloud after keypoint resampling. Here, we take the figure (a) in Fig. 4 as an example, where three constructed regions are given. It can be found that these local regions have different shapes, which are used for the quality-aware feature extraction.

Based on the constructed local region, we exploit the geometry components to derivate geometry density features. To be specific, since the geometry information can be directly obtained from the local region, we separately compute the mean and standard deviation values of all three coordinates to serve as the geometry density features. For example, the mean and standard deviation of geometry information along $x$-coordinate are calculated by:

where ${\tilde{g}}_i^x$ and $I$ represent the $i-th$ geometry information and total number of graph points in the local region along $x$-coordinate, respectively. Similar to $x$-coordinate, we can also compute the mean and standard deviation of geometry information along $y$-coordinate and $z$-coordinate. With all three coordinates, we can obtain 6 dimensional geometry density features from the average operation of extracted features across various local regions.

Since the standard deviation is induced from mean value, we provide the mean value changes for two 3D point clouds at different distortion degrees, as shown in Fig. 6. Here, the point clouds are distorted with light and heavy V-PCC compression artifacts. As can be seen from this figure, the histogram comparisons of the distorted 3D point clouds demonstrate that the mean value from all three coordinates can effectively reveal the quality variation. In addition, the effectiveness of these features is mainly because they can reflect the density of 3D point clouds.

As for 3D point clouds, color information is a significant attribute. Thus, we further extract color naturalness features from the constructed local region. By taking the human perception into account, we first convert the RGB components to YUV space. Then, to normalize the distributions, we apply the zero-phase component analysis followed by local mean subtraction and divisive normalization [28] to the separate YUV channels of distorted local regions as:

where ${\tilde{\rho }}_z(y,u,v)$ is the YUV information of local regions after the whitening filter of zero-phase component analysis [9, 52], and ${\overline{\rho }}_z(y,u,v)$ represents the final normalized YUV channels. $C=1$ is a small constant that avoids instabilities when the denominator tends to $0$. The local mean and standard deviation of YUV channels can be computed by:

where $\omega =\{{\omega }_{l,t}\mid l=-L,\dots ,L,t=-T,\dots ,T\}$ indicates a circularly symmetric Gaussian weighting function. Motivated by [24], we set $L=T=3$.

In Fig. 7, we show an example of statistical probability distribution for normalized coefficients, where the 3D point cloud is distorted by downscaling artifacts. On the basis of its Gaussian-like appearance, we use the generalized Gaussian distribution (GGD) [31] to capture the regularity of NSS, which is given by:

where

and $\Gamma (\cdot )$ is the gamma function as follows:

In the GGD, $\lambda$ controls the shape of statistical probability distribution and ${\epsilon }^2$ reflects the variance. For each color channel of the input distorted point cloud, we estimate 2 parameters $(\lambda ,{\epsilon }^2)$ from the GGD fit by the moment matching-based approach [31]. Additionally, we use four scales containing the original scale as well as the reduction by the factors of 2, 4, and 8. Totally, with YUV channels, we have 24 dimensional color naturalness features which are calculated by the average operation among all the local regions.

Another important attribute information is normal that indicates the orientation of points. To fully use the unique normal vectors, apart from regional pre-processing, they have also been verified to be correlated to visual quality [37, 1, 39]. Therefore, we propose angular consistency features based on the normal information. Specifically, for each 3D point in the distorted point clouds, we first estimate the normal vector ${\overrightarrow{\chi }}_i$ of the point by using multiple neighboring points to fit a local plane for determining the normal vector. It should be noted that the number of neighboring points for normal vector estimation is validated in the experiments. Then, we adopt the $k$-nearest algorithm to choose the set of neighbors for each normal vector. In such case, assume that the normal vector of each neighbor point is denoted by ${\overrightarrow{\chi }}_j$, we compute the cosine similarity between ${\overrightarrow{\chi }}_i$ and ${\overrightarrow{\chi }}_j$ by:

where $\Omega \in [-1,1]$ and $\zeta$ is the angular between the two normal vectors. Based on this, the inverse cosine of the obtained $\Omega$ is calculated as:

where ${\zeta }^{\prime }\in [0,\pi /2]$ and thus the ultimate angular similarity can be computed in the range of $[0,1]$ as follows:

From the above-mentioned computation, we obtain the angular similarity matrix for every distorted point cloud. On one hand, we conduct an average operation for the point number dimension. On the other hand, 5 kinds of statistics are calculated for the $k$-nearest neighbor dimension, where the first two statistical features are the mean $\mu$ and standard deviation $\delta$. Furthermore, the skewness and kurtosis are computed as:

where $E[\cdot ]$ denotes the expectation operator, and ${\psi }_k$ is the angular similarity value. Except for these moments, as suggested in [12], the entropy is also obtained by:

where $p\left({\psi }_k\right)$ is the probability of ${\psi }_k$. These statistics of angular similarity matrix can be regarded as the angular consistency of normal vectors.

To extract the angular consistency features, since the HVS perception is hierarchical, here we employ more scales for better feature representations. As illustrated in Fig. 8, two scales of PCL compressed point clouds are shown, where figures (a) and (b) have $132,518$ points and $66,259$ points, respectively. From the two scales, we have 10 dimensional angular consistency features, which are used for quantifying visual distortions of 3D point clouds.

After the regional pre-processing and quality-aware feature extraction, a quality regression module is applied to map the relevant features onto final quality scores. Many regressors can be used for this purpose, among whom we select the well-known support vector regression (SVR) and random forest regression (RFR) due to their capacity and popularity in quality assessment problems [14, 33, 11, 51]. Especially, they have also been widely used for the spatial-domain and transform-domain NSS in the 2D image quality evaluation task [25, 29, 26, 13].

In our framework, given a test 3D point cloud, all features from geometry density, color naturalness and angular consistency aspects are extracted and concatenated to a feature vector. Before predicting the quality result of the test 3D point cloud, we use the labeled point clouds to train the regressor. With the trained regressor, we can predict the perceptual quality of any input test 3D point cloud. The LIBSVM [6] is utilized to implement SVR with a radial basis function kernel, together with the RFR proposed in [5, 10] are used for our experiments. Additionally, we follow the common regressor parameter settings that have been used in the training of mainstream quality assessment models.

We conduct experiments to validate our proposed SGR on four publicly available subjective point cloud quality databases, including Waterloo [35, 17], M-PCCD [4], SJTU [44] and IRPC [15] databases. The detailed information of these databases is shown in TABLE I. Specifically, followed by the recent NR quality assessment work [18], we first use the selected Waterloo database to compare the proposed SGR with a variety of state-of-the-art FR, RR and NR quality assessment methods for 3D point clouds. This database contains 20 original reference point cloud contents with different geometry and texture complexities. Four distortion types are considered to generate the corresponding quality-degraded point clouds, which include 60 downsampling distorted point clouds, 180 distorted point clouds with Gaussian noise, 320 G-PCC (T) compressed point clouds, and 180 V-PCC compressed point clouds. Moreover, the adopted SJTU database consists of 6 pristine point cloud contents, i.e. longdress, loot, redandblack, shiva, soldier, and statue. To produce distorted point clouds, the Octree-based compression, color noise, downsampling, and geometry Gaussian noise are introduced to the original point clouds. As for M-PCCD and IRPC databases, the numbers of original point cloud contents are 8 and 6, respectively. By involving several distortion types, 232 and 54 distorted 3D point clouds are generated in the two databases.

Apart from the reference and distorted point clouds, each database also provides subjective quality labels in the form of mean opinion score (MOS), which are rated by viewers with several display modes, such as direct point cloud format and converted video format, etc. Details can be referred to the literature [35, 17, 4, 44, 15]. Note that only the 3D point clouds in the IRPC database have normal vectors.

To validate the proposed SGR and compare it with other state-of-the-arts, we adopt three commonly-used evaluation criteria in IQA field, including Spearman Rank-Order Correlation Coefficient (SROCC), Kendall Rank-Order Correlation Coefficient (KROCC), and Pearson Linear Correlation Coefficient (PLCC). Among these evaluation criteria, the SROCC is usually used to measure prediction monotonicity, while KROCC can be applied to evaluate the ordinal association between two measured quantities. Moreover, the PLCC can be used to evaluate prediction accuracy. It should be noted that higher correlation coefficients indicate better performance for objective quality models.

Besides, before computing the PLCC for different objective quality assessment approaches, a five-parameter logistic nonlinear fitting function [27] is adopted to map the predicted quality scores into a common scale as:

where $({\beta }_1...{\beta }_5)$ are five parameters to be fitted. $\gamma$ represents the raw objective score produced by objective quality models and $g\left(\gamma \right)$ denotes the regressed score after the nonlinear mapping.

In order to validate the performance of our proposed SGR, we compare the proposed SGR with state-of-the-art FR, RR and NR models on the largest subjective point cloud quality database, i.e. the Waterloo database [35, 17]. These include 14 FR methods, where both point-based and projection-based approaches are compared. Among them, classical point-based metrics contain $PSN{R}_{MSE,p2po}$, $PSN{R}_{HF,p2po}$, $PSN{R}_{MSE,p2pl}$, $PSN{R}_{HF,p2pl}$, $PSN{R}_Y$, $A{S}_{Mean}$, $A{S}_{RMS}$, $A{S}_{MSE}$, $GraphSIM$, $PCQM$, and $PointSSIM$. The projection-based FR metrics involve $SSI{M}_{projected}$, $MS$-$SSI{M}_{projected}$, and $VIF{P}_{projected}$. Representative RR and NR methods are also taken into consideration, such as $PCMRR$ and $PQA$-$Net$.

By following [18], we conduct the comparison for various visual contents and distortion types. The evaluation results of different point cloud contents are reported in TABLE II. For space convenience, we show several contents and the overall performance. We can observe that our proposed SGR achieves competitive results for all the test visual contents. Besides, by using both the SVR and RFR regressors, our framework delivers promising mapping results, demonstrating that the proposed SGR does not rely on specific regressor. More importantly, our proposed SGR is superior to the $PQA$-$Net$ regarding the overall performance. It should be noted that the $PQA$-$Net$ is a deep learning-based NR quality assessment model for 3D point clouds. In addition, beyond the performance results for individual visual contents, we show the performance comparisons of various distortion types in TABLE III. Again, our proposed SGR outperforms the $PQA$-$Net$ for all distortion types, especially for Gaussian noise and V-PCC compression. Therefore, both the two tables demonstrate the superiority of our SGR algorithm.

In addition to quantitative performance results, we compare the visualized 3D point clouds with the predicted scores by our proposed SGR. Both SVR and RFR predicted quality results are computed.

We show some examples of perceptual quality prediction in Fig. 9. Each row represents one visual content. That is, the original reference contents are banana and mushroom, which contain 807,184 and 1144,603 points, respectively. By introducing the Gaussian noise, the reference point cloud would be distorted with various degrees. From the trends of SGR (SVR) and SGR (RFR) values, we can see that the proposed SGR method successfully distinguishes distorted point cloud data of different qualities. Therefore, the proposed model can effectively evaluate the perceptual quality of distorted 3D point clouds. In addition, for point clouds with relatively lower quality, i.e. figures (e-f) and (g-h), the prediction results from two models are more consistent. This may be because compared to low-quality point clouds, it is harder to predict the perceptual quality of high-quality point cloud data.

Apart from the largest Waterloo database, we further test the performance of the proposed SGR on other subjective databases, including M-PCCD [4], SJTU [44], and IRPC [15]. Note that only the distorted 3D point clouds with individual/single distortion types are tested.

In the experiments, each database is randomly divided into training and testing sets with $80\%$-$20\%$ splitting. TABLE IV provides the comparison results, where the state-of-the-art NR model $PQA$-$Net$ is compared with our proposed SGR. From this table, we can find that the proposed SGR performs better than the $PQA$-$Net$, which demonstrates the effectiveness of our SGR in general.

Since the proposed SGR is composed of three groups of quality-aware features, including geometry density, color naturalness, and angular consistency. It is interesting to validate the performance of each individual feature component. In TABLE V, we conduct the ablation study and show the results. It can be seen that the two components from regional pre-processing are combined to boost the performance. Moreover, further integrating angular consistency leads to the best results of the proposed SGR framework.

Additionally, the keypoint resampling is based on the unique normal vectors, we test the cases by using other information, such as the color attributes and geometry information. The performance results are listed in TABLE VI, where the proposed method outperforms the others. One possible explanation may be that the normal vectors can represent the structures of point clouds. Here, we also give the comparisons of keypoint resampling in Fig. 10. As can be seen in this figure, we take a point cloud distorted by severe Gaussian noise as an example. Compared with the keypoints resampled from geometry or color signals, by using the unique normal vectors, the keypoints are sampled more uniformly and cover a larger range of visual content. Besides, since the extracted quality-aware features after regional pre-processing are based on geometry and color aspects, our method could achieve the disentanglement of the known information to some extent. All these validate the superiority of our proposed structure guided resampling. Furthermore, the used normal number is explored in our experiment. TABLE VII shows that 12 normal numbers can deliver the best performance. With more and less normal numbers, the results would decrease. Therefore, we choose this parameter equaling to 12 in the proposed method.