很难精确地注释对象实例及其在3D空间中的语义，因此，合成数据被广泛用于这些任务，例如类别级别6D对象姿势和大小估计。然而，合成域中的简易注释带来了合成到真实（SIM2REAL）域间隙的下行效应。在这项工作中，我们的目标是在SIM2REAL，无监督的域适应的任务设置中解决此问题，以适应类别级别6D对象姿势和尺寸估计。我们提出了一种基于新型的深层变形网络构建的方法，该网络缩短为DPDN。 DPDN学会了将分类形状先验的变形特征与对象观察的特征相匹配，因此能够在特征空间中建立深层对应，以直接回归对象姿势和尺寸。为了减少SIM2REAL域间隙，我们通过一致性学习在DPDN上制定了一个新颖的自我监督目标。更具体地说，我们对每个对象观察进行了两个刚性转换，并分别将它们送入DPDN以产生双重预测集。除了平行学习之外，还采用了一个矛盾术语来保持双重预测之间的交叉一致性，以提高DPDN对姿势变化的敏感性，而单个的内部矛盾范围则用于在每个学习本身内实施自我适应。我们在合成摄像头25和现实世界Real275数据集的两个训练集上训练DPDN；我们的结果优于无监督和监督设置下的Real275测试集中的现有方法。消融研究还验证了我们设计的功效。我们的代码将在https://github.com/jiehonglin/self-dpdn公开发布。

蒙面自动编码是一种流行而有效的自我监督学习方法，可以指向云学习。但是，大多数现有方法仅重建掩盖点并忽略本地几何信息，这对于了解点云数据也很重要。在这项工作中，据我们所知，我们首次尝试将局部几何信息明确考虑到掩盖的自动编码中，并提出一种新颖的蒙版表面预测（Masksurf）方法。具体而言，考虑到以高比例掩盖的输入点云，我们学习一个基于变压器的编码器码头网络，通过同时预测表面位置（即点）和每条效率方向（即，正常），以估算基础掩盖的表面。 。点和正态的预测由倒角距离和新引入的位置指标的正常距离以设定的方式进行监督。在三种微调策略下，我们的Masksurf在六个下游任务上得到了验证。特别是，MaskSurf在OBJ-BG设置下的ScanoBjectNN的现实世界数据集上胜过其最接近的竞争对手Point-Mae，证明了掩盖的表面预测的优势比蒙版的预测优势比蒙版的预测。代码将在https://github.com/ybzh/masksurf上找到。

作为SE（3）的基本组成部分 - Quivariant的深度特色学习，可转向卷积最近展示了其3D语义分析的优势。然而，优点由昂贵的体积数据上的昂贵计算带来，这可以防止其实际用途，以便有效地处理固有的稀疏的3D数据。在本文中，我们提出了一种新颖的稀疏转向卷积（SS-Char）设计，以解决缺点; SS-DIM大大加快了稀疏张量的可操纵卷积，同时严格保留了SE（3）的性质。基于SS-CONV，我们提出了一种用于精确估计对象姿势的一般管道，其中一个关键设计是一种特征转向模块，其具有SE（3）的完全优势，并且能够进行高效的姿势改进。为了验证我们的设计，我们对三个对象语义分析的三个任务进行了彻底的实验，包括实例级别6D姿势估计，类别级别6D姿势和大小估计，以及类别级6D姿态跟踪。我们基于SS-CONV的提议管道优于三个任务评估的几乎所有指标上的现有方法。消融研究还在准确性和效率方面展示了我们的SS-CONVES对替代卷积的优越性。我们的代码在https://github.com/gorilla-lab-scut/ss-conv公开发布。

Unsupervised domain adaptation (UDA) for semantic segmentation is a promising task freeing people from heavy annotation work. However, domain discrepancies in low-level image statistics and high-level contexts compromise the segmentation performance over the target domain. A key idea to tackle this problem is to perform both image-level and feature-level adaptation jointly. Unfortunately, there is a lack of such unified approaches for UDA tasks in the existing literature. This paper proposes a novel UDA pipeline for semantic segmentation that unifies image-level and feature-level adaptation. Concretely, for image-level domain shifts, we propose a global photometric alignment module and a global texture alignment module that align images in the source and target domains in terms of image-level properties. For feature-level domain shifts, we perform global manifold alignment by projecting pixel features from both domains onto the feature manifold of the source domain; and we further regularize category centers in the source domain through a category-oriented triplet loss and perform target domain consistency regularization over augmented target domain images. Experimental results demonstrate that our pipeline significantly outperforms previous methods. In the commonly tested GTA5$\rightarrow$Cityscapes task, our proposed method using Deeplab V3+ as the backbone surpasses previous SOTA by 8%, achieving 58.2% in mIoU.

Compressed videos often exhibit visually annoying artifacts, known as Perceivable Encoding Artifacts (PEAs), which dramatically degrade video visual quality. Subjective and objective measures capable of identifying and quantifying various types of PEAs are critical in improving visual quality. In this paper, we investigate the influence of four spatial PEAs (i.e. blurring, blocking, bleeding, and ringing) and two temporal PEAs (i.e. flickering and floating) on video quality. For spatial artifacts, we propose a visual saliency model with a low computational cost and higher consistency with human visual perception. In terms of temporal artifacts, self-attention based TimeSFormer is improved to detect temporal artifacts. Based on the six types of PEAs, a quality metric called Saliency-Aware Spatio-Temporal Artifacts Measurement (SSTAM) is proposed. Experimental results demonstrate that the proposed method outperforms state-of-the-art metrics. We believe that SSTAM will be beneficial for optimizing video coding techniques.

Image Virtual try-on aims at replacing the cloth on a personal image with a garment image (in-shop clothes), which has attracted increasing attention from the multimedia and computer vision communities. Prior methods successfully preserve the character of clothing images, however, occlusion remains a pernicious effect for realistic virtual try-on. In this work, we first present a comprehensive analysis of the occlusions and categorize them into two aspects: i) Inherent-Occlusion: the ghost of the former cloth still exists in the try-on image; ii) Acquired-Occlusion: the target cloth warps to the unreasonable body part. Based on the in-depth analysis, we find that the occlusions can be simulated by a novel semantically-guided mixup module, which can generate semantic-specific occluded images that work together with the try-on images to facilitate training a de-occlusion try-on (DOC-VTON) framework. Specifically, DOC-VTON first conducts a sharpened semantic parsing on the try-on person. Aided by semantics guidance and pose prior, various complexities of texture are selectively blending with human parts in a copy-and-paste manner. Then, the Generative Module (GM) is utilized to take charge of synthesizing the final try-on image and learning to de-occlusion jointly. In comparison to the state-of-the-art methods, DOC-VTON achieves better perceptual quality by reducing occlusion effects.

Panoptic Part Segmentation (PPS) unifies panoptic segmentation and part segmentation into one task. Previous works utilize separated approaches to handle thing, stuff, and part predictions without shared computation and task association. We aim to unify these tasks at the architectural level, designing the first end-to-end unified framework named Panoptic-PartFormer. Moreover, we find the previous metric PartPQ biases to PQ. To handle both issues, we make the following contributions: Firstly, we design a meta-architecture that decouples part feature and things/stuff feature, respectively. We model things, stuff, and parts as object queries and directly learn to optimize all three forms of prediction as a unified mask prediction and classification problem. We term our model as Panoptic-PartFormer. Secondly, we propose a new metric Part-Whole Quality (PWQ) to better measure such task from both pixel-region and part-whole perspectives. It can also decouple the error for part segmentation and panoptic segmentation. Thirdly, inspired by Mask2Former, based on our meta-architecture, we propose Panoptic-PartFormer++ and design a new part-whole cross attention scheme to further boost part segmentation qualities. We design a new part-whole interaction method using masked cross attention. Finally, the extensive ablation studies and analysis demonstrate the effectiveness of both Panoptic-PartFormer and Panoptic-PartFormer++. Compared with previous Panoptic-PartFormer, our Panoptic-PartFormer++ achieves 2% PartPQ and 3% PWQ improvements on the Cityscapes PPS dataset and 5% PartPQ on the Pascal Context PPS dataset. On both datasets, Panoptic-PartFormer++ achieves new state-of-the-art results with a significant cost drop of 70% on GFlops and 50% on parameters. Our models can serve as a strong baseline and aid future research in PPS. Code will be available.

In recent years, the Transformer architecture has shown its superiority in the video-based person re-identification task. Inspired by video representation learning, these methods mainly focus on designing modules to extract informative spatial and temporal features. However, they are still limited in extracting local attributes and global identity information, which are critical for the person re-identification task. In this paper, we propose a novel Multi-Stage Spatial-Temporal Aggregation Transformer (MSTAT) with two novel designed proxy embedding modules to address the above issue. Specifically, MSTAT consists of three stages to encode the attribute-associated, the identity-associated, and the attribute-identity-associated information from the video clips, respectively, achieving the holistic perception of the input person. We combine the outputs of all the stages for the final identification. In practice, to save the computational cost, the Spatial-Temporal Aggregation (STA) modules are first adopted in each stage to conduct the self-attention operations along the spatial and temporal dimensions separately. We further introduce the Attribute-Aware and Identity-Aware Proxy embedding modules (AAP and IAP) to extract the informative and discriminative feature representations at different stages. All of them are realized by employing newly designed self-attention operations with specific meanings. Moreover, temporal patch shuffling is also introduced to further improve the robustness of the model. Extensive experimental results demonstrate the effectiveness of the proposed modules in extracting the informative and discriminative information from the videos, and illustrate the MSTAT can achieve state-of-the-art accuracies on various standard benchmarks.

This paper illustrates the technologies of user next intent prediction with a concept knowledge graph. The system has been deployed on the Web at Alipay, serving more than 100 million daily active users. Specifically, we propose AlipayKG to explicitly characterize user intent, which is an offline concept knowledge graph in the Life-Service domain modeling the historical behaviors of users, the rich content interacted by users and the relations between them. We further introduce a Transformer-based model which integrates expert rules from the knowledge graph to infer the online user's next intent. Experimental results demonstrate that the proposed system can effectively enhance the performance of the downstream tasks while retaining explainability.

As natural language processing (NLP) for gender bias becomes a significant interdisciplinary topic, the prevalent data-driven techniques such as large-scale language models suffer from data inadequacy and biased corpus, especially for languages with insufficient resources such as Chinese. To this end, we propose a Chinese cOrpus foR Gender bIas Probing and Mitigation CORGI-PM, which contains 32.9k sentences with high-quality labels derived by following an annotation scheme specifically developed for gender bias in the Chinese context. Moreover, we address three challenges for automatic textual gender bias mitigation, which requires the models to detect, classify, and mitigate textual gender bias. We also conduct experiments with state-of-the-art language models to provide baselines. To our best knowledge, CORGI-PM is the first sentence-level Chinese corpus for gender bias probing and mitigation.