Structure-guided image completion aims to inpaint a local region of an image according to an input guidance map from users. While such a task enables many practical applications for interactive editing, existing methods often struggle to hallucinate realistic object instances in complex natural scenes. Such a limitation is partially due to the lack of semantic-level constraints inside the hole region as well as the lack of a mechanism to enforce realistic object generation. In this work, we propose a learning paradigm that consists of semantic discriminators and object-level discriminators for improving the generation of complex semantics and objects. Specifically, the semantic discriminators leverage pretrained visual features to improve the realism of the generated visual concepts. Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects. Our proposed scheme significantly improves the generation quality and achieves state-of-the-art results on various tasks, including segmentation-guided completion, edge-guided manipulation and panoptically-guided manipulation on Places2 datasets. Furthermore, our trained model is flexible and can support multiple editing use cases, such as object insertion, replacement, removal and standard inpainting. In particular, our trained model combined with a novel automatic image completion pipeline achieves state-of-the-art results on the standard inpainting task.

在本文中，我们考虑了颜色加式双相机系统，并提出了一个端到端的卷积神经网络，以有效且具有成本效益的方式使图像对齐和融合图像。我们的方法将跨域和跨尺度图像作为输入，因此综合了HR着色结果，以促进单相机成像系统中时空分辨率和色彩深度之间的权衡。与以前的着色方法相反，我们的功能可以适应具有独特时空分辨率的色彩和单色相机，从而使实际应用中的灵活性和鲁棒性。我们方法的关键要素是一个跨相机比对模块，该模块生成跨域图像对齐的多尺度对应关系。通过在各种数据集和多个设置上进行广泛的实验，我们验证了方法的灵活性和有效性。值得注意的是，我们的方法始终取得了实质性改进，即在最新方法上，大约10dB PSNR增益。代码为：https：//github.com/indigopurple/ccdc

最近的图像入介方法取得了长足的进步，但在处理复杂图像中的大孔时，通常很难产生合理的图像结构。这部分是由于缺乏有效的网络结构可以捕获图像的远程依赖性和高级语义。我们提出了级联调制GAN（CM-GAN），这是一种新的网络设计，由编码器组成，该设计由带有傅立叶卷积块的编码器组成，该块从带有孔的输入图像中提取多尺度特征表示，并带有带有新型级联全球空间调制的双流式解码器在每个比例尺上块。在每个解码器块中，首先应用全局调制以执行粗糙和语义感知的结构合成，然后进行空间调制以进一步以空间自适应的方式调整特征图。此外，我们设计了一种对象感知的培训方案，以防止网络在孔内部幻觉，从而满足实际情况下对象删除任务的需求。进行了广泛的实验，以表明我们的方法在定量和定性评估中都显着优于现有方法。请参阅项目页面：\ url {https://github.com/htzheng/cm-gan-inpainting}。

在视频deNoising中，相邻的框架通常提供非常有用的信息，但是需要准确的对齐方式，然后才能刺激此类信息。在这项工作中，我们提出了一个多对准网络，该网络生成多个流动建议，然后是基于注意的平均。它用于模仿非本地机制，通过平均多个观测来抑制噪声。我们的方法可以应用于基于流量估计的各种最新模型。大规模视频数据集上的实验表明，我们的方法通过0.2DB提高了Denoisis基线模型，并通过模型蒸馏进一步将参数降低了47％。代码可在https://github.com/indigopurple/manet上找到。

最近，大型预磨损模型（例如，BERT，STYLEGAN，CLIP）在其域内的各种下游任务中表现出很好的知识转移和泛化能力。在这些努力的启发中，在本文中，我们提出了一个统一模型，用于开放域图像编辑，重点是开放式域图像的颜色和音调调整，同时保持原始内容和结构。我们的模型了解许多现有照片编辑软件中使用的操作空间（例如，对比度，亮度，颜色曲线）更具语义，直观，易于操作的统一编辑空间。我们的模型属于图像到图像转换框架，由图像编码器和解码器组成，并且在图像之前和图像的成对上培训以产生多模式输出。我们认为，通过将图像对反馈到学习编辑空间的潜在代码中，我们的模型可以利用各种下游编辑任务，例如语言引导图像编辑，个性化编辑，编辑式聚类，检索等。我们广泛地研究实验中编辑空间的独特属性，并在上述任务上展示了卓越的性能。

Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than CLS token- and feature-based distillation; 2) An intermediate layer of the teacher network as target perform better than that using the last layer when the depth of the student mismatches that of the teacher; 3) Weak regularization is preferred; etc. With these findings, we achieve significant fine-tuning accuracy improvements over the scratch MIM pre-training on ImageNet-1K classification, using all the ViT-Tiny, ViT-Small, and ViT-base models, with +4.2%/+2.4%/+1.4% gains, respectively. Our TinyMIM model of base size achieves 52.2 mIoU in AE20K semantic segmentation, which is +4.1 higher than the MAE baseline. Our TinyMIM model of tiny size achieves 79.6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget. This strong performance suggests an alternative way for developing small vision Transformer models, that is, by exploring better training methods rather than introducing inductive biases into architectures as in most previous works. Code is available at https://github.com/OliverRensu/TinyMIM.

Dataset distillation has emerged as a prominent technique to improve data efficiency when training machine learning models. It encapsulates the knowledge from a large dataset into a smaller synthetic dataset. A model trained on this smaller distilled dataset can attain comparable performance to a model trained on the original training dataset. However, the existing dataset distillation techniques mainly aim at achieving the best trade-off between resource usage efficiency and model utility. The security risks stemming from them have not been explored. This study performs the first backdoor attack against the models trained on the data distilled by dataset distillation models in the image domain. Concretely, we inject triggers into the synthetic data during the distillation procedure rather than during the model training stage, where all previous attacks are performed. We propose two types of backdoor attacks, namely NAIVEATTACK and DOORPING. NAIVEATTACK simply adds triggers to the raw data at the initial distillation phase, while DOORPING iteratively updates the triggers during the entire distillation procedure. We conduct extensive evaluations on multiple datasets, architectures, and dataset distillation techniques. Empirical evaluation shows that NAIVEATTACK achieves decent attack success rate (ASR) scores in some cases, while DOORPING reaches higher ASR scores (close to 1.0) in all cases. Furthermore, we conduct a comprehensive ablation study to analyze the factors that may affect the attack performance. Finally, we evaluate multiple defense mechanisms against our backdoor attacks and show that our attacks can practically circumvent these defense mechanisms.

Benefiting from the intrinsic supervision information exploitation capability, contrastive learning has achieved promising performance in the field of deep graph clustering recently. However, we observe that two drawbacks of the positive and negative sample construction mechanisms limit the performance of existing algorithms from further improvement. 1) The quality of positive samples heavily depends on the carefully designed data augmentations, while inappropriate data augmentations would easily lead to the semantic drift and indiscriminative positive samples. 2) The constructed negative samples are not reliable for ignoring important clustering information. To solve these problems, we propose a Cluster-guided Contrastive deep Graph Clustering network (CCGC) by mining the intrinsic supervision information in the high-confidence clustering results. Specifically, instead of conducting complex node or edge perturbation, we construct two views of the graph by designing special Siamese encoders whose weights are not shared between the sibling sub-networks. Then, guided by the high-confidence clustering information, we carefully select and construct the positive samples from the same high-confidence cluster in two views. Moreover, to construct semantic meaningful negative sample pairs, we regard the centers of different high-confidence clusters as negative samples, thus improving the discriminative capability and reliability of the constructed sample pairs. Lastly, we design an objective function to pull close the samples from the same cluster while pushing away those from other clusters by maximizing and minimizing the cross-view cosine similarity between positive and negative samples. Extensive experimental results on six datasets demonstrate the effectiveness of CCGC compared with the existing state-of-the-art algorithms.

As one of the prevalent methods to achieve automation systems, Imitation Learning (IL) presents a promising performance in a wide range of domains. However, despite the considerable improvement in policy performance, the corresponding research on the explainability of IL models is still limited. Inspired by the recent approaches in explainable artificial intelligence methods, we proposed a model-agnostic explaining framework for IL models called R2RISE. R2RISE aims to explain the overall policy performance with respect to the frames in demonstrations. It iteratively retrains the black-box IL model from the randomized masked demonstrations and uses the conventional evaluation outcome environment returns as the coefficient to build an importance map. We also conducted experiments to investigate three major questions concerning frames' importance equality, the effectiveness of the importance map, and connections between importance maps from different IL models. The result shows that R2RISE successfully distinguishes important frames from the demonstrations.

Compressed videos often exhibit visually annoying artifacts, known as Perceivable Encoding Artifacts (PEAs), which dramatically degrade video visual quality. Subjective and objective measures capable of identifying and quantifying various types of PEAs are critical in improving visual quality. In this paper, we investigate the influence of four spatial PEAs (i.e. blurring, blocking, bleeding, and ringing) and two temporal PEAs (i.e. flickering and floating) on video quality. For spatial artifacts, we propose a visual saliency model with a low computational cost and higher consistency with human visual perception. In terms of temporal artifacts, self-attention based TimeSFormer is improved to detect temporal artifacts. Based on the six types of PEAs, a quality metric called Saliency-Aware Spatio-Temporal Artifacts Measurement (SSTAM) is proposed. Experimental results demonstrate that the proposed method outperforms state-of-the-art metrics. We believe that SSTAM will be beneficial for optimizing video coding techniques.