多模式变压器的最新努力通过合并视觉和文本信息改善了视觉上丰富的文档理解（VRDU）任务。但是，现有的方法主要集中于诸如单词和文档图像贴片之类的细粒元素，这使得他们很难从粗粒元素中学习，包括短语和显着视觉区域（如突出的图像区域）等自然词汇单元。在本文中，我们对包含高密度信息和一致语义的粗粒元素更为重要，这对于文档理解很有价值。首先，提出了文档图来模拟多层次多模式元素之间的复杂关系，其中通过基于群集的方法检测到显着的视觉区域。然后，提出了一种称为mmlayout的多模式变压器，以将粗粒的信息纳入基于图形的现有预训练的细颗粒的多峰变压器中。在mmlayout中，粗粒信息是从细粒度聚集的，然后在进一步处理后，将其融合到细粒度中以进行最终预测。此外，引入常识增强以利用天然词汇单元的语义信息。关于四个任务的实验结果，包括信息提取和文档问答，表明我们的方法可以根据细粒元素改善多模式变压器的性能，并使用更少的参数实现更好的性能。定性分析表明，我们的方法可以在粗粒元素中捕获一致的语义。

估计到达时间（ETA）预测时间（也称为旅行时间估计）是针对各种智能运输应用程序（例如导航，路线规划和乘车服务）的基本任务。为了准确预测一条路线的旅行时间，必须考虑到上下文和预测因素，例如空间 - 周期性的互动，驾驶行为和交通拥堵传播的推断。先前在百度地图上部署的ETA预测模型已经解决了时空相互作用（constgat）和驾驶行为（SSML）的因素。在这项工作中，我们专注于建模交通拥堵传播模式以提高ETA性能。交通拥堵的传播模式建模具有挑战性，它需要考虑到随着时间的推移影响区域的影响区域，以及延迟变化随时间变化的累积影响，这是由于道路网络上的流量事件引起的。在本文中，我们提出了一个实用的工业级ETA预测框架，名为Dueta。具体而言，我们基于交通模式的相关性构建了一个对拥堵敏感的图，并开发了一种路线感知图形变压器，以直接学习路段的长距离相关性。该设计使Dueta能够捕获空间遥远但与交通状况高度相关的路段对之间的相互作用。广泛的实验是在从百度地图收集的大型现实世界数据集上进行的。实验结果表明，ETA预测可以从学习的交通拥堵传播模式中显着受益。此外，Dueta已经在Baidu Maps的生产中部署，每天都有数十亿个请求。这表明Dueta是用于大规模ETA预测服务的工业级和强大的解决方案。

预训练的模型（PTM）已成为自然语言处理和计算机视觉下游任务的基本骨干。尽管通过在BAIDU地图上将通用PTM应用于与地理相关的任务中获得的最初收益，但随着时间的流逝，表现平稳。造成该平稳的主要原因之一是缺乏通用PTM中的可用地理知识。为了解决这个问题，在本文中，我们介绍了Ernie-Geol，这是一个地理和语言预培训模型，设计和开发了用于改善Baidu Maps的地理相关任务。 Ernie-Geol经过精心设计，旨在通过预先培训从包含丰富地理知识的异质图生成的大规模数据来学习地理语言的普遍表示。大规模现实数据集进行的广泛定量和定性实验证明了Ernie-Geol的优势和有效性。自2021年4月以来，Ernie-Geol已经在百度地图上部署在生产中，这显着受益于各种下游任务的性能。这表明Ernie-Geol可以作为各种与地理有关的任务的基本骨干。

预先接受的语言模型实现了最先进的导致各种自然语言处理（NLP）任务。 GPT-3表明，缩放预先训练的语言模型可以进一步利用它们的巨大潜力。最近提出了一个名为Ernie 3.0的统一框架，以预先培训大型知识增强型号，并培训了具有10亿参数的模型。 Ernie 3.0在各种NLP任务上表现出最先进的模型。为了探讨缩放的表现，我们培养了百卢比的3.0泰坦参数型号，在PaddlePaddle平台上有高达260亿参数的泰坦。此外，我们设计了一种自我监督的对抗性损失和可控语言建模损失，以使ERNIE 3.0 TITAN产生可信和可控的文本。为了减少计算开销和碳排放，我们向Ernie 3.0泰坦提出了一个在线蒸馏框架，教师模型将同时教授学生和培训。埃塞尼3.0泰坦是迄今为止最大的中国密集预训练模型。经验结果表明，Ernie 3.0泰坦在68个NLP数据集中优于最先进的模型。

近年来，由于图表代表学习的出色表现，图形神经网络（GNN）技术在许多真实情景中获得了相当大的兴趣，例如推荐系统和社交网络。在推荐系统中，主要挑战是从其互动中学习有效的用户/项目表示。但是，由于它们对数据集和评估度量的差异，比较使用GNNS用于推荐系统的GNN的许多出版物。此外，其中许多只提供了一个演示，以对小型数据集进行实验，这很远可在现实世界推荐系统中应用。为了解决这个问题，我们介绍了Graph4Rec，这是一个Universal Toolkit，它统一地将GNN模型培训到以下部分：图表输入，随机步行生成，自我图形生成，对生成和GNNS选择。从这个训练管道，可以通过一些配置轻松建立自己的GNN模型。此外，我们开发了一个大规模的图形引擎和参数服务器，以支持分布式GNN培训。我们进行系统和全面的实验，以比较不同GNN模型在不同规模中的若干场景中的性能。证明了广泛的实验以识别GNN的关键组分。我们还尝试弄清楚稀疏和密集的参数如何影响GNN的性能。最后，我们研究了包括负面采样，自我图形建设顺序和温暖开始策略的方法，以找到更有效和高效的GNNS在推荐系统上做法。我们的工具包基于PGL HTTPS://github.com/paddlePaddle/pgl，并且在https://github.com/paddlepaddle/pgl/tree/main/apps/graph4rec中打开代码。

Increasing research interests focus on sequential recommender systems, aiming to model dynamic sequence representation precisely. However, the most commonly used loss function in state-of-the-art sequential recommendation models has essential limitations. To name a few, Bayesian Personalized Ranking (BPR) loss suffers the vanishing gradient problem from numerous negative sampling and predictionbiases; Binary Cross-Entropy (BCE) loss subjects to negative sampling numbers, thereby it is likely to ignore valuable negative examples and reduce the training efficiency; Cross-Entropy (CE) loss only focuses on the last timestamp of the training sequence, which causes low utilization of sequence information and results in inferior user sequence representation. To avoid these limitations, in this paper, we propose to calculate Cumulative Cross-Entropy (CCE) loss over the sequence. CCE is simple and direct, which enjoys the virtues of painless deployment, no negative sampling, and effective and efficient training. We conduct extensive experiments on five benchmark datasets to demonstrate the effectiveness and efficiency of CCE. The results show that employing CCE loss on three state-of-the-art models GRU4Rec, SASRec, and S3-Rec can reach 125.63%, 69.90%, and 33.24% average improvement of full ranking NDCG@5, respectively. Using CCE, the performance curve of the models on the test data increases rapidly with the wall clock time, and is superior to that of other loss functions in almost the whole process of model training.

In the scenario of black-box adversarial attack, the target model's parameters are unknown, and the attacker aims to find a successful adversarial perturbation based on query feedback under a query budget. Due to the limited feedback information, existing query-based black-box attack methods often require many queries for attacking each benign example. To reduce query cost, we propose to utilize the feedback information across historical attacks, dubbed example-level adversarial transferability. Specifically, by treating the attack on each benign example as one task, we develop a meta-learning framework by training a meta-generator to produce perturbations conditioned on benign examples. When attacking a new benign example, the meta generator can be quickly fine-tuned based on the feedback information of the new task as well as a few historical attacks to produce effective perturbations. Moreover, since the meta-train procedure consumes many queries to learn a generalizable generator, we utilize model-level adversarial transferability to train the meta-generator on a white-box surrogate model, then transfer it to help the attack against the target model. The proposed framework with the two types of adversarial transferability can be naturally combined with any off-the-shelf query-based attack methods to boost their performance, which is verified by extensive experiments.

Supervised Deep-Learning (DL)-based reconstruction algorithms have shown state-of-the-art results for highly-undersampled dynamic Magnetic Resonance Imaging (MRI) reconstruction. However, the requirement of excessive high-quality ground-truth data hinders their applications due to the generalization problem. Recently, Implicit Neural Representation (INR) has appeared as a powerful DL-based tool for solving the inverse problem by characterizing the attributes of a signal as a continuous function of corresponding coordinates in an unsupervised manner. In this work, we proposed an INR-based method to improve dynamic MRI reconstruction from highly undersampled k-space data, which only takes spatiotemporal coordinates as inputs. Specifically, the proposed INR represents the dynamic MRI images as an implicit function and encodes them into neural networks. The weights of the network are learned from sparsely-acquired (k, t)-space data itself only, without external training datasets or prior images. Benefiting from the strong implicit continuity regularization of INR together with explicit regularization for low-rankness and sparsity, our proposed method outperforms the compared scan-specific methods at various acceleration factors. E.g., experiments on retrospective cardiac cine datasets show an improvement of 5.5 ~ 7.1 dB in PSNR for extremely high accelerations (up to 41.6-fold). The high-quality and inner continuity of the images provided by INR has great potential to further improve the spatiotemporal resolution of dynamic MRI, without the need of any training data.

Recent studies have shown that using an external Language Model (LM) benefits the end-to-end Automatic Speech Recognition (ASR). However, predicting tokens that appear less frequently in the training set is still quite challenging. The long-tail prediction problems have been widely studied in many applications, but only been addressed by a few studies for ASR and LMs. In this paper, we propose a new memory augmented lookup dictionary based Transformer architecture for LM. The newly introduced lookup dictionary incorporates rich contextual information in training set, which is vital to correctly predict long-tail tokens. With intensive experiments on Chinese and English data sets, our proposed method is proved to outperform the baseline Transformer LM by a great margin on both word/character error rate and tail tokens error rate. This is achieved without impact on the decoding efficiency. Overall, we demonstrate the effectiveness of our proposed method in boosting the ASR decoding performance, especially for long-tail tokens.

The objective of this paper is to learn dense 3D shape correspondence for topology-varying generic objects in an unsupervised manner. Conventional implicit functions estimate the occupancy of a 3D point given a shape latent code. Instead, our novel implicit function produces a probabilistic embedding to represent each 3D point in a part embedding space. Assuming the corresponding points are similar in the embedding space, we implement dense correspondence through an inverse function mapping from the part embedding vector to a corresponded 3D point. Both functions are jointly learned with several effective and uncertainty-aware loss functions to realize our assumption, together with the encoder generating the shape latent code. During inference, if a user selects an arbitrary point on the source shape, our algorithm can automatically generate a confidence score indicating whether there is a correspondence on the target shape, as well as the corresponding semantic point if there is one. Such a mechanism inherently benefits man-made objects with different part constitutions. The effectiveness of our approach is demonstrated through unsupervised 3D semantic correspondence and shape segmentation.