Abstract

Vision, as a central component of human perception, plays a fundamental role in shaping natural language. To better understand how text models are connected to our visual perceptions, we propose a method for examining the similarities between neural representations extracted from words in text and objects in images. Our approach uses a lightweight probing model that learns to map language representations of concrete words to the visual domain. We find that representations from models trained on purely textual data, such as BERT, can be nontrivially mapped to those of a vision model. Such mappings generalize to object categories that were never seen by the probe during training, unlike mappings learned from permuted or random representations. Moreover, we find that the context surrounding objects in sentences greatly impacts performance. Finally, we show that humans significantly outperform all examined models, suggesting considerable room for improvement in representation learning and grounding.

Abstract

SSR节点免费发放🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀 ...:2021-12-5 · SSR节点免费 发放🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀🚀 @ssrList ... vpngfwmacvpssrrsurge小飞机场 免流量加速器 https://fanqiangdang.com 监制 💝广告合作: @zhibu Channel's geo & Language ...

Abstract

Fine-tuning pretrained contextual word embedding models to supervised downstream tasks has become commonplace in natural language processing. This process, however, is often brittle: even with the same hyperparameter values, distinct random seeds can lead to substantially different results. To better understand this phenomenon, we experiment with four datasets from the GLUE benchmark, fine-tuning BERT hundreds of times on each while varying only the random seeds. We find substantial performance increases compared to previously reported results, and we quantify how the performance of the best-found model varies as a function of the number of fine-tuning trials. Further, we examine two factors influenced by the choice of random seed: weight initialization and training data order. We find that both contribute comparably to the variance of out-ofsample performance, and that some weight initializations perform well across all tasks explored. On small datasets, we observe that many finetuning trials diverge part of the way through training, and we offer best practices for practitioners to stop training less promising runs early. We publicly release all of our experimental data, including training and validation scores for 2,100 trials, to encourage further analysis of training dynamics during fine-tuning.

Abstract

Systems that can associate images with their spoken audio captions are an important step towards visually grounded language learning. We describe a scalable method to automatically generate diverse audio for image captioning datasets. This supports pretraining deep networks for encoding both audio and images, which we do via a dual encoder that learns to align latent representations from both modalities. We show that a masked margin softmax loss for such models is superior to the standard triplet loss. We fine-tune these models on the Flickr8k Audio Captions Corpus and obtain state-of-the-art results---improving recall in the top 10 from 29.6% to 49.5%. We also obtain human ratings on retrieval outputs to better assess the impact of incidentally matching image-caption pairs that were not associated in the data, finding that automatic evaluation substantially underestimates the quality of the retrieved results.

Abstract

In instruction conditioned navigation, agents interpret natural language and their surroundings to navigate through an environment. Datasets for studying this task typically contain pairs of these instructions and reference trajectories. Yet, most evaluation metrics used thus far fail to properly account for the latter, relying instead on insufficient similarity comparisons. We address fundamental flaws in previously used metrics and show how Dynamic Time Warping (DTW), a long known method of measuring similarity between two time series, can be used for evaluation of navigation agents. For such, we define the normalized Dynamic Time Warping (nDTW) metric, that softly penalizes deviations from the reference path, is naturally sensitive to the order of the nodes composing each path, is suited for both continuous and graph-based evaluations, and can be efficiently calculated. Further, we define SDTW, which constrains nDTW to only successful paths. We collect human similarity judgments for simulated paths and find nDTW correlates better with human rankings than all other metrics. We also demonstrate that using nDTW as a reward signal for Reinforcement Learning navigation agents improves their performance on both the Room-to-Room (R2R) and Room-for-Room (R4R) datasets. The R4R results in particular highlight the superiority of SDTW over previous success-constrained metrics.

Abstract

好用的ssr飞机场 - 好看123:1 天前 · 好用的ssr飞机场,最新免费ssr飞机场,全网最全ssr公益机场地址,免费ssr节点2021,高速稳定ssr机场推荐,免费ssr飞机场,1元机场ssr,免费公益机场ssr,obfs安卓插件下载,ssr免费订阅地址

Abstract

Vision-and-Language Navigation (VLN) tasks such as Room-to-Room (R2R) require machine agents to interpret natural language instructions and learn to act in visually realistic environments to achieve navigation goals. The overall task requires competence in several perception problems: successful agents combine spatio-temporal, vision and language understanding to produce appropriate action sequences. Our approach adapts pre-trained vision and language representations to relevant in-domain tasks making them more effective for VLN. Specifically, the representations are adapted to solve both a cross-modal sequence alignment and sequence coherence task. In the sequence alignment task, the model determines whether an instruction corresponds to a sequence of visual frames. In the sequence coherence task, the model determines whether the perceptual sequences are predictive sequentially in the instruction-conditioned latent space. By transferring the domain-adapted representations, we improve competitive agents in R2R as measured by the success rate weighted by path length (SPL) metric.

Abstract

In a world where more than 70 million people rely on sign language to communicate, a system capable of recognizing and translating gestures to written or spoken language has great social impact. Despite rights claimed in recent decades, the deaf community still faces many challenges due to communication barriers. Gesture recognition, crucial for translation, is an active research topic in the computer vision and machine learning communities, and has been studied for decades. Among the most common approaches for this task, there are the electronic gloves with sensors, depth camera based approaches and simple camera based approaches. This last method has the advantage of completeness, since there are in many sign languages, including Brazilian Sign Language, which is the subject of this study, where other parts of the body such as the face and its expressions are needed to recognize some gestures. Additionally, it relies only on commonly found technologies, in contrast to the other approaches.

We present a state-of-the art approach, using a simple color camera for real-time static and continuous recognition of gestures from Brazilian Sign Language. For static recognition, we create a dataset with 33000 examples of 30 gestures; for continuous recognition, we create a dataset with 2000 videos containing phrases with 72 distinct gestures. Both datasets were built without restrictions with respect to clothing, background, lighting or distance between the camera and the user, commonly found in other studies. We propose end-to-end systems for each case using Deep Learning: for the former, a system based on a Deep Convolutional Residual Neural Network; for the latter, a hybrid architecture using Long-Short Term Memory Cells on top of convolutional layers. Our method shows state of the art accuraccy for both cases and is capable of running in real time on a GPU.

Abstract

In a world where more than 70 million people rely on sign language to communicate, a system capable of recognizing and translating gestures to written or spoken language has great social impact. Despite rights claimed in recent decades, the deaf community still faces many challenges due to communication barriers. Gesture recognition, crucial for translation, is an active research topic in the computer vision and machine learning communities, and has been studied for decades. Among the most common approaches for this task, there are the electronic gloves with sensors, depth camera based approaches and simple camera based approaches. This last method has the advantage of completeness, since there are in many sign languages, including Brazilian Sign Language, which is the subject of this study, where other parts of the body such as the face and its expressions are needed to recognize some gestures. Additionally, it relies only on commonly found technologies, in contrast to the other approaches.

We present a state-of-the art approach, using a simple color camera for real-time static recognition of gestures from Brazilian Sign Language. We create a dataset with 33000 examples of 30 gestures, without restrictions with respect to clothing, background, lighting or distance between the camera and the user, commonly found in other studies. We propose an end-to-end system using Deep Convolutional Residual Neural Networks, without the need to rely on laboriously engineered pipelines and feature extraction steps, as opposed to traditional approaches. The proposed system shows robustness for the classification of the 30 gestures, obtaining a 99.83% accuracy on the test set. The system runs in real time (26.2 ms per frame), using a NVIDIA K80 GPU.