Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Lawrence CarinSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works164 from public data
- UNITER: UNiversal Image-TExt Representation Learning2,731
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2,008
An Attentional Generative Adversarial Network that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation and for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.
- Patient Knowledge Distillation for BERT Model Compression1,021
This work proposes a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student), which translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy.
- Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling836
Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that CLIPBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end- to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full- length videos, proving the proverbial less-is-more principle.
- GIT: A Generative Image-to-text Transformer for Vision and Language808
This paper design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering and establishes new state of the arts on 12 challenging benchmarks with a large margin.
- Ferret: Refer and Ground Anything Anywhere at Any Granularity625
The introduction of Ferret, a new Multimodal Large Language Model capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions, and a novel and powerful hybrid region representation that integrates discrete coordinates and continuous features jointly to represent a region in the image.
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training573
HELP, a novel framework for large-scale video+language omni-representation learning that achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains is presented.
- 563
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning561
To enable large-scale training, VILLA adopts the "free" adversarial training strategy, and combines it with KL-divergence-based regularization to promote higher invariance in the embedding space.
- An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA559
This work proposes PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA, and treats GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge.
- An Empirical Study of Training End-to-End Vision-and-Language Transformers482
Meter, a Multimodal End-to-end TransformER framework, is presented, through which it is investigated how to design and pre-train a fully transformer-based VL model in an end- to-end manner and provides insights on how to train a performant VL transformer.
- UNITER: Learning UNiversal Image-TExt Representations479
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- Semantic Compositional Networks for Visual Captioning453
Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics.
- Generalized Decoding for Pixel, Image, and Language404
X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks with strong transferability to a wide range of downstream tasks in both zero-shot and finetuning settings.
- Relation-Aware Graph Attention Network for Visual Question Answering400
A Relation-aware Graph Attention Network (ReGAT), which encodes each image into a graph and models multi-type inter-object relations via a graph attention mechanism, to learn question-adaptive relation representations.
- Prompting GPT-3 To Be Reliable389
This systematic empirical study sheds new insights on the reliability of prompting LLMs, but more importantly, its prompting strategies can help practitioners more reliably use LLMs like GPT-3.
- SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning356
SwinBERT is an end-to-end transformer-based model for video captioning, which takes video frame patches directly as inputs, and outputs a natural language description, and adopts a video transformer to encode spatial-temporal representations that can adapt to variable lengths of video input without dedicated design for different frame rates.
- Scaling Up Vision-Language Pretraining for Image Captioning342
LEMON O, a LargE-scale iMage captiONer, is presented, and the first empirical study on the scaling behavior of VLP for image captioning is provided, and it is shown LEMON can generate captions with long-tail vi-sual concepts when used in a zero-shot manner.
- StoryGAN: A Sequential Conditional GAN for Story Visualization312
A new story-to-image-sequence generation model, StoryGAN, based on the sequential conditional GAN framework is proposed, which outperformed state-of-the-art models in image quality, contextual consistency metrics, and human evaluation.
- Discourse-Aware Neural Extractive Text Summarization306
DiscoBert extracts sub-sentential discourse units (instead of sentences) as candidates for extractive selection on a finer granularity and outperforms state-of-the-art methods by a significant margin on popular summarization benchmarks compared to other BERT-base models.
- 302
- 268
- 223
- ReCo: Region-Controlled Text-to-Image Generation220
The proposed model, dubbed as ReCo (Region-Controlled T2I), enables the region control for arbitrary objects described by open-ended regional texts rather than by object labels from a constrained category set, and can better control the object count, spatial relationship, and region attributes such as color/size, with the free-form regional description.
- Guiding Instruction-based Image Editing via Multimodal Large Language Models212
This work investigates how MLLMs facilitate edit instructions and presents MLLM-Guided Image Editing (MGIE), which learns to derive expressive instructions and provides explicit guidance and can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-14. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.