Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileWorks92 from public data
- UNITER: UNiversal Image-TExt Representation Learning2,731
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)1,017
This paper analyzes the latest model, GPT-4V(ision), to deepen the understanding of large multimodal models, and curate and organize a collection of carefully designed qualitative samples spanning a variety of domains and tasks to inspire future research on the next-generation multi-sensory task formulation.
- Segment Everything Everywhere All at Once899
This work proposes a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks, aiming at a universal segmentation interface that behaves like large language models (LLMs).
- Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling836
Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that CLIPBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end- to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full- length videos, proving the proverbial less-is-more principle.
- GIT: A Generative Image-to-text Transformer for Vision and Language808
This paper design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering and establishes new state of the arts on 12 challenging benchmarks with a large margin.
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action649
This paper defines and explores a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models and proposes MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training573
HELP, a novel framework for large-scale video+language omni-representation learning that achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains is presented.
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning561
To enable large-scale training, VILLA adopts the "free" adversarial training strategy, and combines it with KL-divergence-based regularization to promote higher invariance in the embedding space.
- UNITER: Learning UNiversal Image-TExt Representations479
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- Generalized Decoding for Pixel, Image, and Language404
X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks with strong transferability to a wide range of downstream tasks in both zero-shot and finetuning settings.
- Relation-Aware Graph Attention Network for Visual Question Answering400
A Relation-aware Graph Attention Network (ReGAT), which encodes each image into a graph and models multi-type inter-object relations via a graph attention mechanism, to learn question-adaptive relation representations.
- SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning356
SwinBERT is an end-to-end transformer-based model for video captioning, which takes video frame patches directly as inputs, and outputs a natural language description, and adopts a video transformer to encode spatial-temporal representations that can adapt to variable lengths of video input without dedicated design for different frame rates.
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning286
This paper introduces the first large and diverse visual instruction tuning dataset, named Large-scale Robust Visual (LRV)-Instruction, and proposes GPT4-Assisted Visual Instruction Evaluation (GAVIE), a novel approach to evaluate visual instruction tuned without the need for human-annotated groundtruth answers and can adapt to diverse instruction formats.
- ReCo: Region-Controlled Text-to-Image Generation220
The proposed model, dubbed as ReCo (Region-Controlled T2I), enables the region control for arbitrary objects described by open-ended regional texts rather than by object labels from a constrained category set, and can better control the object count, spatial relationship, and region attributes such as color/size, with the free-form regional description.
- Disco: Disentangled Control for Realistic Human Dance Generation187
Disco is introduced, which includes a novel model architecture with disentangled control to improve the compositionality of dance synthesis, and an effective human attribute pre-training for better generalizability to unseen humans.
- Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone174
This work presents FIBER (Fusion-In-the-Backbone-based transformER), a new VL model architecture that can seamlessly handle both these types of tasks and provides consistent performance improvements over strong baselines across all tasks, often outperforming methods using magnitudes more data.
- GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation166
The findings demonstrate that large multimodal models, specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities.
- Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark163
E evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming, underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.
- 156
- 127
- UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training114
UC2, the first machine translation-augmented framework for cross-lingual cross-modal representation learning, is introduced, to tackle the scarcity problem of multilingual captions for image datasets and facilitate the learning of a joint embedding space of images and all languages of interest.
- Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models113
Surprisingly, it is found that during dataset collection, non-expert annotators can easily attack SOTA VQA models successfully, revealing the fragility of these models while demonstrating the effectiveness of the adversarial dataset.
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog111
This paper presents a new model for visual dialog, Recurrent Dual Attention Network (ReDAN), using multi-step reasoning to answer a series of questions about an image, and demonstrates that ReDAN can locate context-relevant visual and textual clues via iterative refinement, which can lead to the correct answer step-by-step.
- LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling104
Surprisingly, experimental results show that this unified VidL framework LAVENDER achieves competitive performance on 14 VidL benchmarks, covering video question answering, text-to-video retrieval and video captioning.
- MM-VID: Advancing Video Understanding with GPT-4V(ision)98
MM-VID is an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video understanding and demonstrates the effectiveness of MM-VID in handling distinct video genres with various video lengths.
- An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling95
This work systematically examines the potential of MVM in the context of VidL learning, and conducts comprehensive experiments and provides insights into the factors leading to effective MVM training, resulting in an enhanced model VIOLETv2.
- LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval95
A simple yet highly effective approach, LightningDOT is proposed that accelerates the inference time of ITR by thousands of times, without sacrificing accuracy, and achieves superior performance across mainstream ITR benchmarks such as Flickr30k and COCO datasets.
- 88
- Meta Module Network for Compositional Visual Reasoning85
This work proposes Meta Module Network (MMN), a more powerful NMN architecture centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically, inheriting the strong explainability and compositionality of NMN.
- Equivariant Similarity for Vision-Language Foundation Models74
This study proposes EqSim, a regularization loss that can be efficiently calculated from any two matched training pairs and easily pluggable into existing image-text retrieval fine-tuning and presents a new challenging benchmark EqBen, the first to focus on "visual-minimal change".
- MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities68
MM-Vet v2 is introduced, which includes a new VL capability called "image-text sequence understanding", evaluating models' ability to process VL sequences, and maintains the high quality of evaluation samples while further expanding the evaluation set size.
- Playing Lottery Tickets with Vision and Language64
This work performs the first empirical study to assess whether trainable subnetworks also exist in pre-trained VL models, and finds that LTH remains relevant when using other training methods (e.g., adversarial training).
- GenXD: Generating Any 3D and 4D Scenes56
GenXD, a framework that allows for general 3D and 4D generation by leveraging camera and object movements commonly observed in daily life, and performs extensive evaluations across various real-world and synthetic datasets, demonstrating GenXD's effectiveness and versatility compared to previous methods in 3D and 4D generation.
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback53
This work introduces Test-time Preference Optimization (TPO), a framework that aligns LLM outputs with human preferences during inference, removing the need to update model parameters, and exploits the innate capacity of LLM to interpret and act upon reward signals.
- ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning53
ThinkMorph is built, a unified model fine-tuned on approximately 24K high-quality interleaved reasoning traces spanning tasks with varying visual engagement that learns to generate progressive text-image reasoning steps that concretely manipulate visual content while maintaining coherent verbal logic.
- 50
- MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos43
This work introduces MMWorld, a new benchmark for multi-discipline, multi-faceted multimodal video understanding, and hopes it can serve as an essential step towards world model evaluation in videos.
- 41
- Crossmodal Representation Learning for Zero-shot Action Recognition39
A cross-modal Transformer-based frame-work, which jointly encodes video data and text labels for zero-shot action recognition (ZSAR), which employs a conceptually new pipeline by which visual representations are learned in conjunction with visual-semantic associations in an end-to-end manner.
- 38
- 31
- SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation31
This work introduces SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation that incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference-time fast learning strategy based on a temporal LoRA module.
- 27
- MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos26
This work has meticulously curated the MMSum dataset, providing a comprehensive dataset for this new direction of multimodal summarization with multimodal output, and released the MMSum dataset and the data collection tool as fully open-source resources.
- FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow25
FullFront is presented, a benchmark designed to evaluate Multimodal Large Language Models (MLLMs), and quantitatively demonstrate performance disparities across models and tasks, and highlight a substantial gap between current MLLM capabilities and human expert performance in front-end engineering.
- TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation24
This work introduces TextAtlas5M, a novel dataset specifically designed to evaluate long-text rendering in text-conditioned image generation, consisting of 5 million long-text generated and collected images across diverse data types, enabling comprehensive evaluation of large-scale generative models on long-text image generation.
- DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design23
DEsignBench, a text-to-image (T2I) generation benchmark tailored for visual design scenarios, is introduced and the first automatic image generation evaluator powered by GPT-4V is introduced, providing ratings that align well with human judgments, while being easily replicable and cost-efficient.
- Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations22
STARE is introduced, a benchmark designed to rigorously evaluate multimodal large language models on tasks better solved through multi-step visual simulation, and shows that models excel at reasoning over simpler 2D transformations, but perform close to random chance on more complex tasks.
- COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training22
This work introduces \VideoDatasetName, an inaugural interleaved video-text dataset featuring comprehensive captions, marking a significant step forward in the evolution of Vision-Language Pre-training and enhancing model performance in image-text tasks.
- 22
- Bring Metric Functions into Diffusion Models21
A Cascaded Diffusion Model (Cas-DM) is introduced that improves a Denoising Diffusion Probabilistic Model (DDPM) by effectively incorporating additional metric functions in training, and enables the effective use of the LPIPS loss on various established benchmarks.
- OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation18
A new interleaved generation framework based on prompting large-language models (LLMs) and pre-trained text-to-image (T2I) models, namely OpenLEAF is proposed, which can generate high-quality image-text content for various domains and applications.
- Extracting Human Face Similarity Judgments: Pairs or Triplets?18
Using face similarity data collected on Amazon Mechanical Turk, it is demonstrated that triplet ranking is significantly better for extracting subject-specific preferences, while the two are comparable when pooling across subjects.
- Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising17
This paper proposes AVED, a zero-shot cross-modal delta denoising framework that leverages audio-video interactions to achieve synchronized and coherent edits, and demonstrates superior results on both AVED-Bench and the recent OAVE dataset to validate its generalization capabilities.
- Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness17
A taxonomy of uncertainty specific to vision-language AI systems is presented, distinguishing between epistemic uncertainty (arising from a lack of information) and aleatoric uncertainty (due to inherent unpredictability), and further explore finer categories within.
- EditRoom: LLM-parameterized Graph Diffusion for Composable 3D Room Layout Editing16
This work proposes EditRoom, a unified framework capable of executing a variety of layout edits through natural language commands, without requiring manual intervention, and develops an automatic pipeline to augment existing 3D scene synthesis datasets to address the lack of data.
- 16
- Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation15
LayoutBench is proposed, a diagnostic benchmark for layout-guided image generation that examines four categories of spatial control skills: number, position, size, and shape, and IterInpaint is proposed, a new baseline that generates foreground and background regions step-by-step via inpainting, demonstrating stronger generalizability than existing models on OOD layouts in LayoutBench.
- Adaptive Human Matting for Dynamic Videos15
This work introduces Adaptive Matting for Dynamic Videos, termed AdaM, which is a framework designed for simultaneously differentiating foregrounds from backgrounds and capturing alpha matte details of human subjects in the foreground, and achieves new best-in-class generalizability.
- Cross-Adversarial Learning for Molecular Generation in Drug Design13
A cross-adversarial learning method for molecular generation, CRAG for short, which integrates both the facticity of VAE- based methods and the diversity of GAN-based methods to further exploit the complex properties of Molecules is proposed.
- 12
- Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT9
MVPBench is introduced, a curated benchmark designed to rigorously evaluate visual physical reasoning through the lens of visual chain-of-thought (CoT), and a graph-based CoT consistency metric is introduced that verifies whether the reasoning path of model adheres to valid physical logic.
- TextGround4M: A Prompt-Aligned Dataset for Layout-Aware Text Rendering5
TextGround4M, a large-scale dataset of over 4 million prompt-image pairs, each annotated with span-level text grounded in the prompt and corresponding bounding boxes, enables fine-grained supervision for layout-aware, prompt-grounded text rendering and introduces two layout-aware metrics to address the long-standing lack of spatial evaluation in text rendering.
- 5
- 5
- OpenLEAF: A Novel Benchmark for Open-Domain Interleaved Image-Text Generation4
This work introduces OpenLEAF, a new benchmark designed for the emerging open-domain interleaved image-text generation task, and proposes a novel evaluation pipeline named "detection-summarization-scoring," which breaks down the evaluation into multiple reasoning steps.
- Entity6K: A Large Open-Domain Evaluation Dataset for Real-World Entity Recognition4
This work introduces Entity6K, a comprehensive dataset for real-world entity recognition, featuring 5,700 entities across 26 categories, each supported by 5 human-verified images with annotations, addressing a gap in existing datasets.
- 4
- 4
- 3
- 2
- 2
- Residual Decoder Adapter: ID-Preserving Tokenizer Adaption for Autoregressive Text Rendering1
The Residual Decoder Adapter (RDA) is proposed that upgrades an existing tokenizer post-hoc without changing its token space and substantially improves text rendering significantly by a large margin.
- Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documents1
Vision-Centric Contrastive Learning (VC2L), a unified framework that models text, images, and their combinations using a single vision transformer, is proposed, and empirical results show that VC2L achieves competitive or superior performance compared to CLIP-style models on both the proposed benchmarks and established datasets.
- 1
- 1
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
- Social Impression of Faces: From Prediction to Modification–
A state-of-the-art predictive model of facial impressions, such as facial attractiveness, trustworthiness, and intelligence, is applied to a large-scale natural face dataset in order to create a robust facial impression dataset and is validated with respect to human judgments.
- –
- From prediction to modification - modeling first impression of faces–
This work develops a method to predict human impressions of faces in 40 social dimensions, and empirically shows that model performance grows as the human consensus on a face trait increases, illustrating the learnability of subjective social perception of faces, especially when there is high human consensus.
- Understanding human facial attractiveness from multiple views.–
This study investigates facial attractiveness in the context of social feature evaluation and finds that social attributes like appearing interesting and sociable contribute to facial attractiveness whereas appearing boring, and humble are negatively correlated with attractiveness.
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-12. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.