Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileWorks103 from public data
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities1,446
The proposed MM-Vet is an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks, designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision-language (VL) capabilities.
- The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)1,017
This paper analyzes the latest model, GPT-4V(ision), to deepen the understanding of large multimodal models, and curate and organize a collection of carefully designed qualitative samples spanning a variety of domains and tasks to inspire future research on the next-generation multi-sensory task formulation.
- GIT: A Generative Image-to-text Transformer for Vision and Language808
This paper design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering and establishes new state of the arts on 12 challenging benchmarks with a large margin.
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action649
This paper defines and explores a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models and proposes MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.
- An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA559
This work proposes PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA, and treats GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge.
- TransVG: End-to-End Visual Grounding with Transformers522
A neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image by leveraging transformers and empirically showing that the complex fusion modules can be replaced by a simple stack of transformer encoder layers with higher performance.
- A Fast and Accurate One-Stage Approach to Visual Grounding462
A simple, fast, and accurate one-stage approach to visual grounding that enables end-to-end joint optimization and shows great potential in terms of both accuracy and speed for both phrase localization and referring expression comprehension.
- Prompting GPT-3 To Be Reliable389
This systematic empirical study sheds new insights on the reliability of prompting LLMs, but more importantly, its prompting strategies can help practitioners more reliably use LLMs like GPT-3.
- RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning361
This work proposes StarPO (State-Thinking-Actions-Reward Policy Optimization), a general framework for trajectory-level agent RL, and introduces RAGEN, a modular system for training and evaluating LLM agents and shows that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts.
- Scaling Up Vision-Language Pretraining for Image Captioning342
LEMON O, a LargE-scale iMage captiONer, is presented, and the first empirical study on the scaling behavior of VLP for image captioning is provided, and it is shown LEMON can generate captions with long-tail vi-sual concepts when used in a zero-shot manner.
- Improving One-Stage Visual Grounding by Recursive Sub-query Construction321
This work proposes a recursive sub-query construction framework, which reasons between image and query for multiple rounds and reduces the referring ambiguity step by step, and shows superior performances on longer and more complex queries validates the effectiveness of the query modeling.
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent278
This work develops a vision-language-action model in digital world, namely ShowUI, which features the following innovations: UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks.
- ReCo: Region-Controlled Text-to-Image Generation220
The proposed model, dubbed as ReCo (Region-Controlled T2I), enables the region control for arbitrary objects described by open-ended regional texts rather than by object labels from a constrained category set, and can better control the object count, spatial relationship, and region attributes such as color/size, with the free-form regional description.
- Attentive Relational Networks for Mapping Images to Scene Graphs194
A novel Attentive Relational Network that consists of two key modules with an object detection backbone to approach this problem, and accurate scene graphs are produced by the relation inference module to recognize all entities and corresponding relations.
- Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering190
This work manually curate 484 diverse real-world webpages as test cases and develops a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input.
- Disco: Disentangled Control for Realistic Human Dance Generation187
Disco is introduced, which includes a novel model architecture with disentangled control to improve the compositionality of dance synthesis, and an effective human attribute pre-training for better generalizability to unseen humans.
- SAT: 2D Semantics Assisted Training for 3D Visual Grounding179
2D Semantics Ass Training (SAT) is proposed that utilizes 2D image semantics in the training stage to ease point-cloud-language joint representation learning and assist 3D visual grounding.
- A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation177
A novel graph-based multi-modal fusion encoder for NMT that captures various semantic relationships between multi- modal semantic units (words and visual objects) and provides an attention-based context vector for the decoder.
- 176
- GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation166
The findings demonstrate that large multimodal models, specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities.
- TAP: Text-Aware Pre-training for Text-VQA and Text-Caption166
This paper proposes Text-Aware Pre-training (TAP) for Text-VQA and Text-Caption tasks, and builds a large-scale scene text-related imagetext dataset based on the Conceptual Caption dataset, named OCR-CC, which contains 1:4 million images with scene text.
- Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark163
E evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming, underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.
- End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perceptions162
This work proposes a multi-modal multi-task network to predict speed values and steering angles by taking previous feedback speeds and visual recordings as inputs and improves the failure data synthesis methods to solve the problem of error accumulation in real road tests.
- UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling154
UniTAB’s unified multi-task network and the task-agnostic output sequence design make the model parameter efficient and generalizable to new tasks, and could provide a more comprehensive and interpretable image description.
- PromptCap: Prompt-Guided Task-Aware Image Captioning150
PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs, achieves state-of-the-art accuracy on knowledge-based VQA tasks and generalizes well to unseen domains.
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-12. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.