Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileWorks103 from public data
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities1,446
The proposed MM-Vet is an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks, designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision-language (VL) capabilities.
- The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)1,017
This paper analyzes the latest model, GPT-4V(ision), to deepen the understanding of large multimodal models, and curate and organize a collection of carefully designed qualitative samples spanning a variety of domains and tasks to inspire future research on the next-generation multi-sensory task formulation.
- GIT: A Generative Image-to-text Transformer for Vision and Language808
This paper design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering and establishes new state of the arts on 12 challenging benchmarks with a large margin.
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action649
This paper defines and explores a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models and proposes MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.
- An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA559
This work proposes PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA, and treats GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge.
- TransVG: End-to-End Visual Grounding with Transformers522
A neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image by leveraging transformers and empirically showing that the complex fusion modules can be replaced by a simple stack of transformer encoder layers with higher performance.
- A Fast and Accurate One-Stage Approach to Visual Grounding462
A simple, fast, and accurate one-stage approach to visual grounding that enables end-to-end joint optimization and shows great potential in terms of both accuracy and speed for both phrase localization and referring expression comprehension.
- Prompting GPT-3 To Be Reliable389
This systematic empirical study sheds new insights on the reliability of prompting LLMs, but more importantly, its prompting strategies can help practitioners more reliably use LLMs like GPT-3.
- RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning361
This work proposes StarPO (State-Thinking-Actions-Reward Policy Optimization), a general framework for trajectory-level agent RL, and introduces RAGEN, a modular system for training and evaluating LLM agents and shows that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts.
- Scaling Up Vision-Language Pretraining for Image Captioning342
LEMON O, a LargE-scale iMage captiONer, is presented, and the first empirical study on the scaling behavior of VLP for image captioning is provided, and it is shown LEMON can generate captions with long-tail vi-sual concepts when used in a zero-shot manner.
- Improving One-Stage Visual Grounding by Recursive Sub-query Construction321
This work proposes a recursive sub-query construction framework, which reasons between image and query for multiple rounds and reduces the referring ambiguity step by step, and shows superior performances on longer and more complex queries validates the effectiveness of the query modeling.
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent278
This work develops a vision-language-action model in digital world, namely ShowUI, which features the following innovations: UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks.
- ReCo: Region-Controlled Text-to-Image Generation220
The proposed model, dubbed as ReCo (Region-Controlled T2I), enables the region control for arbitrary objects described by open-ended regional texts rather than by object labels from a constrained category set, and can better control the object count, spatial relationship, and region attributes such as color/size, with the free-form regional description.
- Attentive Relational Networks for Mapping Images to Scene Graphs194
A novel Attentive Relational Network that consists of two key modules with an object detection backbone to approach this problem, and accurate scene graphs are produced by the relation inference module to recognize all entities and corresponding relations.
- Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering190
This work manually curate 484 diverse real-world webpages as test cases and develops a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input.
- Disco: Disentangled Control for Realistic Human Dance Generation187
Disco is introduced, which includes a novel model architecture with disentangled control to improve the compositionality of dance synthesis, and an effective human attribute pre-training for better generalizability to unseen humans.
- SAT: 2D Semantics Assisted Training for 3D Visual Grounding179
2D Semantics Ass Training (SAT) is proposed that utilizes 2D image semantics in the training stage to ease point-cloud-language joint representation learning and assist 3D visual grounding.
- A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation177
A novel graph-based multi-modal fusion encoder for NMT that captures various semantic relationships between multi- modal semantic units (words and visual objects) and provides an attention-based context vector for the decoder.
- 176
- GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation166
The findings demonstrate that large multimodal models, specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities.
- TAP: Text-Aware Pre-training for Text-VQA and Text-Caption166
This paper proposes Text-Aware Pre-training (TAP) for Text-VQA and Text-Caption tasks, and builds a large-scale scene text-related imagetext dataset based on the Conceptual Caption dataset, named OCR-CC, which contains 1:4 million images with scene text.
- Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark163
E evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming, underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.
- End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perceptions162
This work proposes a multi-modal multi-task network to predict speed values and steering angles by taking previous feedback speeds and visual recordings as inputs and improves the failure data synthesis methods to solve the problem of error accumulation in real road tests.
- UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling154
UniTAB’s unified multi-task network and the task-agnostic output sequence design make the model parameter efficient and generalizable to new tasks, and could provide a more comprehensive and interpretable image description.
- PromptCap: Prompt-Guided Task-Aware Image Captioning150
PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs, achieves state-of-the-art accuracy on knowledge-based VQA tasks and generalizes well to unseen domains.
- 127
- TransVG++: End-to-End Visual Grounding With Language Conditioned Vision Transformer116
This work proposes TransVG, which establishes multi-modal correspondences by Transformers and localizes referred regions by directly regressing box coordinates, and introduces TransVG++, which upgrades the framework to a purely Transformer-based one by leveraging Vision Transformer (ViT) for vision feature encoding.
- 115
- Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation103
This work leverage a generic object detector at training time, and proposes a contrastive learning framework that accounts for both region-phrase and image-sentence matching, which achieves state-of-the-art results on visual phrase grounding, surpassing previous methods that require expensive object detectors at test time.
- ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding100
ReFocus is introduced, a simple yet effective framework that equips multimodal LLMs with the ability to generate visual thoughts by performing visual editing on the input image through code, shifting and refining their visual focuses, thereby enhancing the visual reasoning process.
- MM-VID: Advancing Video Understanding with GPT-4V(ision)98
MM-VID is an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video understanding and demonstrates the effectiveness of MM-VID in handling distinct video genres with various video lengths.
- Grounding-Tracking-Integration95
This paper proposes a framework called GTI that decomposes the problem into three sub-tasks: Grounding, Tracking, and Integration, and proposes an "RT-integration" method that defines and predicts two scores to guide the integration.
- Equivariant Similarity for Vision-Language Foundation Models74
This study proposes EqSim, a regularization loss that can be efficiently calculated from any two matched training pairs and easily pluggable into existing image-text retrieval fine-tuning and presents a new challenging benchmark EqBen, the first to focus on "visual-minimal change".
- Curve fitting and optimal interpolation for CNC machining under confined error using quadratic B-splines66
An optimization problem is solved to find a quadratic B-spline curve whose Hausdorff distance to the given polyline tool path is within a given precision, and adopting time parameter for the fitting curve is adopted.
- MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning62
Experimental results on MAD-eval dataset demonstrate that MM-Narrator consistently outperforms both the existing fine-tuning-based approaches and LLM-based approaches in most scenarios, as measured by standard evaluation metrics.
- GPT-4V(ision) as A Social Media Analysis Engine54
GPT-4V shows effectiveness in these tasks, exhibiting capabilities like joint image–text understanding, contextual and cultural awareness, and commonsense knowledge application, however, challenges persist, including struggles with multilingual social multimedia comprehension and difficulty in adapting to the latest social media trends.
- 50
- SGFormer: Semantic Graph Transformer for Point Cloud-Based 3D Scene Graph Generation45
A novel model called SGFormer, Semantic Graph TransFormer for point cloud-based 3D scene graph generation, which introduces the graph embedding layer to best utilize the global information in graph edges while maintaining comparable computation costs and the semantic injection layer to leverage linguistic knowledge from large-scale language model, to enhance objects' visual features.
- 40
- 38
- List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs32
This work proposes a new learning paradigm:"list items one by one," which asks the model to enumerate and describe all visual tags placed on the image following the alphanumerics orders of tags, and finds that this new dataset, even in a relatively small size, significantly enhances visual reasoning capabilities and reduces hallucinations for MLLMs.
- 31
- SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation31
This work introduces SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation that incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference-time fast learning strategy based on a temporal LoRA module.
- Human-Centered Emotion Recognition in Animated GIFs29
This study demonstrates the importance of human related information in GIFs and conducts human-centered GIF emotion recognition with a proposed Keypoint Attended Visual Attention Network (KAVAN), and proposes a framework that outperforms the state-of-the-art on the MIT GIFGIF dataset.
- Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization28
An algorithm named pairwise sample optimization (PSO) is presented, which enables the direct fine-tuning of an arbitrary timestep-distilled diffusion model, and can directly adapt distilled models to human-preferred generation with both offline and online-generated pairwise preference image data.
- 28
- Pose-based Body Language Recognition for Emotion and Psychiatric Symptom Interpretation27
The accuracy and transferability of the proposed body language recognition method on several public action recognition datasets is validated and the framework outperforms other methods on the URMC dataset.
- MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videos26
This work has meticulously curated the MMSum dataset, providing a comprehensive dataset for this new direction of multimodal summarization with multimodal output, and released the MMSum dataset and the data collection tool as fully open-source resources.
- TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation24
This work introduces TextAtlas5M, a novel dataset specifically designed to evaluate long-text rendering in text-conditioned image generation, consisting of 5 million long-text generated and collected images across diverse data types, enabling comprehensive evaluation of large-scale generative models on long-text image generation.
- DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design23
DEsignBench, a text-to-image (T2I) generation benchmark tailored for visual design scenarios, is introduced and the first automatic image generation evaluator powered by GPT-4V is introduced, providing ratings that align well with human judgments, while being easily replicable and cost-efficient.
- 22
- COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training22
This work introduces \VideoDatasetName, an inaugural interleaved video-text dataset featuring comprehensive captions, marking a significant step forward in the evolution of Vision-Language Pre-training and enhancing model performance in image-text tasks.
- Bring Metric Functions into Diffusion Models21
A Cascaded Diffusion Model (Cas-DM) is introduced that improves a Denoising Diffusion Probabilistic Model (DDPM) by effectively incorporating additional metric functions in training, and enables the effective use of the LPIPS loss on various established benchmarks.
- Action Recognition with Visual Attention on Skeleton Images19
This work redesigns skeleton representations with a depth-first tree traversal order, which enhances the semantic meaning of skeleton images and better preserves the structural information, and proposes a two-branch attention architecture that focuses on spatio-temporal key stages and filters out unreliable joint predictions.
- OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation18
A new interleaved generation framework based on prompting large-language models (LLMs) and pre-trained text-to-image (T2I) models, namely OpenLEAF is proposed, which can generate high-quality image-text content for various domains and applications.
- Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising17
This paper proposes AVED, a zero-shot cross-modal delta denoising framework that leverages audio-video interactions to achieve synchronized and coherent edits, and demonstrates superior results on both AVED-Bench and the recent OAVE dataset to validate its generalization capabilities.
- Study on the polymerization shrinkage and mechanical-physical properties of dental resin composite modified by polyurethane dimechacrylate and glass flake fillers16
This new synthesized composite could perform low polymerization shrinkage and better mechanical resistance, which may provide a new idea for improving the lifespan of direct restoration.
- Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation15
LayoutBench is proposed, a diagnostic benchmark for layout-guided image generation that examines four categories of spatial control skills: number, position, size, and shape, and IterInpaint is proposed, a new baseline that generates foreground and background regions step-by-step via inpainting, demonstrating stronger generalizability than existing models on OOD layouts in LayoutBench.
- Toward Generalist Autonomous Research via Hypothesis-Tree Refinement13
This work introduces Arbor, a general framework for autonomous research that combines a long-lived coordinator, short-lived executors, and Hypothesis Tree Refinement (HTR), a persistent tree that links hypotheses, artifacts, evidence, and distilled insights across time.
- A Practical Application for Quantitative Brain Fatigue Evaluation Based on Machine Learning and Ballistocardiogram12
The results suggested that the brain fatigue quantification evaluation method based on the optical fiber sensor cushion and machine learning can carry out real-time brain fatigue detection on the human brain without disturbance, reduce the risk of human accidents in human–machine interaction systems, and improve mental health among the office and driving personnel.
- 12
- Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models11
This paper designs a novel architecture SFUNet to effectively capture the correlation for both spatial and frequency domains in wavelet data and finds the model is able to generate images with higher quality on CIFAR-10, FFHQ, LSUN-Bedroom, and LSun-Church datasets, than the pixel-based counterpart.
- What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding7
FoReaL-Decoding is established as a simple, plug-and-play route to controllable cost-quality trade-offs in reasoning-centric tasks, driven by the Local Misalignment Diminish.
- 7
- 7
- 7
- Cross-modal Contrastive Distillation for Instructional Activity Anticipation7
A novel cross-modal contrastive distillation (CCD) scheme is devised, which facilitates knowledge distillation between teacher and student in heterogeneous modalities with the proposed cross- modal distillation loss.
- 5
- InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Models5
A visual dialogue dataset, named InfoVisDial, which provides rich informative answers in each round even with external knowledge related to the visual content, and proposes a strong baseline by adapting the GIT model for the visual dialogue task and fine-tune the model on InfoVisDial.
- Interfacing Foundation Models' Embeddings5
FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity, is presented, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity.
- Weakly Supervised Body Part Parsing with Pose based Part Priors.5
This work proposes the first weakly supervised body part parsing framework and shows that the proposed framework can be extended to other keypoint-supervised part parsing tasks such as face parsing.
- Conditional Text-to-Image Generation with Reference Guidance4
This paper develops several small-scale expert plugins that efficiently endow a Stable Diffusion model with the capability to take different references, and develops several small-scale expert plugins that efficiently endow a Stable Diffusion model with the capability to take different references.
- 4
- 4
- OpenLEAF: A Novel Benchmark for Open-Domain Interleaved Image-Text Generation4
This work introduces OpenLEAF, a new benchmark designed for the emerging open-domain interleaved image-text generation task, and proposes a novel evaluation pipeline named "detection-summarization-scoring," which breaks down the evaluation into multiple reasoning steps.
- Entity6K: A Large Open-Domain Evaluation Dataset for Real-World Entity Recognition4
This work introduces Entity6K, a comprehensive dataset for real-world entity recognition, featuring 5,700 entities across 26 categories, each supported by 5 human-verified images with annotations, addressing a gap in existing datasets.
- 4
- 4
- 4
- 3
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documents1
Vision-Centric Contrastive Learning (VC2L), a unified framework that models text, images, and their combinations using a single vision transformer, is proposed, and empirical results show that VC2L achieves competitive or superior performance compared to CLIP-style models on both the proposed benchmarks and established datasets.
- Development of innovative light-curing resin composites featuring dual hydrolysis resistance1
This novel resin composites demonstrate superior hydrolytic resistance compared to the commercial resin Z250XT, potentially offering a new approach to extending the service life of dental restorations.
- 1
- –
- –
- Study on a novel dental resin matrix with a fluorinated polyurethane dimethacrylate monomer–
The addition of FPUDMA monomer enhanced hydrolytic resistance, degree of conversion, mechanical properties, and biocompatibility, while reducing polymerization shrinkage and initial bacterial adhesion in vitro suggest a potential for enhanced restoration performance.
- –
- –
- –
- –
- –
- –
- –
- The Standardized Administration of Software Project–
The problems in the development of software project are discussed in this article and the accessible condition, main project administration and the results in different stages are analyzed.
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-12. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.