Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Hongsheng LiSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works108 from public data
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation1,235
This work proposes a new T2V generation setting—One-Shot Video Tuning, where only one text-video pair is presented, and introduces Tune-A-Video, which involves a tailored spatio-temporal attention mechanism and an efficient one-shot tuning strategy.
- YOLO-World: Real-Time Open-Vocabulary Object Detection1,134
YOLO-World is introduced, an innovative approach that enhances YOLO with open-vocabulary detection capabilities through vision-language modeling and pre-training on large-scale datasets and proposes a new Re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN) and region-text contrastive loss to facilitate the interaction between visual and linguistic information.
- SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension1,018
This work addresses the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment ofGenerative models, by introducing a benchmark named SEED-Bench, which spans 12 evaluation dimensions including the comprehension of both the image and video modality.
- Self-paced Contrastive Learning with Hybrid Memory for Domain Adaptive Object Re-ID691
This work proposes a novel self-paced contrastive learning framework with hybrid memory that outperforms state-of-the-arts on multiple domain adaptation tasks of object re-ID and even boosts the performance on the source domain without any extra annotations.
- 402
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation365
This work presents a unified and versatile foundation model, namely, SEED-X, which is able to model multi-granularity visual semantics for comprehension and generation tasks and demonstrates its effectiveness in handling real-world applications across various domains after instruction tuning.
- UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition365
Large kernels are the key to unlocking the exceptional performance of ConvNets in domains where they were originally not proficient, and the proposed model achieves state- of-the-art performance on time-series forecasting and audio recognition tasks even without modality-specific customization to the architecture.
- GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction340
The GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools, generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts using the Low-Rank Adaptation (LoRA) optimization.
- 327
- 294
- All in One: Exploring Unified Video-Language Pre-Training258
This work introduces an end-to-end VLP model, namely all-in-one Transformer, that embeds raw video and textual signals into joint representations using a unified backbone architecture and introduces a novel and effective token rolling operation to encode temporal representations from video clips in a non-parametric manner.
- Making LLaMA SEE and Draw with SEED Tokenizer227
SEED is introduced, an elaborate image tokenizer that empowers LLMs with the ability to SEE and Draw at the same time, and with SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction.
- SmartEdit: Exploring Complex Instruction-Based Image Editing with Multimodal Large Language Models213
A Bidirectional Interaction Module (BIM) is proposed that enables comprehensive bidirectional information interactions between the input image and the MLLM output and demonstrates that a small amount of complex instruction editing data can effectively stimulate SmartEdit’ s editing capabilities for more complex instructions.
- Self-supervising Fine-Grained Region Similarities for Large-Scale Image Localization210
This work proposes to self-supervise image-to-region similarities in order to fully explore the potential of difficult positive images alongside their sub-regions and outperforms state-of-the-arts on the standard localization benchmarks and shows excellent generalization capability on multiple image retrieval datasets.
- SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension156
SEED-Bench-2-Plus is introduced, a benchmark specifically designed for evaluating text-rich visual comprehension of Multimodal Large Language Models and a thorough evaluation involving 34 prominent MLLMs is conducted, to emphasize the current limitations of MLLMs in text-rich visual comprehension.
- Refining Pseudo Labels with Clustering Consensus over Generations for Unsupervised Object Re-identification154
This work proposes to properly estimate pseudo label similarities between consecutive training generations with clustering consensus and refine pseudo labels with temporally propagated and ensembled pseudo labels to improve classification with dynamically changing classes over generations.
- Planting a SEED of Vision in Large Language Model150
This study identifies two crucial principles for the architecture and training of SEED that effectively ease subsequent alignment with LLMs, and emphasizes the great potential of discrete visual tokens in versatile multimodal LLMs and the importance of proper image tokenizers in broader research.
- Bridging Video-text Retrieval with Multiple Choice Questions131
This work enables fine-grained video-text interactions while maintaining high efficiency for retrieval via a novel pretext task, dubbed as Multiple Choice Questions (MCQ), where a parametric module BridgeFormer is trained to answer the “questions” constructed by the text features via resorting to the video features.
- 113
- Progressive Correspondence Pruning by Consensus Learning112
This work introduces a "pruning" block that lets us identify reliable candidates among the initial matches according to consensus scores estimated using local-to-global dynamic graphs and achieves progressive pruning by stacking multiple pruning blocks sequentially.
- DreamDiffusion: Generating High-Quality Images from Brain EEG Signals111
Quantitative and qualitative results demonstrate the effectiveness of the proposed DreamDiffusion method as a significant step towards portable and low-cost ``thoughts-to-image'', with potential applications in neuroscience and computer vision.
- SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing106
A pretrained Multimodal Large Language Model (MLLM) that unifies comprehension and generation with SEED-Data-Edit is fine-tuneed, indicating the potential and effectiveness of SEED-Data-Edit in advancing the field of instructional image editing.
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models104
This work proposes ATP-LLaVA, a novel approach that adaptively determines instance-specific token pruning ratios for each LLM layer, and introduces an Adaptive Token Pruning (ATP) module, which computes the importance score and pruning threshold based on input instance adaptively.
- SEED-Bench-2: Benchmarking Multimodal Large Language Models103
This work categorizes the capabilities of MLLMs into hierarchical levels from $L_0 to $L-4 based on the modalities they can accept and generate, and proposes SEED-Bench-2, a comprehensive benchmark that evaluates the hierarchical capabilities ofMLLMs.
- 100
- DivCo: Diverse Conditional Image Synthesis via Contrastive Generative Adversarial Network95
This paper proposes a novel DivCo framework to properly constrain both "positive" and "negative" relations between the generated images specified in the latent space and introduces a novel latent-augmented contrastive loss, which encourages images generated from adjacent latent codes to be similar and those generated from distinct latent code to be dissimilar.
- VoCo-LLaMA: Towards Vision Compression with Large Language Models92
VoCo-LLaMA is proposed, the first approach to compress vision tokens using LLMs, and demonstrates the ability to understand temporal correlations, outperforming previous methods on popular video question-answering benchmarks.
- Object-aware Video-language Pre-training for Retrieval92
Object-aware Transformers is presented, an object-centric approach that extends video-language transformer to incorporate object representations to leverage the bounding boxes and object tags to guide the training process.
- Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos84
This work introduces Moto, which converts video content into latent Motion Token sequences by a Latent Motion Tokenizer, learning a bridging “language” of motion from videos in an unsupervised manner, and implements a co-fine-tuning strategy that seamlessly bridges latent motion token prediction and real robot control.
- Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection71
The proposed detector, named MimDet, enables a MIM pre-trained vanilla ViT to outperform leading hierarchical architectures such as Swin Transformer, MViTv2 and ConvNeXt on COCO object detection & instance segmentation, and achieves better results compared with the previous best adapted vanilla ViT detector using a more modest fine-tuning recipe while converging 2.8× faster.
- 71
- Masked Image Modeling with Denoising Contrast70
This work introduces a pure MIM method, ConMIM, to produce simple intra-image inter-patch contrastive constraints as the sole learning objectives for masked patch prediction and strengthens the denoising mechanism with asymmetric designs, including image perturbations and model progress rates, to improve the network pre-training.
- 70
- Meta-Adapter: An Online Few-shot Learner for Vision-Language Model65
The Meta-Adapter, a lightweight residual-style adapter, is proposed, to refine the CLIP features guided by the few-shot samples in an online manner, and obtains notable performance improvements in open-vocabulary object detection and segmentation tasks.
- TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation64
TokLIP is introduced, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens.
- Structured Domain Adaptation With Online Relation Regularization for Unsupervised Person Re-ID63
This work argues that domain translation has great potential on exploiting valuable source-domain data but the existing methods did not provide proper regularization on the translation process and proposes an end-to-end structured domain adaptation framework with an online relation-consistency regularization term.
- Towards A Better Metric for Text-to-Video Generation61
This paper investigates the limitations inherent in existing metrics and introduces a novel evaluation pipeline, the Text-to-Video Score (T2VScore), which integrates two pivotal criteria: Text-Video Alignment, which scrutinizes the fidelity of the video in representing the given text description, and Video Quality, which evaluates the video's overall production caliber with a mixture of experts.
- Scalable Image Tokenization with Index Backpropagation Quantization52
Index Backpropagation Quantization (IBQ) is proposed, a new VQ method for the joint optimization of all codebook embeddings and the visual encoder that enables scalable training of visual tokenizers and achieves a large-scale codebook with high dimension and high utilization.
- What Makes for Good Visual Tokenizers for Large Language Models?51
A new MLLM equipped with a tailored Good Visual Tokenizer (GVT), which exhibits strong visual comprehension capability at multiple scales, without introducing extra parameters and task-specific fine-tuning.
- $π$-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation50
Predict-Interpolate Tuning aggregates the parameters of lightweight task-specific experts learned from similar tasks to aid the target downstream task, and surpasses fine-tuning and other parameter-efficient transfer learning methods both in full-shot and low-shot regimes.
- 50
- TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs49
TimeLens is presented, a systematic investigation into building MLLMs with strong VTG ability, along two primary dimensions: data quality and algorithmic design, which culminate in a family of MLLMs with state-of-the-art VTG performance among open-source models and even surpass proprietary models.
- BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning49
Branching Temporal Adapter (BT-Adapter), a novel method for ex-tending image-language pretrained models into the video domain, is proposed, able to empower existing multimodal dialogue models with strong video understanding capabilities without incur-ring excessive GPU costs.
- EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios46
It is found that MLLMs’ planning performance on EgoPlan-Bench2 is critically dependent on temporally structured action sequences in historical task progress and interactions between objects and humans in current observation state, which underscores the necessity for strong reasoning abilities to integrate diverse multimodal cues and analysis before making final decision.
- VIT-LENS: Towards Omni-modal Representations46
Vit-lens provides a unified solution for representation learning of increasing modalities with two appealing advantages: unlocking the great potential of pretrained- ViTs to novel modalities effectively with efficient parameters and data regime and enabling emergent down- stream capabilities through modality alignment and shared ViT parameters.
- EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning46
A benchmark with human annotations, EgoPlan-Bench, is introduced to quantitatively investigate the potential of MLLMs as embodied task planners in real-world scenarios and demonstrates that the model tuned on EgoPlan-IT not only significantly improves performance on this benchmark, but also effectively acts as embodied planner in simulations.
- BoxSnake: Polygonal Instance Segmentation with Box Supervision39
This work proposes a new end-to-end training technique, termed BoxSnake, to achieve effective polygonal instance segmentation using only box annotations for the first time, and shows significant superiority on the Cityscapes dataset.
- Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis37
It is found that improving the reconstruction fidelity of VQ tokenizers does not necessarily improve the generation, instead, learning to compress semantic features within VQ tokenizers significantly improves generative transformers' ability to capture textures and structures.
- TagGPT: Large Language Models are Zero-shot Multimodal Taggers34
This work proposes TagGPT, a fully automated system capable of tag extraction and multimodal tagging in a completely zero-shot fashion, and provides a system-level solution based on a modular framework equipped with a pre-trained LLM and a sentence embedding model.
- 33
- 32
- 31
- Video-Text Pre-training with Learned Regions for Retrieval30
This work proposes a simple yet effective module for video-text representation learning, namely RegionLearner, which can take into account the structure of objects during pre-training on large-scale video- text pairs and is much more computationally efficient.
- 29
- 29
- Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R127
SEED-Bench-R1 is introduced, a benchmark designed to systematically evaluate post-training methods for MLLMs in video understanding and identifies key limitations such as inconsistent reasoning and overlooked visual cues, and suggest future improvements in base model reasoning, reward modeling, and RL robustness against noisy signals.
- 27
- 25
- 25
- Vision-Language Instruction Tuning: A Review and Analysis24
For the first time, a detailed multi-perspective categorization for existing VLIT datasets is offered and the characteristics that high-quality VLit data should possess are identified by incorporating these characteristics as guiding principles into the existingVLIT data construction process.
- GrootVL: Tree Topology is All You Need in State Space Model23
The GrootVL network is proposed, which first dynamically generates a tree topology based on spatial relationships and input features, and then feature propagation is performed based on this graph, thereby breaking the original sequence constraints to achieve stronger representation capabilities.
- MambaTree: Tree Topology is All You Need in State Space Model22
The MambaTree network is proposed, which first dynamically generates a tree topology based on spatial relationships and input features, and then feature propagation is performed based on this graph, thereby breaking the original sequence constraints to achieve stronger representation capabilities.
- PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance21
This model, Prompt-guided Pooling LLaVA (PPLLaVA), introduces three key components: a CLIP-based visual-prompt alignment module that identifies regions of interest based on user instructions, a prompt-guided pooling mechanism that adaptively compresses the visual sequence using convolution-style pooling, and a clip context extension module tailored for processing long and complex prompts in visual dialogues.
- DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA20
DIAL, a framework bridging high-level decision making and low-level motor execution through a differentiable latent intent bottleneck, is introduced, establishing a new state-of-the-art, achieving superior performance with 10x fewer demonstrations than prior methods.
- GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers19
This work empirically found that visually perfect generations are not always optimal for representation enhancement, and proposes a two-stage training strategy to prioritize learning useful visual knowledge, namely GenHancer, which consistently outperforms prior arts on the MMVP-VLM benchmark.
- Supervised Fine-tuning in turn Improves Visual Foundation Models18
A two-stage method ViSFT (Vision SFT) is proposed to unleash the fine-grained knowledge of vision foundation models and shows improvements across various out-of-domain benchmarks including vision and vision-linguistic scenarios.
- TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale17
This work proposes a degradation-free pre-training strategy to retain the generalization ability of the text encoder via freezing shallow layers while enabling the task-related semantics capturing in tunable deep layers, and produces a series of models, dubbed TVTSv2, with up to one billion parameters.
- UniT: Toward a Unified Physical Language for Human-to-Humanoid Policy Learning and World Modeling16
Unified Latent Action Tokenizer via Visual Anchoring is introduced, a framework that establishes a unified physical language for human-to-humanoid transfer and offers a scalable path to distill vast human knowledge into general-purpose humanoid capabilities.
- Exploring Model Transferability through the Lens of Potential Energy16
This work reframe the challenge of model selection through the lens of potential energy and directly model the interaction forces that influence fine-tuning dynamics, which can acquire an enhanced and more stable observation for estimating transferability.
- Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities15
This work proposes a methodology named Multimodal Pathway - given a target modality and a transformer designed for it, an auxiliary transformer trained with data of another modality is used and constructed to connect components of the two models so that data of the target modality can be processed by both models.
- Sticker820K: Empowering Interactive Retrieval with Stickers15
The StickerCLIP is proposed as a benchmark model on the Sticker820K dataset, demonstrating strong superiority over the CLIP for the text-to-image retrieval task, and the recently popularized LLM is extended by means of prompt tuning, integrating its ability for sticker retrieval and allowing users to retrieve stickers through instructions.
- 15
- HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation14
This paper explores an efficient training paradigm to build a single transformer to build a single transformer for unified multimodal understanding and generation, and proposes a multimodal warmup strategy utilizing prior knowledge to extend capabilities.
- RILS: Masked Visual Reconstruction in Language Semantic Space14
A novel masked visual Reconstruction In Language semantic Space (RILS) pre-training framework, in which sentence representations serve as prototypes to transform the vision-only signals into patch-sentence probabilities as semantically meaningful MIM reconstruction targets.
- Accelerating Vision-Language Pretraining with Free Language Modeling14
This work proposes a new pretraining task, namely, free language modeling (FLM), that enables a 100% prediction rate with arbitrary corruption rates and shows FLM could achieve an impressive 2.5 × pretraining time reduction in comparison to the MLM-based methods, while keeping competitive performance on both vision-language understanding and generation tasks.
- Binary Embedding-based Retrieval at Tencent14
A binary embedding-based retrieval engine equipped with a recurrent binarization algorithm that enables customized bits per dimension and successfully employed the introduced BEBR to web search and copyright detection of Tencent products, including Sogou, Tencent Video, QQ World, etc.
- 14
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding13
This work proposes a new early-fusion LMM that can fuse multi-modal inputs in the early stage and respond to visual instructions in an auto-regressive manner and devise an efficient training recipe for the proposed model, which harnesses the prior knowledge of the pre-trained models, addressing both the performance limitations and the challenge of resource consumption.
- Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval13
This work proposes a new model upgrade paradigm, termed Bidirectional Compatible Training (BiCT), which will upgrade the old gallery embeddings by forward-compatible training towards the embedding space of the backward-compatible new model.
- Making Foresight Actionable: Repurposing Representation Alignment in World Action Models11
AGRA is proposed, an Action-Grounded Representation Alignment objective that regularizes the world-action interface by aligning intermediate video diffusion features with spatially coherent semantic representations from a foundation visual encoder and consistently improves both in-distribution performance and out-of-distribution generalization over the baseline world action model.
- Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation11
This work introduces Divot, a Diffusion-Powered Video Tokenizer, which leverages the diffusion process for self-supervised video representation learning, and presents Divot-LLM, a diffusion-based video tokenizer that achieves competitive performance across various video comprehension and generation benchmarks.
- AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction10
This work proposes AnimeGamer, which is built upon Multimodal Large Language Models (MLLMs) to generate each game state, including dynamic animation shots that depict character movements and updates to character states, as illus-trated in Figure 1.
- TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter10
A parameter-efficient and task-agnostic adapter, dubbed TaCA, that facilitates compatibility across distinct foundation models while ensuring enhanced performance for the new models and allows downstream applications to seamlessly integrate better-performing foundation models without necessitating retraining.
- LoRA-Gen: Specializing Large Language Model via Online LoRA Generation9
The LoRA-Gen framework is proposed, which utilizes a large cloud-side model to generate LoRA parameters for edge-side models based on task descriptions and merges the LoRA parameters into the edge-side model to achieve flexible specialization.
- AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation9
Experimental results show that the model trained on AnimeShooter achieves superior cross-shot visual consistency and adherence to reference visual guidance, which highlight the value of the dataset for coherent animated video generation.
- Learning Transferable Spatiotemporal Representations from Natural Script Knowledge9
A new pre-text task, Turning to Video for Transcript Sorting (TVTS), which sorts shuffled ASR scripts by attending to learned video representations, and demonstrates strong out-of-the-box spatiotemporal representations on diverse benchmarks, e.g., +13.6% gains over VideoMAE on SSV2 via linear probing.
- 9
- Modeling Uncertain Feature Representation for Domain Generalization8
This paper improves the network generalization ability by modeling domain shifts with uncertainty (DSU), i.e., characterizing the feature statistics as uncertain distributions during training and hypothesizing that the feature statistic, after considering the potential uncertainties, follows a multivariate Gaussian distribution.
- Cached Transformers: Improving Transformers with Differentiable Memory Cache7
This work introduces a new Transformer model that uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens, and surpasses previous memory-based techniques in tasks such as language modeling and displays the ability to be applied to a broader range of situations.
- Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval6
This work revitalizes region features of sparsely sampled video clips to significantly reduce both spatial and temporal visual redundancy towards democratizing VLP research at the same time achieving state-of-the-art results.
- 6
- 6
- DiCoDe: Diffusion-Compressed Deep Tokens for Autoregressive Video Generation with Language Models5
DiCoDe is a novel approach that leverages Diffusion-Compressed Deep Tokens to generate videos with a language model in an autoregressive manner, and is scalable using readily available AR architectures, and is capable of generating videos ranging from a few seconds to one minute using only 4 A100 GPUs for training.
- 3
- 3
- 3
- Aligning Latent Spaces with Flow Priors2
This paper presents a novel framework for aligning learnable latent spaces to arbitrary target distributions by leveraging flow-based generative models as priors, and formally proves that minimizing this alignment loss establishes a computationally tractable surrogate objective for maximizing a variational lower bound on the log-likelihood of latents under the target distribution.
- 2
- 2
- 1
- ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning1
ROVE introduces a human-in-the-loop pipeline capable of collecting deployment and intervention data for humanoid manipulation, and utilizes Optimistic Value Estimation (OVE) to prioritize high-value behaviors from mixed-quality trajectories to robustify value estimation.
- 1
- –
- –
- Equivariant Filter for High Performance Image Tracking using an Event Camera–
An equivariant filter design for high performance tracking of planar image transformations using an event camera that incorporates an equivalent-measurement update step that de-correlates the (highly temporally correlated) feature-position measurements provided by the AEB tracker.
- –
- –
- –
Publication data from OpenAlex; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-12. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.