Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Suggested from co-authorship
- Ping LuoSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works47 from public data
- SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension1,018
This work addresses the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment ofGenerative models, by introducing a benchmark named SEED-Bench, which spans 12 evaluation dimensions including the comprehension of both the image and video modality.
- DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images446
A strong baseline is proposed, called Match R- CNN, which builds upon Mask R-CNN to solve the above four tasks in an end-to-end manner to address issues of DeepFashion2.
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation365
This work presents a unified and versatile foundation model, namely, SEED-X, which is able to model multi-granularity visual semantics for comprehension and generation tasks and demonstrates its effectiveness in handling real-world applications across various domains after instruction tuning.
- 327
- Parser-Free Virtual Try-on via Distilling Appearance Flows267
This work proposes a novel approach, "teacher-tutor-student" knowledge distillation, which is able to produce highly photo-realistic images without human parsing, possessing several appealing advantages compared to prior arts.
- All in One: Exploring Unified Video-Language Pre-Training258
This work introduces an end-to-end VLP model, namely all-in-one Transformer, that embeds raw video and textual signals into joint representations using a unified backbone architecture and introduces a novel and effective token rolling operation to encode temporal representations from video clips in a non-parametric manner.
- Making LLaMA SEE and Draw with SEED Tokenizer227
SEED is introduced, an elaborate image tokenizer that empowers LLMs with the ability to SEE and Draw at the same time, and with SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction.
- SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension156
SEED-Bench-2-Plus is introduced, a benchmark specifically designed for evaluating text-rich visual comprehension of Multimodal Large Language Models and a thorough evaluation involving 34 prominent MLLMs is conducted, to emphasize the current limitations of MLLMs in text-rich visual comprehension.
- Planting a SEED of Vision in Large Language Model150
This study identifies two crucial principles for the architecture and training of SEED that effectively ease subsequent alignment with LLMs, and emphasizes the great potential of discrete visual tokens in versatile multimodal LLMs and the importance of proper image tokenizers in broader research.
- Bridging Video-text Retrieval with Multiple Choice Questions131
This work enables fine-grained video-text interactions while maintaining high efficiency for retrieval via a novel pretext task, dubbed as Multiple Choice Questions (MCQ), where a parametric module BridgeFormer is trained to answer the “questions” constructed by the text features via resorting to the video features.
- Disentangled Cycle Consistency for Highly-realistic Virtual Try-On130
The proposed Disentangled Cycle-consistency Try-On Network (DCTON) is able to produce highly-realistic try-on images by disentangling important components of virtual try- on including clothes warping, skin synthesis, and image composition.
- SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing106
A pretrained Multimodal Large Language Model (MLLM) that unifies comprehension and generation with SEED-Data-Edit is fine-tuneed, indicating the potential and effectiveness of SEED-Data-Edit in advancing the field of instructional image editing.
- SEED-Bench-2: Benchmarking Multimodal Large Language Models103
This work categorizes the capabilities of MLLMs into hierarchical levels from $L_0 to $L-4 based on the modalities they can accept and generate, and proposes SEED-Bench-2, a comprehensive benchmark that evaluates the hierarchical capabilities ofMLLMs.
- SCAN: Self-and-Collaborative Attention Network for Video Person Re-Identification92
A novel and practical deep architecture for video person re-identification termed self-and-collaborative attention network (SCAN), which adopts the video pairs as the input and outputs their matching scores and can be predicted by the binary classifier.
- Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos84
This work introduces Moto, which converts video content into latent Motion Token sequences by a Latent Motion Tokenizer, learning a bridging “language” of motion from videos in an unsupervised manner, and implements a co-fine-tuning strategy that seamlessly bridges latent motion token prediction and real robot control.
- EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning79
EgoPlan-Bench is introduced, a comprehensive benchmark to evaluate the planning abilities of MLLMs in real-world scenarios from an egocentric perspective, mirroring human perception, and EgoPlan-IT, a specialized instruction-tuning dataset that effectively enhances model performance on EgoPlan-Bench.
- TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation64
TokLIP is introduced, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens.
- 50
- TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs49
TimeLens is presented, a systematic investigation into building MLLMs with strong VTG ability, along two primary dimensions: data quality and algorithmic design, which culminate in a family of MLLMs with state-of-the-art VTG performance among open-source models and even surpass proprietary models.
- EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios46
It is found that MLLMs’ planning performance on EgoPlan-Bench2 is critically dependent on temporally structured action sequences in historical task progress and interactions between objects and humans in current observation state, which underscores the necessity for strong reasoning abilities to integrate diverse multimodal cues and analysis before making final decision.
- VIT-LENS: Towards Omni-modal Representations46
Vit-lens provides a unified solution for representation learning of increasing modalities with two appealing advantages: unlocking the great potential of pretrained- ViTs to novel modalities effectively with efficient parameters and data regime and enabling emergent down- stream capabilities through modality alignment and shared ViT parameters.
- 33
- Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R127
SEED-Bench-R1 is introduced, a benchmark designed to systematically evaluate post-training methods for MLLMs in video understanding and identifies key limitations such as inconsistent reasoning and overlooked visual cues, and suggest future improvements in base model reasoning, reward modeling, and RL robustness against noisy signals.
- 22
- 21
- DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLA20
DIAL, a framework bridging high-level decision making and low-level motor execution through a differentiable latent intent bottleneck, is introduced, establishing a new state-of-the-art, achieving superior performance with 10x fewer demonstrations than prior methods.
- GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancers19
This work empirically found that visually perfect generations are not always optimal for representation enhancement, and proposes a two-stage training strategy to prioritize learning useful visual knowledge, namely GenHancer, which consistently outperforms prior arts on the MMVP-VLM benchmark.
- Supervised Fine-tuning in turn Improves Visual Foundation Models18
A two-stage method ViSFT (Vision SFT) is proposed to unleash the fine-grained knowledge of vision foundation models and shows improvements across various out-of-domain benchmarks including vision and vision-linguistic scenarios.
- UniT: Toward a Unified Physical Language for Human-to-Humanoid Policy Learning and World Modeling16
Unified Latent Action Tokenizer via Visual Anchoring is introduced, a framework that establishes a unified physical language for human-to-humanoid transfer and offers a scalable path to distill vast human knowledge into general-purpose humanoid capabilities.
- Making Foresight Actionable: Repurposing Representation Alignment in World Action Models11
AGRA is proposed, an Action-Grounded Representation Alignment objective that regularizes the world-action interface by aligning intermediate video diffusion features with spatially coherent semantic representations from a foundation visual encoder and consistently improves both in-distribution performance and out-of-distribution generalization over the baseline world action model.
- Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation11
This work introduces Divot, a Diffusion-Powered Video Tokenizer, which leverages the diffusion process for self-supervised video representation learning, and presents Divot-LLM, a diffusion-based video tokenizer that achieves competitive performance across various video comprehension and generation benchmarks.
- 11
- Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models10
An automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR, and a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs, which condenses question semantics and extracts question-relevant knowledge from pretrained 4D reconstruction priors into a compact set of geometry tokens.
- AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction10
This work proposes AnimeGamer, which is built upon Multimodal Large Language Models (MLLMs) to generate each game state, including dynamic animation shots that depict character movements and updates to character states, as illus-trated in Figure 1.
- Align, Adapt and Inject: Sound-guided Unified Image Generation10
This paper proposes a unified framework 'Align, Adapt, and Inject' (AAI), able to extract the dynamic information of varied sounds, while utilizing the formidable capability of existing T2I models to facilitate sound-guided image generation, editing, and stylization in a convenient and cost-effective manner.
- AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation9
Experimental results show that the model trained on AnimeShooter achieves superior cross-shot visual consistency and adherence to reference visual guidance, which highlight the value of the dataset for coherent animated video generation.
- From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model9
ReDiff is introduced, a refining-enhanced diffusion framework that teaches the model to identify and correct its own errors, enabling stable and efficient parallel generation far superior to traditional denoising methods.
- Learning Transferable Spatiotemporal Representations from Natural Script Knowledge9
A new pre-text task, Turning to Video for Transcript Sorting (TVTS), which sorts shuffled ASR scripts by attending to learned video representations, and demonstrates strong out-of-the-box spatiotemporal representations on diverse benchmarks, e.g., +13.6% gains over VideoMAE on SSV2 via linear probing.
- 8
- DiCoDe: Diffusion-Compressed Deep Tokens for Autoregressive Video Generation with Language Models5
DiCoDe is a novel approach that leverages Diffusion-Compressed Deep Tokens to generate videos with a language model in an autoregressive manner, and is scalable using readily available AR architectures, and is capable of generating videos ranging from a few seconds to one minute using only 4 A100 GPUs for training.
- 4
- Aligning Latent Spaces with Flow Priors2
This paper presents a novel framework for aligning learnable latent spaces to arbitrary target distributions by leveraging flow-based generative models as priors, and formally proves that minimizing this alignment loss establishes a computationally tractable surrogate objective for maximizing a variational lower bound on the log-likelihood of latents under the target distribution.
- 2
- 1
- ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning1
ROVE introduces a human-in-the-loop pipeline capable of collecting deployment and intervention data for humanoid manipulation, and utilizes Optimistic Value Estimation (OVE) to prioritize high-value behaviors from mixed-quality trajectories to robustify value estimation.
- –
- –
Publication data from OpenAlex; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-13. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.