Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Chi‐Man PunSuggested from co-authorship
- Hao GaoSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works93 from public data
- Uformer: A General U-Shaped Transformer for Image Restoration2,405
Uformer, an effective and efficient Transformer-based architecture for image restoration, in which a hierarchical encoder-decoder network is built using the Transformer block and a learnable multi-scale restoration modulator in the form of a multi- scale spatial bias to adjust features in multiple layers of the Uformer decoder is proposed.
- Generating Human Motion from Textual Descriptions with Discrete Representations798
A simple and must-known conditional generative framework based on Vector Quantised-Variational AutoEncoder (VQ-VAE) and Generative Pre-trained Transformer for human motion generation from textural descriptions shows better performance than competitive approaches, including recent diffusion-based approaches.
- VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models710
This work explores the training scheme of video models extended from Stable Diffusion and investigates the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model, resulting in a generic high-quality video model.
- VideoCrafter1: Open Diffusion Models for High-Quality Video Generation633
This model is the first open-source I2V foundation model capable of transforming a given image into a video clip while maintaining content preservation constraints, and can generate realistic and cinematic-quality videos with a resolution of $1024 \times 576.
- FateZero: Fusing Attentions for Zero-shot Text-based Video Editing594
F FateZero, a zero-shot text-based editing method on real-world videos without per-prompt training or use-specific mask, is proposed, which is the first one to show the ability of zero-shot text-driven video style and local attribute editing from the trained text-to-image model.
- SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation542
SadTalker is presented, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation and designs PoseVAE via a conditional VAE to synthesize head motion in different styles.
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models416
This work argues that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities, and proposes a novel framework and pipeline for exhaustively evaluating the performance of the generated videos.
- DEIM: DETR with Improved Matching for Fast Convergence394
The Matchability-Aware Loss (MAL), a novel loss function that optimizes matches across various quality levels, enhancing the effectiveness of Dense O2O, is proposed.
- Towards Ghost-Free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN295
A shadow matting generative adversarial network (SMGAN) is designed to synthesize realistic shadow mattings from a given shadow mask and shadow-free image to outperforms other state-of-the-art methods by a large margin.
- CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior286
This paper proposes to cast speech-driven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty.
- Explicit Visual Prompting for Low-Level Structure Segmentations249
This work takes inspiration from the widely-used pre-training and then prompt tuning protocols in NLP and proposes a new visual prompting model, named Explicit Visual Prompting (EVP), which significantly outperforms other parameter-efficient tuning protocols under the same amount of tunable parameters.
- DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos243
Comprehensive evaluations reveal that DepthCrafter achieves state-of-the-art performance in open-world video depth estimation under zero-shot settings and facilitates various downstream applications, including depth-based visual effects and conditional video generation.
- SmartEdit: Exploring Complex Instruction-Based Image Editing with Multimodal Large Language Models213
A Bidirectional Interaction Module (BIM) is proposed that enables comprehensive bidirectional information interactions between the input image and the MLLM output and demonstrates that a small amount of complex instruction editing data can effectively stimulate SmartEdit’ s editing capabilities for more complex instructions.
- 188
- Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance166
This paper explores customized video generation by utilizing text as context description and motion structure (e.g., frame-wise depth) as concrete guidance as concrete guidance in joint-conditional video generation using a Latent Diffusion Model.
- 165
- Improving the Harmony of the Composite Image by Spatial-Separated Attention Module165
This work proposes a novel attention module named Spatial-Separated Attention Module (S2AM) and designs a novel image harmonization framework by inserting the S2AM in the coarser low-level features of the Unet structure by two different ways.
- ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models150
This work investigates the capability of generating images from pre-trained diffusion models at much higher resolutions than the training image sizes and proposes the dispersed convolution and noise-damped classifier-free guidance, which can enable ultra-high-resolution image generation.
- 124
- Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation106
This work develops a framework comprised of two functional modules, which provides video candidates with desired scene or motion context described by query texts, and generates plot-aligned videos under the guidance of motion structure and text prompts.
- Interactive Story Visualization with Multiple Characters84
A system for generic interactive story visualization, capable of handling multiple novel characters and supporting the editing of layout and local structure is proposed, developed by leveraging the prior knowledge of large language and T2I models, trained on massive corpora.
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation83
DiTCtrl is proposed, a training-free multi-prompt video generation method under MM-DiT architectures for the first time that achieves state-of-the-art performance without additional training and presents MPVBench, a new benchmark specially designed for multi-prompt video generation to evaluate the performance of multi-prompt generation.
- High-Resolution Document Shadow Removal via A Large-Scale Real-World Dataset and A Frequency-Aware Shadow Erasing Net74
This work handles high-resolution document shadow removal directly via a larger-scale real-world dataset and a carefully-designed frequency-aware network, which shows a clearly better performance than previous methods in terms of visual quality and numerical results.
- DPE: Disentanglement of Pose and Expression for General Video Portrait Editing69
This paper introduces a novel self-supervised disentanglement framework to decouple pose and expression without 3DMMs and paired data, which consists of a motion editing module, a pose generator, and an expression generator and proposes a bidirectional cyclic training strategy with well-designed constraints.
- Split then Refine: Stacked Attention-guided ResUNets for Blind Single Image Visible Watermark Removal69
A novel two-stage framework with a stacked attention-guided ResUNets to simulate the process of detection, removal and refinement of digital watermark, which outperforms other state-of-the-art methods by a large margin.
- MagicStick: Controllable Video Editing via Control Handle Transformations66
This method is the first method to show the ability of video property editing from the pre-trained text-to-image model, and compares with shape-aware text-based editing and handcrafted motion video generation, demonstrating the superior temporal consistency and editing capability than previous works.
- 64
- 3D GAN Inversion with Facial Symmetry Prior63
This work designs a pipeline and constraints to make full use of the pseudo auxiliary view obtained via image flipping, which helps obtain a view-consistent and well-structured geometry shape during the inversion process, and introduces facial symmetry prior.
- Towards A Better Metric for Text-to-Video Generation61
This paper investigates the limitations inherent in existing metrics and introduces a novel evaluation pipeline, the Text-to-Video Score (T2VScore), which integrates two pivotal criteria: Text-Video Alignment, which scrutinizes the fidelity of the video in representing the given text description, and Video Quality, which evaluates the video's overall production caliber with a mixture of experts.
- LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation59
This work introduces LivelySpeaker, a framework that realizes semantics-aware co-speech gesture generation and offers several control handles, and devise a simple but effective diffusion-based gesture generation backbone simply using pure MLPs, that is conditioned on only audio signals and learns to gesticulate with realistic motions.
- Depth-Aware Test-Time Training for Zero-Shot Video Object Segmentation49
This work introduces a test-time training (TTT) strategy to address the problem of zero-shot Video Object Segmentation, and suggests that the momentum-based weight initialization and looping-based training scheme lead to more stable improvements.
- Spatial-Separated Curve Rendering Network for Efficient and High-Resolution Image Harmonization49
A novel spatial-separated curve rendering network for efficient and high-resolution image harmonization for the first time and reduces more than 90% parameters compared with previous methods but still achieves the state-of-the-art performance on both synthesized iHarmony4 and real-world DIH test sets.
- Make-Your-Anchor: A Diffusion-based 2D Avatar Generation Framework39
Make-Your-Anchor is proposed, a novel system necessitating only a one-minute video clip of an individual for training, subsequently enabling the automatic generation of anchor-style videos with precise torso and hand movements, outperforming SOTA diffusion/non-diffusion methods.
- StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos38
A novel framework for converting 2D videos to immersive stereoscopic 3D, addressing the growing demand for 3D content in immersive experience, using pre-trained stable video diffusion as the backbone and introducing a fine-tuning protocol for the stereo video inpainting task.
- 37
- Defocus Blur Detection via Depth Distillation37
This work considers the depth information as the approximate soft label of DBD and proposes a joint learning framework inspired by knowledge distillation that learns the defocus blur from ground truth and the depth distilled from a well-trained depth estimation network at the same time.
- Shadocnet: Learning Spatial-Aware Tokens in Transformer for Document Shadow Removal35
A Transformer-based model for document shadow removal is proposed that utilizes shadow context encoding and decoding in both shadow and shadow-free regions and is competitive with state-of-the-art methods.
- Image Splicing Localization via Semi-global Network and Fully Connected Conditional Random Fields28
This work addresses the problem of image splicing localization by exploiting the observation that the spliced region should not only highly relate to local features, but also global features from the whole image, and builds a Semi-Global Network.
- X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model24
X-Adapter, a universal upgrader to enable the pretrained plug-and-play modules to work directly with the upgraded text-to-image diffusion model without further retraining, is introduced and the results show that X-Adapter may facilitate wider application in the upgraded foundational diffusion model.
- AnimateZero: Video Diffusion Models are Zero-Shot Image Animators21
This work proposes AnimateZero to unveil the pre-trained text-to-video diffusion model, i.e., AnimateDiff, and provide more precise appearance and motion control abilities for it, and demonstrates the effectiveness of the proposed method in both T2V and related applications.
- 20
- Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach19
By addressing fundamental shortcomings in existing VDMs, FVDM sets a new paradigm in video synthesis, offering a robust framework with significant implications for generative modeling and multimedia applications.
- VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models18
A novel framework that leverages VDM's inherent force to enable high-quality zero-shot customized video generation and devise an innovative bidirectional interaction between subject features and generated content through spatial self-attention within VDM, ensuring that VDM has better subject fidelity while maintaining the diversity of the generated video.
- ToonTalker: Cross-Domain Face Reenactment18
This paper proposes a transformer-based framework to align the motions from different domains into a common latent space where motion transfer is conducted via latent code addition and proposes a novel cross-domain reenactment without paired data.
- AB-Cache: Training-Free Acceleration of Diffusion Models via Adams-Bashforth Cached Feature Reuse17
This work provides a theoretical understanding by analyzing the denoising process through the second-order Adams-Bashforth method, revealing a linear relationship between the outputs of consecutive steps, and proposes a novel caching-based acceleration approach for diffusion models.
- ForgeryTTT: Zero-Shot Image Manipulation Localization with Test-Time Training17
ForgeryTTT is introduced, the first method leveraging test-time training (TTT) to identify manipulated regions in images and achieves a 20.1% improvement in localization accuracy compared to other zero-shot methods and a 4.3% improvement over non-zero-shot techniques.
- 17
- A shell dataset, for shell features extraction and recognition16
A shell dataset, containing 7894 shell species with 29622 samples, where totally 59244 shell images for shell features extraction and recognition are used and it is hoped more researchers can present new methods to extract shell features and develop new classifiers based on this dataset, in order to improve the recognition performance of shell species.
- 15
- 14
- 12
- CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training10
This work first analyzes the prompt influence in the current video diffusion model and finds the LoRAs are only needed for the specific layers for appearance and motion customization, and proposes CustomTTT, where they can joint custom the appearance and the motion of the given video easily.
- Learning Enriched Illuminants for Cross and Single Sensor Color Constancy10
A cross-sensor self-supervised training to train the network that outperform other state-of-the-art methods on cross and single sensor evaluations, respectively, with only 16% parameters of the previous best model.
- MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence9
MLLM-4D is introduced, a comprehensive framework designed to bridge the gaps in training data curation and model post-training for spatiotemporal understanding and reasoning for multimodal large language models (MLLMs).
- Applying stochastic second-order entropy images to multi-modal image registration8
The shortcomings of structural image representation are overcome by devising a new similarity metric called Stochastic Second-Order Entropy Image (SSOEI), and the results showed the advantages of SSOEI over the state-of-art methods.
- CutClaw: Agentic Hours-Long Video Editing via Music Synchronization7
CutClaw is introduced, an autonomous multi-agent framework designed to edit hours-long raw footage into meaningful short videos that leverages the capabilities of multiple Multimodal Language Models~(MLLMs) as an agent system.
- 7
- Sketch Video Synthesis7
By transforming a video into SVG lines through the sketching process, this method unlocks applications in sketch-based video editing and video doodling, enabled through video composition, as exemplified in the teaser.
- 7
- 6
- 6
- MV-Performer: Taming Video Diffusion Model for Faithful and Synchronized Multi-view Performer Synthesis5
This paper proposes a multi-view human-centric video diffusion model that fuses information from the reference video, partial rendering, and different viewpoints, and provides a robust inference procedure for in-the-wild video cases, which greatly mitigates the artifacts induced by imperfect monocular depth estimation.
- ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation5
A training-free video interpolation method for generative video diffusion models, which is generalizable to different models in a plug-and-play manner and transform a video model into a self-cascaded video diffusion model with incorporating the designed hidden state correction modules.
- 4DVD: Cascaded Dense-view Video Diffusion Model for High-quality 4D Content Generation4
4DVD is presented, a cascaded video diffusion model that generates 4D content in a decoupled manner that decouples this into two subtasks: coarse multi-view layout generation and structure-aware conditional generation, and effectively unifies them.
- 4
- LightCtrl: Training-free Controllable Video Relighting3
LightCtrl is presented, the first controllable video relighting method that enables explicit control of video illumination through a user-supplied light trajectory in a training-free manner and shows improved controllability over baseline methods.
- EmoCAST: Emotional Talking Portrait via Emotive Text Description3
EmoCAST, a diffusion-based talking head framework for precise, text-driven emotional synthesis, achieves state-of-the-art performance in generating realistic, emotionally expressive, and audio-synchronized talking-head videos.
- BlobCtrl: Taming Controllable Blob for Element-level Image Editing3
This work presents BlobCtrl, a framework for element-level image editing based on a probabilistic blob-based representation that disentangles layout from appearance, affording fine-grained, controllable object-level elements manipulation.
- 3
- 2
- 2
- EasyOmnimatte: Taming Pretrained Inpainting Diffusion Models for End-to-End Video Layered Decomposition2
EasyOmnimatte, the first unified, end-to-end video omnimatte method, is introduced, which finetune a pretrained video inpainting diffusion model to learn dual complementary experts while keeping its original weights intact.
- 2
- Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation1
Seg-Agent is proposed, a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning and leverages Set-of-Mark visual prompting to render candidate regions directly onto the image, allowing Seg-Agent to achieve performance comparable to state-of-the-art training-based methods without any parameter updates.
- Text-to-Motion Generation With Discrete Representations and Large Language Models1
This work investigates a simple yet effective conditional generative framework for text-to-motion generation and proposes T2M-GIT+, which employs a non-autoregressive method to generate discrete motion representations in parallel, and is therefore more efficient than T2M-GPT+ while achieving comparable results.
- 1
- –
- –
- Decoupling Vocal and Rhythmic Conditioning for Music-Driven Singing Avatar Animation–
This work proposes MusicAvatar, a diffusion-based framework that directly synthesizes 2D singing avatars without relying on 3D priors, and designs a dual-stream music attention module that decouples the roles of singing voice and BGM.
- Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm–
This work introduces V2V-Zero, a training-free framework that exposes this interface in existing vision-language model (VLM) conditioned generators by replacing text-only conditioning with final-layer hidden states extracted from visual pages, exploiting the fact that the frozen VLM already maps both text and images into the generator's conditioning space.
- RainbowDreamer: Taming Semantic Controls for Attribute-Consistent Text-to-3D Generation–
RainbowDreamer is presented, a three-stage framework that builds up the semantic controls to generate attribute-consistent 3D Gaussian Splattings (3DGS) and achieves state-of-the-art performance in both quantitative and qualitative evaluations.
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-12. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.