Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Zhengyou ZhangSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works348 from public data
- Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data2,160
This work extends the powerful ESRGAN to a practical restoration application (namely, Real-ESRGAN), which is trained with pure synthetic data and employs a U-Net discriminator with spectral normalization to increase discriminator capability and stabilize the training dynamics.
- Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation1,235
This work proposes a new T2V generation setting—One-Shot Video Tuning, where only one text-video pair is presented, and introduces Tune-A-Video, which involves a tailored spatio-temporal attention mechanism and an efficient one-shot tuning strategy.
- YOLO-World: Real-Time Open-Vocabulary Object Detection1,134
YOLO-World is introduced, an innovative approach that enhances YOLO with open-vocabulary detection capabilities through vision-language modeling and pre-training on large-scale datasets and proposes a new Re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN) and region-text contrastive loss to facilitate the interaction between visual and linguistic information.
- SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension1,018
This work addresses the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment ofGenerative models, by introducing a benchmark named SEED-Bench, which spans 12 evaluation dimensions including the comprehension of both the image and video modality.
- MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing902
MasaCtrl converts existing self-attention in diffusion models into mutual self-attention, so that it can query correlated local contents and textures from source images for consistency, and proposes a mask-guided mutual self-attention strategy.
- 881
- VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models710
This work explores the training scheme of video models extended from Stable Diffusion and investigates the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model, resulting in a generic high-quality video model.
- Towards Real-World Blind Face Restoration with Generative Facial Prior676
This work proposes GFP-GAN that leverages rich and diverse priors encapsulated in a pretrained face GAN for blind face restoration that achieves superior performance to prior art on both synthetic and real-world datasets.
- MotionCtrl: A Unified and Flexible Motion Controller for Video Generation673
MotionCtrl is presented, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion and is a relatively generalizable model that can adapt to a wide array of camera poses and trajectories once trained.
- VideoCrafter1: Open Diffusion Models for High-Quality Video Generation633
This model is the first open-source I2V foundation model capable of transforming a given image into a video clip while maintaining content preservation constraints, and can generate realistic and cinematic-quality videos with a resolution of $1024 \times 576.
- FateZero: Fusing Attentions for Zero-shot Text-based Video Editing594
F FateZero, a zero-shot text-based editing method on real-world videos without per-prompt training or use-specific mask, is proposed, which is the first one to show the ability of zero-shot text-driven video style and local attribute editing from the trained text-to-image model.
- InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models569
InstantMesh is presented, a feed-forward framework for instant 3D mesh generation from a single image, featuring state-of-the-art generation quality and significant training scalability, and is able to create diverse 3D assets within 10 seconds.
- SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation542
SadTalker is presented, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation and designs PoseVAE via a conditional VAE to synthesize head motion in different styles.
- Deep Crossing460
The Deep Crossing model is proposed which is a deep neural network that automatically combines features to produce superior models and was able to build, from scratch, two web-scale models for a major paid search engine, and achieve superior results with only a sub-set of the features used in the production models.
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models416
This work argues that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities, and proposes a novel framework and pipeline for exhaustively evaluating the performance of the generated videos.
- PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding410
PhotoMaker is introduced, an efficient personalized text-to-image generation method, which mainly encodes an arbitrary number of input ID images into a stack ID embed-ding for preserving ID information, and an ID-oriented data construction pipeline to assemble the training data.
- Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos382
A novel two-stage training scheme that can utilize easily obtained datasets and the pre-trained text-to-image (T2I) model to obtain the pose-controllable character videos and successfully generates continuously pose-controllable character videos while keeps the editing and concept composition ability of the pre-trained T2I model.
- LayoutDiffusion: Controllable Diffusion Model for Layout-to-Image Generation369
A diffusion model named LayoutDiffusion is proposed that can obtain higher generation quality and greater controllability than the previous works and is proposed to construct a structural image patch with region information and transform the patched image into a special layout to fuse with the normal layout in a unified form.
- SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation365
This work presents a unified and versatile foundation model, namely, SEED-X, which is able to model multi-granularity visual semantics for comprehension and generation tasks and demonstrates its effectiveness in handling real-world applications across various domains after instruction tuning.
- UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition365
Large kernels are the key to unlocking the exceptional performance of ConvNets in domains where they were originally not proficient, and the proposed model achieves state- of-the-art performance on time-series forecasting and audio recognition tasks even without modality-specific customization to the architecture.
- Instances as Queries362
For the first time, it is demonstrated that a simple end-to-end query based framework can achieve the state-of-the-art performance in various instance-level recognition tasks.
- GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction340
The GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools, generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts using the Low-Rank Adaptation (LoRA) optimization.
- Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models334
This paper proposes a new framework called Mix-of-Show, which adopts an embedding-decomposed LoRA (ED-LoRA) for single-client tuning and gradient fusion for the center node to preserve the in-domain essence of single concepts and support theoretically limitless concept fusion.
- 327
- GS-IR: 3D Gaussian Splatting for Inverse Rendering300
The GS-IR is a novel inverse rendering approach based on 3D Gaussian Splatting that leverages forward mapping volume rendering to achieve photorealistic novel view synthesis and relighting results and demonstrates the superiority of the method over baseline methods through qualitative and quantitative evaluations of various challenging scenes.
- UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection275
This is the first scheme to integrate multi-modal (visual-audio) learning for either joint optimization or the individual moment retrieval task, and tackles moment retrieval as a keypoint detection problem using a novel query generator and query decoder.
- Latent Video Diffusion Models for High-Fidelity Long Video Generation260
This work introduces lightweight video diffusion models that synthesize high-fidelity and arbitrary-long videos from pure noise that outperforms previous methods on 3D pixel space when under a limited computational budget and introduces conditional latent perturbation to reduce performance degradation during generating long-duration videos.
- Expressive expression mapping with ratio images260
This paper captures the illumination change of one person's expression in what they call an expression ratio image (ERI), and together with geometric warping, they map an ERI to any other person's face image to generate more expressive facial expressions.
- All in One: Exploring Unified Video-Language Pre-Training258
This work introduces an end-to-end VLP model, namely all-in-one Transformer, that embeds raw video and textual signals into joint representations using a unified backbone architecture and introduces a novel and effective token rolling operation to encode temporal representations from video clips in a non-parametric manner.
- DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos243
Comprehensive evaluations reveal that DepthCrafter achieves state-of-the-art performance in open-world video depth estimation under zero-shot settings and facilitates various downstream applications, including depth-based visual effects and conditional video generation.
- DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models241
A novel image editing method, DragonDiffusion, enabling Drag-style manipulation on Diffusion models, that can transform the editing signals into gradients via feature correspondence loss to modify the intermediate representation of the diffusion model.
- Taming Rectified Flow for Inversion and Editing227
This work proposes RF-Solver, a novel training-free sampler that effectively enhances inversion precision by mitigating the errors in the ODE-solving process of rectified flow, and proposes RF-Edit, a general feature-sharing-based framework for image and video editing.
- Making LLaMA SEE and Draw with SEED Tokenizer227
SEED is introduced, an elaborate image tokenizer that empowers LLMs with the ability to SEE and Draw at the same time, and with SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction.
- VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolution216
An automatic and scalable pipeline to collect a high-quality video face dataset (VFHQ), which contains over 16, 000 high-fidelity clips of diverse interview scenarios, and shows that the temporal information plays a pivotal role in eliminating video consistency issues as well as further improving visual performance.
- SmartEdit: Exploring Complex Instruction-Based Image Editing with Multimodal Large Language Models213
A Bidirectional Interaction Module (BIM) is proposed that enables comprehensive bidirectional information interactions between the input image and the MLLM output and demonstrates that a small amount of complex instruction editing data can effectively stimulate SmartEdit’ s editing capabilities for more complex instructions.
- Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models209
This paper makes the first attempt to introduce explicit 3D shape priors into the CLIP-guided 3D optimization process, and proposes to jointly optimize a learnable text prompt and fine-tune the text-to-image diffusion model for rendering-style image generation.
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling205
This work proposes FreeNoise, a tuning-free and time-efficient paradigm to enhance the generative capabilities of pretrained video diffusion models while preserving content consistency, and designs a novel motion injection method to support the generation of videos conditioned on multiple text prompts.
- DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models199
A dual-observation-space design that involves the joint optimization of a canonical space and a posed space that are related by a learnable deformation field is introduced, which facilitates the generation of more complete textures and geometry faithful to the target pose.
- 181
- Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance166
This paper explores customized video generation by utilizing text as context description and motion structure (e.g., frame-wise depth) as concrete guidance as concrete guidance in joint-conditional video generation using a Latent Diffusion Model.
- MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions164
This work proposes MiraData, a high-quality video dataset that surpasses previous ones in video duration, caption detail, motion strength, and visual quality, and introduces MiraBench, which enhances existing benchmarks by adding 3D consistency and tracking-based motion strength metrics.
- 158
- SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension156
SEED-Bench-2-Plus is introduced, a benchmark specifically designed for evaluating text-rich visual comprehension of Multimodal Large Language Models and a thorough evaluation involving 34 prominent MLLMs is conducted, to emphasize the current limitations of MLLMs in text-rich visual comprehension.
- DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes151
DeVRF, a novel representation to accelerate learning dynamic radiance fields via utilizing the 3D volumetric canonical space learnt from multi-view static images to ease the learning of 4D voxel deformation field with only few-view dynamic sequences is presented.
- ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models150
This work investigates the capability of generating images from pre-trained diffusion models at much higher resolutions than the training image sizes and proposes the dispersed convolution and noise-damped classifier-free guidance, which can enable ultra-high-resolution image generation.
- Planting a SEED of Vision in Large Language Model150
This study identifies two crucial principles for the architecture and training of SEED that effectively ease subsequent alignment with LLMs, and emphasizes the great potential of discrete visual tokens in versatile multimodal LLMs and the importance of proper image tokenizers in broader research.
- Visual panel: virtual mouse, keyboard and 3D controller with an ordinary piece of paper140
By detecting the clicking and dragging actions, the system can fulfill many tasks such as controlling a remote large display, and simulating a physical keyboard, as well as providing 3D information, serving as a virtual joystick, to control 3D virtual objects.
- 135
- Towards Vivid and Diverse Image Colorization with Generative Color Prior135
This work aims at recovering vivid colors by leveraging the rich and diverse color priors encapsulated in a pretrained Generative Adversarial Networks (GAN) via a GAN encoder and incorporating these features into the colorization process with feature modulations.
- Fast Video Object Segmentation Using the Global Context Module132
A real-time, high-quality semi-supervised video object segmentation algorithm that effectively summarizes and propagates information through the entire video, and uses constant memory regardless of the video length and costs substantially less memory and computation.
- Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning131
The experiments demonstrate that the proposed MU-LLaMA model, trained on the designed MusicQA dataset, achieves outstanding performance in both music question answering and music caption generation across various metrics, outperforming current state-of-the-art (SOTA) models in both fields and offering a promising advancement in the T2M-Gen research field.
- Bridging Video-text Retrieval with Multiple Choice Questions131
This work enables fine-grained video-text interactions while maintaining high efficiency for retrieval via a novel pretext task, dubbed as Multiple Choice Questions (MCQ), where a parametric module BridgeFormer is trained to answer the “questions” constructed by the text features via resorting to the video features.
- Crossover Learning for Fast Online Video Instance Segmentation129
A novel crossover learning scheme that uses the instance feature in the current frame to pixel-wisely localize the same instance in other frames to enable efficient cross-frame instance-to-pixel relation learning and brings cost-free improvement during inference.
- 128
- HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting127
An efficient yet effective framework that generates high-quality 3D humans with fine-grained geometry and realistic appearance, based on the insight that 3D Gaussian Splatting is an efficient renderer with peri-odic Gaussian shrinkage or growing, where such adaptive density control can be naturally guided by intrinsic human structures.
- TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts121
TIP-Editor is proposed, a 3D scene editing framework that accepts both text and image prompts and a 3D bounding box to specify the editing region, and employs a stepwise 2D personalization strategy to better learn the representation of the existing scene and the reference image.
- Open-book Video Captioning with Retrieve-Copy-Generate Network118
A novel Retrieve-Copy-Generate network is proposed, where a pluggable video-to-text retriever is constructed to retrieve sentences as hints from the training corpus effectively, and a copy-mechanism generator is introduced to extract expressions from multi-retrieved sentences dynamically.
- Cross-modal Consensus Network for Weakly Supervised Temporal Action Localization115
This work argues that the features extracted from the pre-trained extractors are not specific for WS-TAL task, leading to inevitable redundancy and sub-optimization, and proposes a cross-modal consensus network (CO2-Net) to tackle this problem.
- DiffEditor: Boosting Accuracy and Flexibility on Diffusion-Based Image Editing114
In this paper, DiffEditor is proposed to rectify two weaknesses in existing diffusion-based image editing: in complex scenarios, editing results often lack editing accuracy and exhibit unexpected artifacts; and lack of flexibility to harmonize editing operations, e.g., imagine new content.
- DreamDiffusion: Generating High-Quality Images from Brain EEG Signals111
Quantitative and qualitative results demonstrate the effectiveness of the proposed DreamDiffusion method as a significant step towards portable and low-cost ``thoughts-to-image'', with potential applications in neuroscience and computer vision.
- 107
- 107
- SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing106
A pretrained Multimodal Large Language Model (MLLM) that unifies comprehension and generation with SEED-Data-Edit is fine-tuneed, indicating the potential and effectiveness of SEED-Data-Edit in advancing the field of instructional image editing.
- Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation106
This work develops a framework comprised of two functional modules, which provides video candidates with desired scene or motion context described by query texts, and generates plot-aligned videos under the guidance of motion structure and text prompts.
- SEED-Bench-2: Benchmarking Multimodal Large Language Models103
This work categorizes the capabilities of MLLMs into hierarchical levels from $L_0 to $L-4 based on the modalities they can accept and generate, and proposes SEED-Bench-2, a comprehensive benchmark that evaluates the hierarchical capabilities ofMLLMs.
- Distilling Audio-Visual Knowledge by Compositional Contrastive Learning103
The main idea is to learn a compositional embedding that closes the cross-modal semantic gap and captures the task-relevant semantics, which facilitates pulling together representations across modalities by compositional contrastive learning.
- 100
- Advances in 3D Generation: A Survey99
The 3D representations that serve as the backbone for 3D generation are introduced and a comprehensive overview of the rapidly growing literature on generation methods is provided, categorized by the type of algorithmic paradigms.
- VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control98
This work proposes VideoPainter, an efficient dual-branch framework featuring a lightweight context encoder that processes masked videos and injects background guidance into any pre-trained video diffusion transformer, generalizing across arbitrary mask types, enhancing background integration and foreground generation, and enabling user-customized control.
- Object-aware Video-language Pre-training for Retrieval92
Object-aware Transformers is presented, an object-centric approach that extends video-language transformer to incorporate object representations to leverage the bounding boxes and object tags to guide the training process.
- Temporally Efficient Vision Transformer for Video Instance Segmentation89
The proposed Temporally Efficient Vision Transformer (TeViT) is nearly convolution-free, which contains a transformer backbone and a query-based video instance segmentation head, which fully utilizes both frame-level and instance-level temporal context information and obtains strong temporal modeling capacity with negligible extra computational cost.
- Robust and Rapid Generation of Animated Faces from Video Images: A Model-Based Modeling Approach87
An easy-to-use and cost-effective system to construct textured 3D animated face models from videos with minimal user interaction, which makes full use of generic knowledge of faces in head motion determination, head tracking, model fitting, and multiple-view bundle adjustment.
- ToonCrafter: Generative Cartoon Interpolation86
A novel approach that transcends traditional correspondence-based cartoon video interpolation, paving the way for generative interpolation, and designs a toon rectification learning strategy that seamlessly adapts live-action video priors to the cartoon domain, resolving the domain gap and content leakage issues.
- 85
- CV-VAE: A Compatible Video VAE for Latent Generative Video Models85
A method for training a video VAE of latent video models, namely CV-VAE, whose latent space is compatible with that of a given image VAE, e.g., image VAE of Stable Diffusion (SD).
- Interactive Story Visualization with Multiple Characters84
A system for generic interactive story visualization, capable of handling multiple novel characters and supporting the editing of layout and local structure is proposed, developed by leveraging the prior knowledge of large language and T2I models, trained on massive corpora.
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation83
DiTCtrl is proposed, a training-free multi-prompt video generation method under MM-DiT architectures for the first time that achieves state-of-the-art performance without additional training and presents MPVBench, a new benchmark specially designed for multi-prompt video generation to evaluate the performance of multi-prompt generation.
- Inserting Anybody in Diffusion Models via Celeb Basis83
Empowered by the proposed celeb basis, the new identity in the customized model showcases a better concept combination ability than previous personalization methods and can also learn several new identities at once and interact with each other where the previous customization model fails to.
- RecDCL: Dual Contrastive Learning for Recommendation82
This work investigates how to employ both batch-wise CL (BCL) and feature-wise CL (FCL) for recommendation and proposes a dual contrastive learning recommendation framework---RecDCL, which can consistently outperform the state-of theart GNNs-based and SSL-based models.
- Improved Test-Time Adaptation for Domain Generalization81
This work proposes an Improved Test-Time Adaptation (ITTA) method, and shows that the proposed two strategies are beneficial for the learned model, and ITTA could achieve superior performance to the current state-of-the-art methods on several DG benchmarks.
- 80
- EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning79
EgoPlan-Bench is introduced, a comprehensive benchmark to evaluate the planning abilities of MLLMs in real-world scenarios from an egocentric perspective, mirroring human perception, and EgoPlan-IT, a specialized instruction-tuning dataset that effectively enhances model performance on EgoPlan-Bench.
- Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields75
L2G-NeRF is proposed, a Local-to-Global registration method for bundle-adjusting Neural Radiance Fields that outperforms the current state-of-the-art in terms of high-fidelity reconstruction and resolving large camera pose misalignment.
- 72
- Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection71
The proposed detector, named MimDet, enables a MIM pre-trained vanilla ViT to outperform leading hierarchical architectures such as Swin Transformer, MViTv2 and ConvNeXt on COCO object detection & instance segmentation, and achieves better results compared with the previous best adapted vanilla ViT detector using a more modest fine-tuning recipe while converging 2.8× faster.
- Masked Image Modeling with Denoising Contrast70
This work introduces a pure MIM method, ConMIM, to produce simple intra-image inter-patch contrastive constraints as the sole learning objectives for masked patch prediction and strengthens the denoising mechanism with asymmetric designs, including image perturbations and model progress rates, to improve the network pre-training.
- 70
- DPE: Disentanglement of Pose and Expression for General Video Portrait Editing69
This paper introduces a novel self-supervised disentanglement framework to decouple pose and expression without 3DMMs and paired data, which consists of a motion editing module, a pose generator, and an expression generator and proposes a bidirectional cyclic training strategy with well-designed constraints.
- Skinned Motion Retargeting with Residual Perception of Motion Semantics & Geometry68
This work proposes a novel Residual RETargeting network (R2ET) structure, which relies on two neural modification modules, to adjust the source motions to fit the target skeletons and shapes progressively, and presents a balancing gate to conduct linear interpolation between them.
- 68
- DeSRA: Detect and Delete the Artifacts of GAN-based Real-World Super-Resolution Models67
Equipped with the DeSRA, the cause and characteristics of the GAN artifacts produced in unseen test data without ground-truths are analyzed and artifacts from inference are eliminated and the ability of SR models to be applied in real-world scenarios is improved.
- NOFA: NeRF-based One-shot Facial Avatar Reconstruction67
This work proposes a one-shot 3D facial avatar reconstruction framework that only requires a single source image to reconstruct a high-fidelity 3d facial avatar, and develops an efficient encoder-decoder network to reconstruct the canonical neural volume of the source image, and proposes a compensation network to complement facial details.
- Meta-Adapter: An Online Few-shot Learner for Vision-Language Model65
The Meta-Adapter, a lightweight residual-style adapter, is proposed, to refine the CLIP features guided by the few-shot samples in an online manner, and obtains notable performance improvements in open-vocabulary object detection and segmentation tasks.
- 65
- TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation64
TokLIP is introduced, a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens and incorporating CLIP-level semantics while enabling end-to-end multimodal autoregressive training with standard VQ tokens.
- 3D GAN Inversion with Facial Symmetry Prior63
This work designs a pipeline and constraints to make full use of the pseudo auxiliary view obtained via image flipping, which helps obtain a view-consistent and well-structured geometry shape during the inversion process, and introduces facial symmetry prior.
- A Progressive Scheme for Stereo Matching62
A progressive scheme for stereo matching which uses the disparity gradient limit principle and the least commitment strategy to select only the most reliable matches and postpone unreliable decisions until enough confidence is accumulated is proposed.
- Towards A Better Metric for Text-to-Video Generation61
This paper investigates the limitations inherent in existing metrics and introduces a novel evaluation pipeline, the Text-to-Video Score (T2VScore), which integrates two pivotal criteria: Text-Video Alignment, which scrutinizes the fidelity of the video in representing the given text description, and Video Quality, which evaluates the video's overall production caliber with a mixture of experts.
- MM-RealSR: Metric Learning based Interactive Modulation for Real-World Super-Resolution61
A metric learning strategy to map the unquantifiable degradation levels in real-world scenarios to a metric space, which is trained in an unsupervised manner and introduces an anchor point strategy in the metric learning process to normalize the distribution of metric space.
- HRDFuse: Monocular 360° Depth Estimation by Collaboratively Learning Holistic-with-Regional Depth Distributions60
A novel framework, HRDFuse, is proposed that subtly combines the potential of convolutional neural networks (CNNs) and transformers by collaboratively learning the holistic contextual information from the ERP and the regional structuralInformation from the TP to obtain the final depth map.
- 58
- RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization57
A new re-parameterizable block tailored for SR, namely RepSR, which consists of a clean residual path and two expand-and-squeeze convolution paths with the modified BN and can achieve a better trade-off between performance and actual running time than previous SR methods.
- 54
- Deep Embedding Forest53
A two-step Deep Embedding Forest algorithm is demonstrated to achieve on-par or slightly better performance as compared with the DNN counterpart, with only a fraction of serving time on conventional hardware.
- 53
- Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos52
This work introduces Moto, which converts video content into latent Motion Token sequences by a Latent Motion Tokenizer, learning a bridging"language" of motion from videos in an unsupervised manner, and implements a co-fine-tuning strategy that seamlessly bridges latent motion token prediction and real robot control.
- StyleAdapter: A Unified Stylized Image Generation Model52
The StyleAdapter is proposed, a unified stylized image generation model capable of producing a variety of stylized images that match both the content of a given prompt and the style of reference images, without the need for per-style fine-tuning.
- FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction51
This work introduces FreeSplatter, a scalable feed-forward framework that generates high-quality 3D Gaussians from uncalibrated sparse-view images while estimating camera parameters within seconds, and develops two specialized variants-for object-centric and scene-level reconstruction-trained on comprehensive datasets.
- What Makes for Good Visual Tokenizers for Large Language Models?51
A new MLLM equipped with a tailored Good Visual Tokenizer (GVT), which exhibits strong visual comprehension capability at multiple scales, without introducing extra parameters and task-specific fine-tuning.
- $π$-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation50
Predict-Interpolate Tuning aggregates the parameters of lightweight task-specific experts learned from similar tasks to aid the target downstream task, and surpasses fine-tuning and other parameter-efficient transfer learning methods both in full-shot and low-shot regimes.
- StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter50
StyleCrafter is introduced, a generic method that enhances pre-trained T2V models with a style control adapter, enabling video generation in any style by providing a reference image, and designs a scale-adaptive fusion module to balance the influences of text-based content features and image-based style features.
- 50
- BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning49
Branching Temporal Adapter (BT-Adapter), a novel method for ex-tending image-language pretrained models into the video domain, is proposed, able to empower existing multimodal dialogue models with strong video understanding capabilities without incur-ring excessive GPU costs.
- OSRT: Omnidirectional Image Super-Resolution with Distortion-aware Transformer48
This paper designs a distortion-aware Transformer (OSRT) to modulate ERP distortions continuously and self-adaptively, and proposes Fisheye downsampling, which mimics the real-world imaging process and synthesizes more realistic low-resolution samples.
- 47
- 47
- Sub-pixel location of edges with non-uniform blurring: a finite closed-form approach47
Experiments show that the accuracy of the proposed method is comparable to the more computationally demanding nonlinear optimization method, and the integral nature of low order spatial moments provide the algorithm some resilience against additive white noise.
- EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios46
It is found that MLLMs’ planning performance on EgoPlan-Bench2 is critically dependent on temporally structured action sequences in historical task progress and interactions between objects and humans in current observation state, which underscores the necessity for strong reasoning abilities to integrate diverse multimodal cues and analysis before making final decision.
- PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM46
This research introduces a unified framework for automated graphic layout generation, leveraging the multi-modal large language model (MLLM) to accommodate diverse design tasks and develops an automated text-to-poster system that generates editable SVG posters based on users' design intentions, bridging the gap between layout generation and real-world graphic design applications.
- VIT-LENS: Towards Omni-modal Representations46
Vit-lens provides a unified solution for representation learning of increasing modalities with two appealing advantages: unlocking the great potential of pretrained- ViTs to novel modalities effectively with efficient parameters and data regime and enabling emergent down- stream capabilities through modality alignment and shared ViT parameters.
- EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning46
A benchmark with human annotations, EgoPlan-Bench, is introduced to quantitatively investigate the potential of MLLMs as embodied task planners in real-world scenarios and demonstrates that the model tuned on EgoPlan-IT not only significantly improves performance on this benchmark, but also effectively acts as embodied planner in simulations.
- AnimeSR: Learning Real-World Super-Resolution Models for Animation Videos46
The proposed method, AnimeSR, is capable of restoring real-world low-quality animation videos effectively and efficiently, achieving superior performance to previous state-of-the-art methods.
- High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors45
This work proposes an efficient method to construct the personalized generative prior based on a small set of facial images of a given individual that allows for photo-realistic rendering with novel views, and the face reenactment can be realized by performing navigation in the latent space of 3D-GAN.
- Semantic-Guided Relation Propagation Network for Few-shot Action Recognition44
A novel semantic-guided relation propagation network (SRPN), which leverages semantic information together with visual information for few-shot action recognition and outperforms several state-of-the-art methods with a significant margin.
- SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes43
This work introduces SurfelNeRF, a variant of neural radiance field which employs a flexible and scalable neural surfel representation to store geometric attributes and extracted appearance features from input images, and extends the conventional surfel-based fusion scheme to progressively integrate incoming input frames into the reconstructed global neural scene representation.
- Geometrycrafter: Consistent Geometry Estimation for Open-World Videos With Diffusion Priors42
A novel point map Variational Autoencoder (VAE) is proposed for encoding and decoding unbounded point maps, which can recover high-fidelity point map sequences with temporal coherence from open-world videos, facilitating accurate 3D/4D reconstruction, camera parameter estimation, and other depth-based applications.
- 41
- 40
- StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos38
A novel framework for converting 2D videos to immersive stereoscopic 3D, addressing the growing demand for 3D content in immersive experience, using pre-trained stable video diffusion as the backbone and introducing a fine-tuning protocol for the stereo video inpainting task.
- 38
- HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video38
HOSNeRF is introduced, a novel 360° free-viewpoint rendering method that reconstructs neural radiance fields for dynamic human-object-scene from a single monocular in-the-wild video that significantly outperforms SOTA approaches on two challenging datasets by a large margin.
- 37
- Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis37
It is found that improving the reconstruction fidelity of VQ tokenizers does not necessarily improve the generation, instead, learning to compress semantic features within VQ tokenizers significantly improves generative transformers' ability to capture textures and structures.
- 37
- Mani-GS: Gaussian Splatting Manipulation with Triangular Mesh36
By utilizing a triangle shape-aware Gaussian binding and adapting method, this work can achieve 3DGS manipulation and preserve high-fidelity rendering and is also effective with inaccurate meshes extracted from 3DGS.
- VMesh: Hybrid Volume-Mesh Representation for Efficient View Synthesis36
VMesh is presented, which depicts an object with a textured mesh along with an auxiliary sparse volume, which retains the advantages of mesh-based assets, such as efficient rendering and compact storage, while also incorporating the ability to represent subtle geometric structures provided by the volumetric counterpart.
- Dual Semantic Fusion Network for Video Object Detection35
This work proposes a dual semantic fusion network (abbreviated as DSFNet) to fully exploit both frame-level and instance-level semantics in a unified fusion framework without external guidance and introduces a geometric similarity measure into the fusion process to alleviate the influence of information distortion caused by noise.
- 35
- CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models34
The CustomNet is introduced, a novel object customization approach that explicitly incorporates 3D novel view synthesis capabilities into the object customization process, and facilitates zero-shot object customization without test-time optimization, offering simultaneous control over the viewpoints, location, and background.
- Anti-Aliased Neural Implicit Surfaces with Encoding Level of Detail34
This work introduces a multi-scale tri-plane-based scene representation that is capable of capturing the LoD of the signed distance function (SDF) and the space radiance and proposes an error-guided sampling strategy to guide the growth of the SDF during the optimization.
- TagGPT: Large Language Models are Zero-shot Multimodal Taggers34
This work proposes TagGPT, a fully automated system capable of tag extraction and multimodal tagging in a completely zero-shot fashion, and provides a system-level solution based on a modular framework equipped with a pre-trained LLM and a sentence embedding model.
- 34
- 33
- 33
- VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control32
VerseCrafter is introduced, a geometry-driven video world model that generates dynamic, realistic videos from a unified 4D geometric world state that achieves superior visual quality and more accurate control over camera and multi-object motion than prior methods.
- FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios32
RefAdapter is introduced, a lightweight image-conditioned adapter that excels in spatial adaptation and consistency preservation, surpassing existing methods in balancing appearance consistency and structural flexibility.
- 32
- HumanRef: Single Image to 3D Human Generation via Reference-Guided Diffusion31
To ensure the generated 3D model is photorealistic and consistent with the input image, HumanRef introduces a novel method called reference-guided score distillation sampling (Ref-SDS), which effectively incorporates image guidance into the generation process.
- 31
- NeRF-Texture: Texture Synthesis with Neural Radiance Fields31
This work proposes a novel texture synthesis method with Neural Radiance Fields (NeRF) to capture and synthesize textures from given multi-view images to enhance matching performance and regularize the distribution of latent features by incorporating a clustering constraint.
- 31
- 29
- Programmable Motion Generation for Open-Set Motion Control Tasks29
This work introduces a new paradigm, programmable motion generation, in which any given motion control task is broken down into a combination of atomic constraints and programmed into an error function that quantifies the degree to which a motion sequence adheres to them.
- ACNet: Approaching-and-Centralizing Network for Zero-Shot Sketch-Based Image Retrieval29
An Approaching-and-Centralizing Network (termed “ACNet”) is proposed to jointly optimize sketch-to-photo synthesis and image retrieval and achieves state-of-the-art performance on two widely used ZS-SBIR datasets and surpasses previous methods by a large margin.
- 29
- Feature Augmented Memory with Global Attention Network for VideoQA29
A novel videoQA framework which progressively refines the representations of videos and questions from fine to coarse grain in a sequence-sensitive manner and achieves state-of-the-art performance on VideoQA benchmark datasets.
- DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing28
This work proposes the image-based video-NeRF editing pipeline with a set of innovative designs, including multi-view multi-pose Score Distillation Sampling (SDS) from both the 2D personalized diffusion prior and 3D diffusion prior, reconstruction losses, text-guided local parts super-resolution, and style transfer.
- PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas28
PanoGRF, Generalizable Spherical Radiance Fields for Wide-baseline Panoramas, which construct spherical radiance fields incorporating $360^{\circ}$ scene priors incorporating spherical depth estimation to improve the geometry features of some regions of the panorama.
- Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R127
SEED-Bench-R1 is introduced, a benchmark designed to systematically evaluate post-training methods for MLLMs in video understanding and identifies key limitations such as inconsistent reasoning and overlooked visual cues, and suggest future improvements in base model reasoning, reward modeling, and RL robustness against noisy signals.
- 27
- MuMu-LLaMA: Multi-modal music understanding and generation via large language models26
This work introduces a novel dataset containing 167.69 hours of multi-modal data, including text, images, videos, and music annotations, tailored for multi-modal music understanding and generation, annotated using advanced visual models like LLaVA and Video-LLaVA, and proposes a multi-modal music understanding and generation model named MuMu-LLaMA.
- 25
- 25
- Do We Really Need Temporal Convolutions in Action Segmentation?24
This paper presents a pure Transformer-based model without temporal convolutions in action segmentation, called Temporal U-Transformer, and proposes a boundary-aware loss based on the distribution of similarity scores between frames from attention modules to improve the ability to recognize boundaries.
- 24
- 24
- TapMo: Shape-aware Motion Generation of Skeleton-free Characters24
The results reveal that TapMo consistently outperforms existing auto-animation methods, delivering superior-quality animations for both seen or unseen heterogeneous 3D characters.
- Vision-Language Instruction Tuning: A Review and Analysis24
For the first time, a detailed multi-perspective categorization for existing VLIT datasets is offered and the characteristics that high-quality VLit data should possess are identified by incorporating these characteristics as guiding principles into the existingVLIT data construction process.
- 24
- GrootVL: Tree Topology is All You Need in State Space Model23
The GrootVL network is proposed, which first dynamically generates a tree topology based on spatial relationships and input features, and then feature propagation is performed based on this graph, thereby breaking the original sequence constraints to achieve stronger representation capabilities.
- MonoNeuralFusion: Online Monocular Neural 3D Reconstruction with Geometric Priors23
This paper presents MonoNeuralFusion to perform the online neural 3D reconstruction from monocular videos, by which the 3D scene geometry is efficiently generated and optimized during the on-the-fly 3D monocular scanning.
- 22
- MambaTree: Tree Topology is All You Need in State Space Model22
The MambaTree network is proposed, which first dynamically generates a tree topology based on spatial relationships and input features, and then feature propagation is performed based on this graph, thereby breaking the original sequence constraints to achieve stronger representation capabilities.
- 21
- 21
- Neural Point-based Volumetric Avatar: Surface-guided Neural Points for Efficient and Photorealistic Volumetric Head Avatar21
This paper proposes Neural Point-based Volumetric Avatar (NPVA), a method that adopts the neural point representation as well as the neural volume rendering process and discards the predefined connectivity and hard correspondence imposed by mesh-based approaches.
- AnimateZero: Video Diffusion Models are Zero-Shot Image Animators21
This work proposes AnimateZero to unveil the pre-trained text-to-video diffusion model, i.e., AnimateDiff, and provide more precise appearance and motion control abilities for it, and demonstrates the effectiveness of the proposed method in both T2V and related applications.
- ConTex-Human: Free-View Rendering of Human from a Single Image with Texture-Consistent Synthesis19
A texture-consistent back view synthesis module that could transfer the reference im-age content to the back view through depth and text-guided attention injection and a visibility-aware patch consistency regularization for texture mapping and refinement combined with the synthesized back view texture are introduced.
- 19
- PC-Dance: Posture-controllable Music-driven Dance Synthesis19
A powerful framework named PC-Dance is proposed to perform adaptive posture-controllable music-driven dance synthesis and an efficient scheme for adaptive motion graph construction (AMGC) is introduced, which could improve the efficiency of graph-based optimization and preserve the diversity of motions.
- VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models18
A novel framework that leverages VDM's inherent force to enable high-quality zero-shot customized video generation and devise an innovative bidirectional interaction between subject features and generated content through spatial self-attention within VDM, ensuring that VDM has better subject fidelity while maintaining the diversity of the generated video.
- Supervised Fine-tuning in turn Improves Visual Foundation Models18
A two-stage method ViSFT (Vision SFT) is proposed to unleash the fine-grained knowledge of vision foundation models and shows improvements across various out-of-domain benchmarks including vision and vision-linguistic scenarios.
- Efficient Scale-Space Spatiotemporal Saliency Tracking for Distortion-Free Video Retargeting18
This paper proposes a novel approach to distortion-free video retargeting by scale-space spatiotemporal saliency tracking, and develops an efficient coarse-to-fine solution that combines exhaustive coarse and gradient-based fine search.
- HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback17
Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses, effectively addressing the issue of a decrease in satisfaction rate after basic RLAIF.
- 17
- Neural Concatenative Singing Voice Conversion: Rethinking Concatenation-Based Approach for One-Shot Singing Voice Conversion17
Experimental results demonstrate that the proposed NeuCoSVC system outperforms the disentanglement-based speaker embedding approach in one-shot SVC across intra-language, cross-language, and cross-domain evaluations.
- Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video17
A decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos.
- 17
- TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale17
This work proposes a degradation-free pre-training strategy to retain the generalization ability of the text encoder via freezing shallow layers while enabling the task-related semantics capturing in tunable deep layers, and produces a series of models, dubbed TVTSv2, with up to one billion parameters.
- M$^{2}$UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models17
The experimental results demonstrate that the proposedLLaMA model achieves or surpasses the performance of the current state-of-the-art models.
- Recurrent Binary Embedding for GPU-Enabled Exhaustive Retrieval from Billion-Scale Semantic Vectors17
A Recurrent Binary Embedding model that learns compact representations for real-time retrieval based on a particular class of semantic models is proposed, and generalizing to other types is straightforward, as exemplified by two different models at the end of the paper.
- Pixal3D: Pixel-Aligned 3D Generation from Images16
Pixal3D for the first time demonstrates 3D-native pixel-aligned generation at scale, and provides a new inspiring way towards high-fidelity 3D generation of object or scene from single or multi-view images.
- 16
- 16
- 16
- Exploring Model Transferability through the Lens of Potential Energy16
This work reframe the challenge of model selection through the lens of potential energy and directly model the interaction forces that influence fine-tuning dynamics, which can acquire an enhanced and more stable observation for estimating transferability.
- InstructP2P: Learning to Edit 3D Point Clouds with Text Instructions16
This paper presents InstructP2P, an end-to-end framework for 3D shape editing on point clouds, guided by high-level textual instructions, that extends the capabilities of existing methods by synergizing the strengths of a text-conditioned point cloud diffusion model, Point-E, and powerful language models, enabling color and geometry editing using language instructions.
- New Measurements and Corner-Guidance for Curve Matching with Probabilistic Relaxation16
A new algorithm for curve matching (including lines) within a probabilistic relaxation framework, and an approach of incorporating previously matched points/corners to guide curve matching are described.
- 15
- 15
- Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities15
This work proposes a methodology named Multimodal Pathway - given a target modality and a transformer designed for it, an auxiliary transformer trained with data of another modality is used and constructed to connect components of the two models so that data of the target modality can be processed by both models.
- 15
- Sticker820K: Empowering Interactive Retrieval with Stickers15
The StickerCLIP is proposed as a benchmark model on the Sticker820K dataset, demonstrating strong superiority over the CLIP for the text-to-image retrieval task, and the recently popularized LLM is extended by means of prompt tuning, integrating its ability for sticker retrieval and allowing users to retrieve stickers through instructions.
- Guide3D: Create 3D Avatars from Text and Image Guidance15
This work introduces Guide3D, a zero-shot text-and-image-guided generative model for 3D avatar generation based on diffusion models, which outperforms the current state-of-the-art in producing topologically and structurally correct geometry and high-resolution textures.
- Discovering class specific composite features through discriminative sampling with Swendsen-Wang Cut15
- HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation14
This paper explores an efficient training paradigm to build a single transformer to build a single transformer for unified multimodal understanding and generation, and proposes a multimodal warmup strategy utilizing prior knowledge to extend capabilities.
- RILS: Masked Visual Reconstruction in Language Semantic Space14
A novel masked visual Reconstruction In Language semantic Space (RILS) pre-training framework, in which sentence representations serve as prototypes to transform the vision-only signals into patch-sentence probabilities as semantically meaningful MIM reconstruction targets.
- Accelerating Vision-Language Pretraining with Free Language Modeling14
This work proposes a new pretraining task, namely, free language modeling (FLM), that enables a 100% prediction rate with arbitrary corruption rates and shows FLM could achieve an impressive 2.5 × pretraining time reduction in comparison to the MLM-based methods, while keeping competitive performance on both vision-language understanding and generation tasks.
- Binary Embedding-based Retrieval at Tencent14
A binary embedding-based retrieval engine equipped with a recurrent binarization algorithm that enables customized bits per dimension and successfully employed the introduced BEBR to web search and copyright detection of Tencent products, including Sogou, Tencent Video, QQ World, etc.
- 14
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding13
This work proposes a new early-fusion LMM that can fuse multi-modal inputs in the early stage and respond to visual instructions in an auto-regressive manner and devise an efficient training recipe for the proposed model, which harnesses the prior knowledge of the pre-trained models, addressing both the performance limitations and the challenge of resource consumption.
- Assembler: Scalable 3D Part Assembly via Anchor Point Diffusion13
This work presents Assembler, a scalable and generalizable framework for 3D part assembly that reconstructs complete objects from input part meshes and a reference image and introduces an interesting part-aware 3D modeling system that generates high-resolution, editable objects from images, demonstrating potential for interactive and compositional design.
- 13
- DOGR: Towards Versatile Visual Document Grounding and Referring13
The DO cument G rounding and r E ferring data engine ( DOGE-Engine ), which produces two types of high-quality fine-grained document data: multi-granular parsing data for enhancing fun-* and multi-granular parsing data for enhancing fun-* documents.
- Learning Anchor Transformations for 3D Garment Animation13
An anchor-based deformation model, namely AnchorDEF, to predict 3D garment animation from a body motion sequence that deforms a garment mesh template by a mixture of rigid transformations with extra nonlinear displacements and an adaptive anchor updating is proposed to optimize the anchor position.
- 13
- Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval13
This work proposes a new model upgrade paradigm, termed Bidirectional Compatible Training (BiCT), which will upgrade the old gallery embeddings by forward-compatible training towards the embedding space of the backward-compatible new model.
- NeRF-Texture: Synthesizing Neural Radiance Field Textures12
This work proposes a novel texture synthesis method with Neural Radiance Fields (NeRF) to capture and synthesize textures from given multi-view images and can also synthesize NeRF textures over curved surfaces, which are practically useful.
- Weakly-Supervised Temporal Action Localization by Progressive Complementary Learning12
This work addresses the Weakly-Supervised Temporal Action Localization from a novel category exclusion perspective, which gradually enhances the snippet-level supervision to bridge the gap between video-level supervision and unavailable snippet-level supervision.
- Learning Music-Dance Representations Through Explicit-Implicit Rhythm Synchronization12
MuDaR is introduced, a novel Music-Dance Representation learning framework to perform the synchronization of music and dance rhythms both in explicit and implicit ways, and outperforms other self-supervised methods by a large margin.
- 12
- From an ordinary paper to a wireless and mobile input device12
This paper presents a vision-based gesture interface system, VISUAL PANEL, which employs an arbitrary quadrangle-shaped panel and a tip pointer as an intuitive input device that can accurately and reliably track the panel and the tip pointer.
- VISUAL PANEL: From an ordinary paper to a wireless and mobile input device12
This paper presents a vision-based gesture interface system, VISUAL PANEL, which employs an arbitrary quadrangle-shaped panel and a tip pointer as an intuitive input device that can accurately and reliably track the panel and the tip pointer.
- Image Conductor: Precision Control for Interactive Video Synthesis11
Image Conductor, a method for precise control of camera transitions and object movements to generate video assets from a single image, is proposed, and a trajectory-oriented video motion data curation pipeline for training is developed.
- Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation11
This work introduces Divot, a Diffusion-Powered Video Tokenizer, which leverages the diffusion process for self-supervised video representation learning, and presents Divot-LLM, a diffusion-based video tokenizer that achieves competitive performance across various video comprehension and generation benchmarks.
- 11
- How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?11
A novel Contrastive Partial Ranking Distillation (CPRD) method is proposed, which implements the objective of mimicking relative order between hard negative samples with contrastive learning, and surpasses other distillation methods and significantly improves the accuracy of dual-encoder.
- IC-Custom: Diverse Image Customization via In-Context Learning10
The In-context Multi-Modal Attention (ICMA) mechanism is proposed, which employs learnable task-oriented register tokens and boundary-aware positional embeddings to enable the model to effectively handle diverse tasks and distinguish between inputs in polyptych configurations.
- Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models10
An automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR, and a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs, which condenses question semantics and extracts question-relevant knowledge from pretrained 4D reconstruction priors into a compact set of geometry tokens.
- AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction10
This work proposes AnimeGamer, which is built upon Multimodal Large Language Models (MLLMs) to generate each game state, including dynamic animation shots that depict character movements and updates to character states, as illus-trated in Figure 1.
- Cobra: Efficient Line Art COlorization with BRoAder References10
Cobra is an efficient and versatile method that supports color hints and utilizes over 200 reference images while maintaining low latency, and leverages specially designed positional encodings, causal sparse attention, and Key-Value Cache to effectively manage long-context references and ensure color identity consistency.
- CustomNet: Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models10
CustomNet is presented, a unified encoder-based object customization framework that explicitly incorporates 3D novel view synthesis capabilities into the customization process, and outperforms other customization methods regarding identity preservation, diversity, and harmony.
- OmniZoomer: Learning to Move and Zoom in on Sphere at High-Resolution10
A novel deep learning-based approach, called OmniZoomer, to incorporate the Möbius transformation into the network for movement and zoom on ODIs, and proposes a spherical resampling module that combines the index map and HR feature maps to transform the feature maps for better spherical correlation.
- NeAI: A Pre-convoluted Representation for Plug-and-Play Neural Ambient Illumination10
A fully differentiable framework named neural ambient illumination (NeAI) that uses Neural Radiance Fields (NeRF) as a lighting model to handle complex lighting in a physically based way and represents a significant step towards integrating physically based rendering into the NeRF representation.
- TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter10
A parameter-efficient and task-agnostic adapter, dubbed TaCA, that facilitates compatibility across distinct foundation models while ensuring enhanced performance for the new models and allows downstream applications to seamlessly integrate better-performing foundation models without necessitating retraining.
- Audio-To-Symbolic Arrangement Via Cross-Modal Music Representation Learning10
A cross-modal representation-learning model is contributed, which extracts chord and melodic information from the audio and learns texture representation from both audio and a corrupted ground truth arrangement and introduces a tailored training strategy that gradually shifts the source of texture information from corrupted score to audio.
- Mono2Stereo: A Benchmark and Empirical Study for Stereo Conversion9
The Mono2Stereo dataset is introduced, providing high-quality training data and benchmark to support in-depth exploration of stereo conversion and a new evaluation metric, Stereo Intersection-over-Union, which prioritizes disparity and achieves a high correlation with human judgments on stereo effect.
- LoRA-Gen: Specializing Large Language Model via Online LoRA Generation9
The LoRA-Gen framework is proposed, which utilizes a large cloud-side model to generate LoRA parameters for edge-side models based on task descriptions and merges the LoRA parameters into the edge-side model to achieve flexible specialization.
- AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation9
Experimental results show that the model trained on AnimeShooter achieves superior cross-shot visual consistency and adherence to reference visual guidance, which highlight the value of the dataset for coherent animated video generation.
- From Denoising to Refining: A Corrective Framework for Vision-Language Diffusion Model9
ReDiff is introduced, a refining-enhanced diffusion framework that teaches the model to identify and correct its own errors, enabling stable and efficient parallel generation far superior to traditional denoising methods.
- ColorFlow: Retrieval-Augmented Image Sequence Colorization9
Results show that ColorFlow outperforms existing models across multiple metrics, setting a new standard in sequential image colorization and potentially benefiting the art industry.
- ID-Pose: Sparse-view Camera Pose Estimation by Inverting Diffusion Models9
This work presents ID-Pose, which inverses the denoising diffusion process to estimate the relative pose given two input images, and requires no training and generalizes to open-world images.
- SemanticBoost: Elevating Motion Generation with Augmented Textual Cues9
The experimental results demonstrate that SemanticBoost, as a diffusion-based method, outperforms auto-regressive-based techniques, achieving cutting-edge performance on the Humanml3D dataset while maintaining realistic and smooth motion generation quality.
- 9
- CREATE: A Benchmark for Chinese Short Video Retrieval and Title Generation9
CREATE is proposed, the first large-scale Chinese shoRt vidEo retrievAl and Title gEneration benchmark, and a novel model ALWIG which combines video retrieval and video titling tasks to achieve the purpose of multi-modal ALignment WIth Generation with the help of video tags and a GPT pre-trained model.
- 9
- DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation8
DI-PCG offers a promising approach for efficient inverse PCG and represents a valuable exploration step towards a 3D generation path that models how to construct a 3D asset using parametric models.
- 8
- 8
- Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Models8
This work presents Story3D-Agent, a pioneering approach that leverages the capabilities of LLMs to transform provided narratives into 3D-rendered visualizations, and enables precise control over multi-character actions and motions, as well as diverse decorative elements, ensuring the long-range and dynamic 3D representation.
- Hybrid Warping Fusion for Video Frame Interpolation8
This work explicitly investigates the warping step and proposes a way to combine the strength from using both forward and backward warping, and introduces hybrid warping fusion for frame interpolation.
- Visual Screen: Transforming an Ordinary Screen into a Touch Screen8
An image-screen mapping algorithm to correct the non-flatness of the computer screen and a segmentation method specially tailored for computer screen background are proposed.
- 7
- A Non-Invasive Fetal QRS Complex Detection Method Based on a Multi-Feature Fusion Neural Network7
A method for extracting fetal QRS complexes from a single-channel non-invasive fetal ECG based on a multi-feature fusion neural network using three multi-feature fusion strategies—feature-level fusion, decision-level fusion, and model-level fusion—achieves the effect of quickly identifying fetal QRS complexes.
- Analysis and Benchmarking of Extending Blind Face Image Restoration to Videos7
A Temporal Consistency Network (TCN) cooperated with alignment smoothing to reduce jitters and flickers in restored videos is proposed, a flexible component that can be seamlessly plugged into the most advanced face image restoration algorithms, ensuring the quality of image-based restoration is maintained as closely as possible.
- Humtrans: A Novel Open-Source Dataset for Humming Melody Transcription and Beyond7
The HumTrans dataset is introduced, which is publicly available and primarily designed for humming melody transcription, and consists of 500 musical compositions of different genres and languages, with each composition divided into multiple segments.
- 7
- 7
- Toward Human Perception-Centric Video Thumbnail Generation7
Extensive experimental results show that the HPCVTG framework outperforms existing methods in objective and subjective evaluations, highlighting its potential to improve the user experience when browsing videos and inspire future research in human perception-centric content generation tasks.
- HMC: Hierarchical Mesh Coarsening for Skeleton-free Motion Retargeting7
The qualitative results show that the proposed mesh-coarsening module is significantly helpful in preserving the moving consistency of different body parts on the target character due to disentangling body-part structures and mesh details in a hierarchical way.
- TransFusion: Multi-Modal Fusion for Video Tag Inference via Translation-based Knowledge Embedding7
This paper proposes TransFusion, an end-to-end supervised learning framework that fuses multi-modal embeddings with the knowledge embedding to derive the video representation and adopts a dual attentive approach to learn both the modality importance in fusion and relation importance in inference.
- 7
- 7
- NovelGS: Consistent Novel-view Denoising via Large Gaussian Reconstruction Model6
Experimental results on publicly available datasets indicate that NovelGS substantially surpasses existing image-to-3D frameworks, both qualitatively and quantitatively, and demonstrates state-of-the-art performance in addressing the multi-view image reconstruction challenge.
- What Does Your Face Sound Like? 3D Face Shape towards Voice6
3D face shape is introduced which has an anatomical relationship between voice characteristics, partaking in the "bone conduction" of human timbre production, and is naturally independent of irrelevant factors by excluding the blending process.
- Music-driven Dance Regeneration with Controllable Key Pose Constraints6
A novel framework for music-driven dance motion synthesis with controllable key pose constraint is proposed, which involves two single-modal transformer encoders for music and initial seed motion embedding, and a cross- modal transformer decoder for motion generation controlled by key pose constraints.
- Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval6
This work revitalizes region features of sparsely sampled video clips to significantly reduce both spatial and temporal visual redundancy towards democratizing VLP research at the same time achieving state-of-the-art results.
- Vis2Mus: Exploring Multimodal Representation Mapping for Controllable Music Generation6
This study discovers that visual-to-music mapping has a nice property similar to equivariant, and releases the Vis2Mus system as a controllable interface for symbolic music generation.
- 6
- Partial Object Matching with Shapeme Histograms6
Results of matching partially viewed range objects with a 243 model database demonstrate better performance than the original shapeme histogram matching algorithm and other approaches.
- Identifying Eclipsing Binary Stars with TESS Data Based on a New Hybrid Deep Learning Model5
This work proposes a novel method that combines LC data and generalized Lomb-Scargle periodograms (GLS) data to classify EBs, and applies it to TESS 2-minute LCs from Sectors 1 to 74, identifying 9351 new EBs after manual visual inspection.
- DiCoDe: Diffusion-Compressed Deep Tokens for Autoregressive Video Generation with Language Models5
DiCoDe is a novel approach that leverages Diffusion-Compressed Deep Tokens to generate videos with a language model in an autoregressive manner, and is scalable using readily available AR architectures, and is capable of generating videos ranging from a few seconds to one minute using only 4 A100 GPUs for training.
- Learning layout generation for virtual worlds5
A generative model of land-use distribution that learns from geographic data is proposed, based on a transformer architecture that generates a 2D map of the land-use layout, which can be conditioned on spatial and semantic controls, depending on whether either one or both are provided.
- 5
- A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields5
A fully differentiable framework named Neural Illumination Fields (NeIF) that uses radiance fields as a lighting model to handle complex lighting in a physically based way and represents a significant step towards integrating physically based rendering into the NeRF representation.
- 5
- ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation5
A training-free video interpolation method for generative video diffusion models, which is generalizable to different models in a plug-and-play manner and transform a video model into a self-cascaded video diffusion model with incorporating the designed hidden state correction modules.
- 5
- 5
- Characterizing the Impacts of Instances on Robustness5
It is shown that robust and non-robust instances in the training dataset, though are both important for test performance, have contrary impacts on robustness, which makes it possible to build a highly robust model by leveraging the training dataset in a more effective way.
- 5
- 5
- Image Inpainting Models are Effective Tools for Instruction-guided Image Editing4
This is the technique report for the winning solution of the CVPR2024 GenAI Media Generation Challenge Workshop's Instruction-guided Image Editing track, and shows that simply connecting large language models and image generation models through intermediary guidance such as masks instead of joint fine-tuning leads to a better editing performance and success rate.
- SparseGNV: Generating Novel Views of Indoor Scenes with Sparse RGB-D Images4
This work presents SparseGNV: a learning framework that incorporates 3D structures and image generative models to generate novel views with three modules that outperforms state-of-the-art methods based on either neural radiance fields or conditional image generation.
- SynopGround: A Large-Scale Dataset for Multi-Paragraph Video Grounding from TV Dramas and Synopses4
A large-scale video grounding dataset named SynopGround is presented, in which more than 2800 hours of videos are sourced from popular TV dramas and are paired with accurately localized human-written synopses, and a novel Local-Global Multimodal Reasoner (LGMR) is proposed to explicitly model the local-global structures of long-term multimodal inputs for MPVG.
- 4
- 4
- 4
- Internet Vision [Scanning the Issue]4
This issue includes coverage of a number of significant advances in Internet vision, defined as the intersection of computer vision and the Internet.
- DepthSync: Diffusion Guidance-Based Depth Synchronization for Scale- and Geometry-Consistent Video Depth Estimation3
This paper proposes DepthSync, a novel, training_free framework using diffusion guidance to achieve scale- and geometry-consistent depth predictions for long videos, and introduces scale guidance to synchronize the depth scale across windows and geometry guidance to enforce geometric alignment within windows based on the inherent 3D constraints in video depths.
- BlobCtrl: Taming Controllable Blob for Element-level Image Editing3
This work presents BlobCtrl, a framework for element-level image editing based on a probabilistic blob-based representation that disentangles layout from appearance, affording fine-grained, controllable object-level elements manipulation.
- Unified Pretraining Target Based Video-Music Retrieval with Music Rhythm and Video Optical Flow Information3
The proposed approach leverages a unified target set to perform video/music pretraining and produces clip-level embeddings to preserve temporal information and can achieve superior performance over the state-of-the-art methods by a significant margin.
- 3
- Prosody Modeling with 3D Visual Information for Expressive Video Dubbing3
This paper proposes a 3D-VD (3D Video Dubber) system to incorporate visual prosody, utilizing a visual-text step-wise aligner to con-trol the generated prosody and demonstrates the correlation between expression and pitch.
- 3
- Enhancing the Vocal Range of Single-Speaker Singing Voice Synthesis with Melody-Unsupervised Pre-Training3
This work proposes a melody-unsupervised multi-speaker pretraining method conducted on a multi-singer dataset to enhance the vocal range of the single-speakers, while not degrading the timbre similarity.
- 3
- 3
- Compressive Sensing Approaches for Sparse Distribution Estimation Under Local Privacy3
This paper considers sparse or approximately sparse distribution estimation under local differential privacy constraints, and shows that the number of samples needed could be significantly reduced, and proposes new privatization mechanisms based on compressive sensing.
- 3
- 3
- 3
- 3
- 3
- 2
- 2
- Aligning Latent Spaces with Flow Priors2
This paper presents a novel framework for aligning learnable latent spaces to arbitrary target distributions by leveraging flow-based generative models as priors, and formally proves that minimizing this alignment loss establishes a computationally tractable surrogate objective for maximizing a variational lower bound on the log-likelihood of latents under the target distribution.
- Learning High-Quality Navigation and Zooming on Omnidirectional Images in Virtual Reality2
A novel system, called OmniVR, designed to enhance visual clarity during VR navigation and user engagement by improving the viewers' recognition, reducing discomfort, and improving the overall immersive experience is presented.
- 2
- DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos2
A self-supervised method to jointly learn 3D motion and depth from monocular videos, containing a depth estimation module to predict depth, and a new decomposed object-wise 3D motion (DO3D) estimation module to predict ego-motion and 3D object motion.
- 2
- VTLayout: A Multi-Modal Approach for Video Text Layout2
A Video-level Text Layout model (VTLayout) is proposed to address the layout problem, which fuses textual, visual, and spatial-temporal embedding of video text trajectories and is the first method to tackle text layout on video level.
- GET3D--: Learning GET3D from Unconstrained Image Collections2
This work proposes GET3D--, the first method that directly generates textured 3D shapes from 2D images with unknown pose and scale and proposes a novel training schedule to stably optimize both the shape generator and camera sampler in a unified framework.
- SparseGNV: Generating Novel Views of Indoor Scenes with Sparse Input Views2
This work presents SparseGNV: a learning framework that incorporates 3D structures and image generative models to generate novel views with three modules that outperforms state-of-the-art methods based on either neural radiance fields or conditional image generation.
- Depth‐Aware Shadow Removal2
A novel end‐to‐end depth‐aware shadow removal method without using depth images is proposed, which estimates depth information from RGB images and leverages the depth feature as guidance to enhance shadow removal and refinement.
- 2
- Overview of the NLPCC 2019 Shared Task: Open Domain Conversation Evaluation2
An overview of the Open Domain Conversation Evaluation task in NLPCC 2019 is presented and the baseline system achieves a single-round score of 55 out of 100 and a multi- round score of 292 out of 400, indicating the system is more of an answering system rather than a chatting system.
- 2
- 1
- 1
- Gestational diabetes mellitus prediction using image-encoded electronic medical records and Transformer-based fusion1
A Transformer-based multimodal fusion model that integrates tabular clinical features and image-encoded electronic health records (EHRs) aiming for accurate end-to-end classification of gestational diabetes mellitus is developed.
- 1
- Consistent Human Image and Video Generation with Spatially Conditioned Diffusion1
By fine-tuning existing base diffusion models on human video data, the method demonstrates strong generalization to unseen human identities and poses without requiring additional per-instance fine-tuning, and ensures flexible control over the appearance of generated human images and videos.
- 1
- 1
- Towards Unseen Triples: Effective Text-Image-joint Learning for Scene Graph Generation1
A Text-Image-joint Scene Graph Generation (TISGG) model to resolve the unseen triples and improve the generalisation capability of the SGG models, and a novel balanced learning strategy, including a Charater Guided Sampling and an Informative Re-weighting module, to provide tailor-made learning methods for each predicate according to their characters.
- 1
- 1
- 1
- 1
- Gain-Scheduled LPV Control of a Single-Machine Infinite-Bus Power System1
This chapter presents a new nonlinear control method for the transient stability control that can be transformed equivalently into an LPV system dependent on the rotor angle as the scheduling parameter and can be solved by MATLAB tool.
- 1
- Curve Matching with Probabilistic Relaxation.1
The 1riai11 corltribution in this paper is to propose a new algorithr~i for curve matching (illcludi~lg lines) and the nlethod is described within a probabilistic relaxatio~l fra~rlework.
- ViewWeaver: Geometry-Grounded Generative Rendering for 3D-Aware Image Customization–
ViewWeaver is proposed, a geometry-grounded generative rendering framework that anchors synthesis with explicit target-view rendering cues aligned with camera geometry, and aggregates multi-view evidence through a lightweight Mixture-of-Views module inside a rectified-flow DiT generator.
- –
- –
- –
- –
- Development and validation of non-invasive prediction models for assessing kidney histopathological activity index in lupus nephritis–
The authors' Model 2 incorporating demographic and systemic indicators exhibited good performance in estimating the AI of LN, providing a simple yet effective algorithm to predict AI in patients with LN, potentially aiding clinicians in noninvasively assessing disease activity and guiding treatment decisions.
- –
- –
- –
- –
- –
- Content and Attention Aware Overlay for Online Video Advertising–
This work is the first attempt to generate video overlay ads that balances the two conflicting characteristics, by jointly optimizing a non-intrusive metric and a set of metrics associated with video ad templates designed by UI experts.
- Application of Digital Frequency Synthesis Technique in Missile-borne Equipment–
Presents a design method based on programmable logic devices, and discusses the future applications of DDS, and tries to present some predict in this field.
- CLUSTERING MULTIPLE IMAGE SEQUENCES WITH A SEQUENCE-TO-SEQUENCE SIMILARITY MEASURE–
A novel similarity measure of two image sequences based on shapeme histograms is proposed, which is used here to solve the challenge of single image/texture recognition.
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-12. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.