Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Chenliang XuSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works103 from public data
- Residual Dense Network for Image Super-Resolution4,163
This paper proposes residual dense block (RDB) to extract abundant local features via dense connected convolutional layers and uses global feature fusion in RDB to jointly and adaptively learn global hierarchical features in a holistic way.
- 1,955
- Deep Learning for Single Image Super-Resolution: A Brief Review1,151
This survey reviews representative deep learning-based SISR methods and group them into two categories according to their contributions to two essential aspects of S ISR: The exploration of efficient neural network architectures for SISS and the development of effective optimization objectives for deep SISr learning.
- Residual Dense Network for Image Restoration895
This work proposes residual dense block (RDB) to extract abundant local features via densely connected convolutional layers and proposes local feature fusion in RDB to adaptively learn more effective features from preceding and current local features and stabilize the training of wider network.
- TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution655
Experimental results demonstrate that theTDAN is capable of alleviating occlusions and artifacts for temporal alignment and the TDAN-based VSR model outperforms several recent state-of-the-art VSR networks with a comparable or even much smaller model size.
- 491
- DiffIR: Efficient Diffusion Model for Image Restoration476
This work proposes an efficient DM for IR (DiffIR), which consists of a compact IR prior extraction network (CPEN), dynamic IR transformer (DIRformer), and denoising network, which can adopt a joint optimization of CPENS2, DIRformer, and denoising network, which can further reduce the estimation error influence.
- Learning to Answer Questions in Dynamic Audio-Visual Scenarios295
The results demonstrate that AVQA benefits from multisensory perception and the model outperforms recent A-, V-, andAVQA approaches.
- Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing234
Experimental results show that the challenging audio-visual video parsing can be achieved even with only video-level weak labels, and the proposed framework can effectively leverage unimodal and cross-modal temporal contexts and alleviate modality bias and noisy labels problems.
- Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution214
A one-stage space-time video super-resolution framework is proposed, which directly synthesizes an HR slow-motion video from an LFR, LR video and proposes a deformable ConvLSTM to align and aggregate temporal information simultaneously for better leveraging global temporal contexts.
- Transformer-empowered Multi-scale Contextual Matching and Aggregation for Multi-contrast MRI Super-resolution124
A novel network of innovative Transformer-empowered multi-scale contextual matching and aggregation techniques is proposed, which outperforms state-of-the-art approaches and has great potential to be applied in clinical practice.
- Efficient Non-local Contrastive Attention for Image Super-resolution106
A novel Efficient Non-Local Contrastive Attention (ENLCA) to perform long-range visual modeling and leverage more relevant non-local features and reaches superior performance over state-of-the-art approaches on both quantitative and qualitative evaluations.
- Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation99
This paper proposes a cyclic co-learning (CCoL) paradigm that can jointly learn sounding object visual grounding and audio-visual sound separation in a unified framework and improves training example sampling for sounding object grounding, which builds a co- learning cycle for the two tasks and makes them mutually beneficial.
- Learning in Audio-visual Context: A Review, Analysis, and New Perspective81
This survey reviews and outlooks the current audio-visual learning field from different aspects and proposes a new perspective on audio- Visual scene understanding, then discusses and analyze the feasible future direction of the audio- visual learning area.
- Audio-Visual Grouping Network for Sound Localization from Mixtures75
This paper proposes a novel audio-visual grouping network, namely AVGN, that can directly learn category-wise semantic features for each source from the input audio mixture and image to localize multiple sources simultaneously.
- CFSNet: Toward a Controllable Feature Space for Image Restoration70
The proposed framework, named Controllable Feature Space Network (CFSNet), is entangled by two branches based on different objectives, which can adaptively learn the coupling coefficients of different layers and channels, which provides finer control of the restored image quality.
- AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation65
This work proposes a simple yet effective audio-visual localization and segmentation framework based on the Segment Anything Model, namely AV-SAM, that can generate sounding object masks corresponding to the audio.
- Cross Modality Bias in Visual Question Answering: A Causal View With Possible Worlds VQA63
This is the first work to reduce biases resulting from confounding effects of vision and language in VQA, leveraging causal explain-away relations and outperforms the state-of-the-art methods in VQA-CP v2 datasets.
- Egocentric Audio-Visual Object Localization55
This paper proposes a geometryaware temporal aggregation module that handles the egomotion explicitly and proposes a cascaded feature enhancement module to over-come the second issue and improves cross-modal localization robustness by disentangling visually-indicated audio representation.
- Audio-Visual Class-Incremental Learning55
AV-CIL is proposed, which incorporates Dual-Audio-Visual Similarity Constraint (D-AVSC) to maintain both instance-aware and class-aware semantic similarity between audio-visual modalities and Visual Attention Distillation (VAD) to retain previously learned audio-guided visual attentive ability.
- 54
- Can audio-visual integration strengthen robustness under multimodal attacks?46
A systematic study on machines’ multisensory perception under attacks using the audio-visual event recognition task against multi-modal adversarial attacks as a proxy to investigate the robustness of audio- visual learning.
- 45
- AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation44
AV-DiT is introduced, a novel and efficient audio-visual diffusion transformer designed to generate high-quality, realistic videos with both visual and audio tracks, and highlights that a single shared image generative backbone with modality-specific adaptations is sufficient for constructing a joint audio-video generator.
- CMRxRecon: A publicly available k-space dataset and benchmark to advance deep learning for cardiac MRI43
A dataset that includes multi-contrast, multi-view, multi-slice and multi-coil CMR imaging data from 300 subjects is released, introducing standardized evaluation criteria and making the dataset freely accessible to the research community.
- SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers42
An AI-powered system that enables BLV users to interactively explore video content and augments existing ADs by adding interactivity, spatial sound effects, and individual object descriptions without requiring additional human annotation is introduced.
- Class-Incremental Grouping Network for Continual Audio-Visual Learning40
A novel class-incremental grouping network (CIGN) that can learn category-wise semantic features to achieve continual audio-visual learning and utilizes class tokens distillation and continual grouping to prevent forgetting parameters learned from previous tasks, thereby improving the model’s ability to capture discriminative audiovisual categories.
- Structured Sparsity Learning for Efficient Video Super-Resolution40
In SSL, a structured pruning scheme called Structured Sparsity Learning (SSL) is developed according to the properties of VSR, which can significantly outperform recent methods quantitatively and qualitatively.
- Meta-Learning-Based Degradation Representation for Blind Super-Resolution38
An implicit degradation estimator that can extract discriminative degradation representations for all types of degradations without requiring the supervision of degradation ground-truth is proposed by a Meta-Learning based Region Degradation Aware SR Network (MRDA).
- DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation38
DREAM-Talk is introduced, a two-stage diffusion-based audio-driven framework, tailored for generating diverse expressions and accurate lip-sync concurrently, and outperforms state-of-the-art methods in terms of expressiveness, lip-sync accuracy and perceptual quality.
- SignDiff: Diffusion Model for American Sign Language Production37
A dual-condition diffusion pre-training model named SignDIFF that can generate human sign language speakers from a skeleton pose that has a novel Frame Reinforcement Network called FR-Net, similar to dense human pose estimation work, which enhances the correspondence between text lexical symbols and sign language dense pose frames.
- Basic Binary Convolution Unit for Binarized Image Restoration Network36
This study reconsider components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for IR tasks, and finds that residual connection can reduce the information loss caused by binarization, and BBCU can serve as a basic unit for binarized IR networks.
- Coarse-to-Fine Embedded PatchMatch and Multi-Scale Dynamic Aggregation for Reference-Based Super-resolution36
An Accelerated Multi-scale Aggregation network (AMSA) for Reference-based Super-Resolution, including Coarse-to-Fine Embedded PatchMatch (CFE-PatchMatch) and Multi-Scale Dynamic Aggregation (MSDA) module is proposed and Experimental results show that the proposed AMSA achieves superior performance over state-of-the-art approaches on both quantitative and qualitative evaluations.
- Text-to-Audio Generation Synchronized with Videos35
A groundbreaking benchmark for Text-to-Audio generation that aligns with Videos, named T2AV-Bench, and a simple yet effective video-aligned TTA generation model, namely T2AV, which refines the latent diffusion approach by integrating visual-aligned text embeddings as its conditional foundation.
- CMRxRecon: An open cardiac MRI dataset for the competition of accelerated image reconstruction35
The aim is to facilitate the advancement of state-of-the-art CMR image reconstruction by introducing standardized evaluation criteria and making the dataset freely accessible to the research community.
- Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields35
A novel Neural Acoustic Context Field approach to parameterize an audio scene by leveraging multiple acoustic contexts, such as geometry, material property, and spatial information, and Experimental results show that NACF outperforms existing field-based methods by a notable margin.
- STDAN: Deformable Attention Network for Space-Time Video Super-Resolution34
A deformable attention network called STDAN is proposed for STVSR, in which spatial and temporal contexts in dynamic video frames are adaptively captured and aggregated to enhance SR reconstruction and experimental results demonstrate that the approach outperforms state-of-the-art STVSR methods.
- DuDoCAF: Dual-Domain Cross-Attention Fusion with Recurrent Transformer for Fast Multi-contrast MR Imaging34
The proposed DuDoCAF scheme with recurrent transformer enables deep andective fusion of features extracted from two modalities, and dual-domain recurrent learning allows the model to restore signals in both k -space and image domains, and hence more comprehensively remove the artifacts.
- 34
- DiffI2I: Efficient Diffusion Model for Image-to-Image Translation32
Compared to traditional DMs, the compact IPR enables DiffI2I to obtain more accurate outcomes and employ a lighter denoising network and fewer iterations, and it is demonstrated that DiffI2I achieves SOTA performance while significantly reducing computational burdens.
- DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures28
DiffTED is introduced, a new approach for one-shot audio-driven TED-style talking video generation from a single image that utilizes classifier-free guidance, empowering the gestures to flow naturally with the audio input without relying on pre-trained classifiers.
- Scaling Diffusion Mamba with Bidirectional SSMs for Efficient Image and Video Generation28
This work introduces a novel diffusion architecture, Diffusion Mamba (DiM), which foregoes traditional attention mechanisms in favor of a scalable alternative, and not only scales effectively but also outperforms existing diffusion transformers in both image and video generation tasks.
- CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision27
The technical evaluation demonstrates that the model outperforms the baseline on the authors' tool affordance dataset, while the user study indicates a preference for affordance augmentations over the traditional whole object augmentations.
- T-VSL: Text-Guided Visual Sound Source Localization in Mixtures27
This paper proposes incorpo-rating the text modality as an intermediate feature guide using tri-modal joint embedding models (e.g., Audio Clip) to disentangle the semantic audio-visual source correspon-dence in multi-source mixtures.
- 26
- Audio-Visual Interpretable and Controllable Video Captioning25
This paper proposes a novel multimodal convolutional neural network that aims to disentangle the interplay of the two modalities and make the first attempt to interpretable audio-visual video captioning.
- Toward Long Form Audio-Visual Video Understanding23
This article proposes an event-centric framework for localizing multisensory events as well as understanding their relations in long form videos and demonstrates considerable effectiveness in localizing multiple modality-aware events within long form videos.
- Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level22
Motion-Grounded Video Reasoning is introduced, a new motion understanding task that requires generating visual answers according to the input question, and hence needs implicit spatiotemporal reasoning and grounding, and is introduced with a novel baseline model named MoRA.
- VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation22
VinTAGe is introduced, a flow-based transformer model that jointly considers text and video to guide audio generation and achieves state-of-the-art results on the VGGSound benchmark.
- DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment22
This work proposes a novel and personalized text-to-sound generation approach with visual alignment based on latent diffusion models, namely DiffAVA, that can simply fine-tune lightweight visual-text alignment modules with frozen modality-specific encoders to update visual-aligned text embeddings as the condition.
- 21
- STADNet: Spatial-Temporal Attention-Guided Dual-Path Network for cardiac cine MRI super-resolution20
A novel Spatial-Temporal Attention-Guided Dual-Path Network (STADNet) for cardiac cine MRI super-resolution is proposed and a recurrent flow-enhanced attention module in the motion-aware temporal path that exploits the correlation between cine MRI frames and extracts the motion information of the heart is introduced.
- 20
- 18
- High-Quality Visually-Guided Sound Separation from Diverse Categories16
Compared to existing state-of-the-art discriminative audio-visual separation methods on the AVE and MUSIC datasets, results show that DAVIS outperforms other methods in separation quality, demonstrating the advantages of the framework for tackling the audio-visual source separation task.
- MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers16
This paper proposes MA-AVT, a new parameter-efficient audio-visual transformer employing deep modality alignment for corresponding multimodal semantic features, and introduces joint unimodal and multimodal token learning for aligning the two modalities with a frozen modality-shared transformer.
- PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data16
Peanut’s human-AI collaborative pipeline separates the multi-modal task into two single- modal tasks, and utilizes state-of-the-art object detection and sound-tagging models to reduce the annotators’ effort to process each frame and the number of manually-annotated frames needed.
- Adaptive channel-modulated personalized federated learning for magnetic resonance image reconstruction15
A novel Adaptive Channel-Modulated Federal learning framework for personalized MRI reconstruction, dubbed as ACM-FedMRI is proposed, which achieves outstanding results compared to other cutting-edge federated learning techniques in the field of MRI reconstruction.
- 15
- MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos14
Mimosa is a human-AI co-creation tool that enables amateur users to computationally generate and manipulate spatial audio effects and exemplifies a human-AI collaboration approach that uses a multistep pipeline that aligns its interpretable intermediate results with the user’s workflow.
- Space-Time Memory Network for Sounding Object Localization in Videos14
This work proposes a space-time memory network for sounding object localization in videos that can simultaneously learn spatio-temporal attention over both uni- modal and cross-modal representations from audio and visual modalities and outperforms recent state-of-the-art methods.
- Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models13
This work introduces and formulate a new task of generating first-order Ambisonics (FOA) given a sound category and sound source spatial location, and proposes Diff-SAGe, an end-to-end, flow-based diffusion-transformer model for this task.
- AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information Between Reality and Videos13
Aroma is introduced, an AI system that provides timely responses to the user based on real-time, context-aware assistance by integrating non-visual cues perceived by the user, a wearable camera feed, and video recipe content.
- LAVSS: Location-Guided Audio-Visual Spatial Audio Separation13
The proposed LAVSS is inspired by the correlation between spatial audio and visual location and adopts a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation.
- Scaling Concept With Text-Guided Diffusion Models11
ScalingConcept is introduced, a simple yet effective method to scale decomposed concepts up or down in real input without introducing new elements, and enables a variety of novel zero-shot applications across image and audio domains, including tasks such as canonical pose generation and generative sound highlighting or removal.
- 10
- Towards AI-Powered AR for Enhancing Sports Playability for People with Low Vision: An Exploration of ARSports9
It is suggested that combining real-time computer vision and augmented reality to create scene-aware visual augmentations is a promising approach to enhancing sports participation for LV individuals.
- Audio-Visual Event Localization in the Wild9
The extensive experiments support the following findings: modeling jointly over auditory and visual modalities outperforms modeling independently over them, audio-visual event localization in a noisy condition can still achieve promising results, the audio-guided visual attention can well capture semantic regions covering sounding objects and can even distinguish audio- visual unrelated videos.
- Image super resolution using deep convolutional network based on topology aggregation structure9
A new architecture of the deep convolutional network for single-image super-resolution (SR) inspired by GoogLeNet and Res Ne Xt, improved on VDSR which is a representative state-of-the-art method for deep learning-based SR approach is proposed.
- AV-DiT: Taming Image Diffusion Transformers for Efficient Joint Audio and Video Generation7
AV-DiT is introduced, a novel and efficient audio-visual diffusion transformer designed to generate high-quality, realistic videos with synchronized audio tracks and highlights that a single shared image generative backbone with modality-specific adaptations is sufficient for constructing a joint audio-video generator.
- Modality-Inconsistent Continual Learning of Multimodal Large Language Models7
MoInCL is proposed, which employs a Pseudo Targets Generation Module to mitigate forgetting caused by task type shifts in previously seen modalities and incorporates Instruction-based Knowledge Distillation to preserve the model's ability to handle previously learned modalities when new ones are introduced.
- 7
- Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?6
This study reveals a fundamental bias in current AVS methods: they tend to generate segmentation masks based predominantly on visual salience, irrespective of the audio context, resulting in unreliable predictions when sounds are absent or irrelevant, and introduces AVSBench-Robust, a comprehensive benchmark incorporating diverse negative audio scenarios.
- FreSca: Scaling in Frequency Space Enhances Diffusion Models6
This work introduces FreSca, a straightforward method that applies guidance scaling independently to different frequency bands in the Fourier domain, and demonstrably enhances existing image editing methods without retraining.
- Semantic Grouping Network for Audio Source Separation5
A novel Semantic Grouping Network, termed as SGN, is presented that can directly disentangle sound representations and extract high-level semantic information for each source from input audio mixture and significantly outperforms previous audio-only methods and audio-visual models without utilizing additional visual cues.
- 5
- Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation5
AVSA-Sep successfully separates both sound types, with joint training and cross-modal alignment enhancing effectiveness, and includes a semantic parser for visible and invisible sounds and a separator for scene-informed separation.
- 5
- 5
- EgoVSR: Toward High-Quality Egocentric Video Super-Resolution4
This work explicitly tackle motion blurs in egocentric videos using a Dual Branch Deblur Network (DB2Net) in the VSR framework, and introduces an EgoVSR dataset containing a large amount of fast-motion egocentric video sequences.
- 4
- 4
- Robust Active Speaker Detection in Noisy Environments3
A novel framework that utilizes audio-visual speech separation as guidance to learn noise-free audio features and a dynamic weighted loss approach to train the speech separator to improve ASD performance in noisy environments is proposed.
- SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering3
The Source-aware Semantic Representation Network (SaSR-Net), a novel model designed for AVQA that streamlines the fusion of audio and visual information using spatial and temporal attention mechanisms to identify answers in multi-modal scenes is introduced.
- Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA3
This work first model a confounding effect that causes language and vision bias simultaneously, then proposes a counterfactual inference to remove the influence of this effect, and concludes that the model trained in this strategy can concurrently and efficiently reduce vision and language bias.
- Toward Gaze Target Detection of Young Autistic Children2
A new, real-world AI application for gaze target detection in autistic children, which predicts a child's point of gaze from an activity image and proposes a novel Socially Aware Coarse-to-Fine (SACF) gaze detection framework that explicitly leverages the social context of a scene to overcome the class imbalance common in autism datasets.
- Joint Co-Speech Gesture and Expressive Talking Face Generation Using Diffusion with Adapters2
A novel model architecture is proposed that jointly generates face and body motions within a single network that leverages shared weights between modalities, facilitated by adapters that enable adaptation to a common latent space.
- Demonstration of VRSight: AI-Driven Real-Time Descriptions to Enhance VR Accessibility for Blind People2
This work presents VRSight, an end-to-end system using state-of-the-art machine learning models to empower BLV to more effectively interact with VR environments, and increases VR accessibility for BLV.
- Efficiently Leveraging Linguistic Priors for Scene Text Spotting2
This paper proposes a method that leverages linguistic knowledge from a large text corpus to replace the traditional one-hot encoding used in auto-regressive scene text spotting and recognition models, which allows the model to capture the relationship between characters in the same word.
- 2
- 2
- Touch with Meaning: A Contextual Analysis of Social Touch1
A contextual analysis of 5,016 social touch events, grounded in a large collection of annotated scenes from films, dramas, and documentaries, shows that identical gestures can convey distinct meanings depending on body location, relationship type, and context.
- Audio-Visual Intelligence in Large Foundation Models1
This survey provides the first comprehensive review of AVI through the lens of large foundation models, establishing a unified taxonomy covering the broad landscape of AVI tasks, and synthesizes methodological foundations, including modality tokenization, cross-modal fusion, autoregressive and diffusion-based generation, large-scale pretraining, instruction alignment, and preference optimization.
- Towards Unified, Explainable, and Robust Multisensory Perception1
The research has pioneered marrying computer vision with computer audition to create multimodal systems that can learn to understand audio and visual data and strives to develop unified, explainable, and robust multisensory perception machines.
- Co-Learn Sounding Object Visual Grounding and Visually Indicated Sound Separation in A Cycle1
A cyclic co-learning (CCoL) paradigm that can jointly learn sounding object visual grounding and visually indicated audio separation in a unified framework is proposed and achieves state-of-the-art performance on both sounding objectVisual grounding and visual indicated sound separation tasks.
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-14. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.