Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Jason J. CorsoSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works173 from public data
- Towards Automatic Learning of Procedures From Web Instructional Videos1,112
A segment-level recurrent network is proposed for generating procedure segments by modeling the dependencies across segments and it is shown that the proposed model outperforms competitive baselines in procedure segmentation.
- TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution655
Experimental results demonstrate that theTDAN is capable of alleviating occlusions and artifacts for temporal alignment and the TDAN-based VSR model outperforms several recent state-of-the-art VSR networks with a comparable or even much smaller model size.
- 494
- Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss480
A cascade GAN approach to generate talking face video, which is robust to different face shapes, view angles, facial characteristics, and noisy audio conditions, and compared to a direct audio-to-image approach, this approach avoids fitting spurious correlations between audiovisual signals that are irrelevant to the speech content.
- 333
- Video Understanding With Large Language Models: A Survey332
The emergent capabilities of Vid-LLMs are surprisingly advanced, particularly their ability for open-ended multi-granularity (abstract, temporal, and spatiotemporal) reasoning combined with common-sense knowledge, suggesting a promising path for future video understanding.
- Lip Movements Generation at a Glance331
This paper devise a method to fuse audio and image embeddings to generate multiple lip images at once and propose a novel correlation loss to synchronize lip changes and speech changes and significantly outperforms other state-of-the-art methods extended to this task.
- Learning to Answer Questions in Dynamic Audio-Visual Scenarios295
The results demonstrate that AVQA benefits from multisensory perception and the model outperforms recent A-, V-, andAVQA approaches.
- Streaming Hierarchical Video Segmentation262
This work proposes an approximation framework for streaming hierarchical video segmentation motivated by data stream algorithms: each video frame is processed only once and does not change the segmentation of previous frames.
- Deep Cross-Modal Audio-Visual Generation235
This paper uses conditional generative adversarial networks to achieve cross-modal audio-visual generation of musical performances and demonstrates that the model has the ability to generate one modality from the other modality, i.e., visual/audio, to a good extent.
- Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing234
Experimental results show that the challenging audio-visual video parsing can be achieved even with only video-level weak labels, and the proposed framework can effectively leverage unimodal and cross-modal temporal contexts and alleviate modality bias and noisy labels problems.
- 231
- Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution214
A one-stage space-time video super-resolution framework is proposed, which directly synthesizes an HR slow-motion video from an LFR, LR video and proposes a deformable ConvLSTM to align and aggregate temporal information simultaneously for better leveraging global temporal contexts.
- Talking-Head Generation with Rhythmic Head Motion213
This work proposes a 3D-aware generative network along with a hybrid embedding module and a non-linear composition module that achieves controllable, photo-realistic, and temporally coherent talking-head videos with natural head movements.
- Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment204
A novel action modeling framework is proposed, which consists of a new temporal convolutional network, named Temporal Convolutional Feature Pyramid Network (TCFPN), for predicting frame-wise action labels, and a novel training strategy for weakly-supervised sequence modeling, named Iterative Soft Boundary Assignment (ISBA), to align action sequences and update the network in an iterative fashion.
- 138
- MRI tumor segmentation with densely connected 3D CNN137
This work develops a 3D Convolutional Neural Network (3D CNN) based model to automatically segment gliomas and proposes a novel architecture that hierarchically segments different lesion regions of the necrotic and non-enhancing tumor (NCR/NET), peritumoral edema (ED) and GD-enhanced tumor (ET).
- 130
- Transformer-empowered Multi-scale Contextual Matching and Aggregation for Multi-contrast MRI Super-resolution124
A novel network of innovative Transformer-empowered multi-scale contextual matching and aggregation techniques is proposed, which outperforms state-of-the-art approaches and has great potential to be applied in clinical practice.
- A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others114
The Last Layer Ensemble is proposed, a simple-yet-effective method to mitigate multiple shortcuts without Whac-A-Mole behavior, which surface multi-shortcut mitigation as an overlooked challenge critical to advancing the reliability of vision systems.
- Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation99
This paper proposes a cyclic co-learning (CCoL) paradigm that can jointly learn sounding object visual grounding and audio-visual sound separation in a unified framework and improves training example sampling for sounding object grounding, which builds a co- learning cycle for the two tasks and makes them mutually beneficial.
- Learning to Generate Scene Graph from Natural Language Supervision96
This paper proposes one of the first methods that learn from image-sentence pairs to extract a graphical representation of localized objects and their relationships within an image, known as scene graph, and designs a Transformer-based model to predict these "pseudo" labels via a masked token prediction task.
- Watch What You Just Said: Image Captioning with Text-Conditional Attention85
The experimental results show that the proposed novel attention mechanism, called text-conditional attention, outperforms state-of-the-art captioning methods on various quantitative metrics as well as in human evaluation, which supports the use of the text- Conditional attention in image captioning.
- Automated classification of big X-ray diffraction data using deep learning models70
Given the critical need for high-throughput automated XRD pattern analysis, this work presents a generalized deep learning model to classify a diverse set of materials’ crystal systems and space groups and optimize model architecture to elicit classification based on Bragg’s Law.
- Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning68
This work leverage instructional videos to study humans’ decision-making processes, focusing on learning a model to plan goal-directed actions in real-life videos and proposes novel algorithms to model human behaviors through Bayesian Inference and model-based Imitation Learning.
- Cross Modality Bias in Visual Question Answering: A Causal View With Possible Worlds VQA63
This is the first work to reduce biases resulting from confounding effects of vision and language in VQA, leveraging causal explain-away relations and outperforms the state-of-the-art methods in VQA-CP v2 datasets.
- Generating Talking Face Landmarks from Speech60
A system that can generate landmark points of a talking face from an acoustic speech in real time using a long short-term memory (LSTM) network and is trained on frontal videos of 27 different speakers with automatically extracted face landmarks.
- 59
- Discover the Unknown Biased Attribute of an Image Classifier56
The qualitative results show that the method can discover unnoticeable biased attributes for various object and scene classifiers, proving the method’s generalizability for detecting biased attributes in diverse domains of images.
- 56
- Egocentric Audio-Visual Object Localization55
This paper proposes a geometryaware temporal aggregation module that handles the egomotion explicitly and proposes a cascaded feature enhancement module to over-come the second issue and improves cross-modal localization robustness by disentangling visually-indicated audio representation.
- StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis53
A new framework, StyleT2I, to improve the compositionality of text-to-image synthesis and proposes a CLIP-guided Contrastive Loss to better distinguish different compositions among different sentences and a novel Semantic Matching Loss and a Spatial Constraint to identify attributes' latent directions for intended spatial region manipulations, leading to better disentangled latent representations of attributes.
- 51
- Learning to Transform Dynamically for Better Adversarial Transferability50
This study introduces a novel approach named Learning to Transform (L2T), which increases the diversity of transformed images by selecting the optimal combination of operations from a pool of candidates, consequently proving adversarial transferability.
- Graph neural network based coarse-grained mapping prediction49
This work presents a graph neural network based CG mapping predictor called Deep Supervised Graph Partitioning Model (DSGPM) that treats mapping operators as a graph segmentation problem and finds that predicted CG mapping operators indeed result in good CG MD models when used in simulation.
- Can audio-visual integration strengthen robustness under multimodal attacks?46
A systematic study on machines’ multisensory perception under attacks using the audio-visual event recognition task against multi-modal adversarial attacks as a proxy to investigate the robustness of audio- visual learning.
- 46
- 45
- Image Caption Generation with Text-Conditional Semantic Attention.44
A semantic attention mechanism for image caption generation, called text-conditional semantic attention, which allows the caption generator to automatically learn which parts of the image feature to focus on given previously generated text is proposed.
- Actor-Action Semantic Segmentation with Grouping Process Models43
A new model is proposed that combines the labeling CRF with a supervoxel hierarchy, where supervoxels at various scales provide cues for possible groupings of nodes in the CRF to encourage adaptive and long-ranging interactions.
- Learning by Planning: Language-Guided Global Image Editing42
A novel operation planning algorithm to generate possible editing sequences from the target image as pseudo ground truth is proposed and compared on the newly collected MA5k-Req dataset and GIER dataset to show the advantages of these methods.
- A Benchmark and Baseline for Language-Driven Image Editing39
This work presents a new language-driven image editing dataset that supports both local and global editing with editing operation and mask annotations and proposes a baseline method that fully utilizes the annotation to solve this problem.
- Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields35
A novel Neural Acoustic Context Field approach to parameterize an audio scene by leveraging multiple acoustic contexts, such as geometry, material property, and spatial information, and Experimental results show that NACF outperforms existing field-based methods by a notable margin.
- 35
- 35
- High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation34
A deep learning lighting model is learned, that in combination with a high-quality 3D face tracking algorithm, provides a method for subtle and robust facial motion transfer from a regular video to a 3D photo-realistic avatar.
- Electronic Cigarette–Related Contents on Instagram: Observational Study and Exploratory Analysis34
Instagram is dominated by pro-Vaping content, and pro-vaping posts and user accounts seem to have more user engagement than vaping-warning accounts have, which highlights the importance of regulating e-cigarette posts on social media.
- 34
- 31
- 29
- 27
- 27
- 26
- Face Forgery Detection via Symmetric Transformer26
This work proposes a symmetric transformer for channel and spatial feature extraction, which ensures the robustness of detection from two different aspects of a robust forgery detector.
- Audio-Visual Interpretable and Controllable Video Captioning25
This paper proposes a novel multimodal convolutional neural network that aims to disentangle the interplay of the two modalities and make the first attempt to interpretable audio-visual video captioning.
- 24
- Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)24
A thorough and up-to-date comparison of reasoning techniques in both textual and multimodal LLMs is offered, highlighting practical methods for post-training optimization and test-time inference.
- Emotional Listener Portrait: Realistic Listener Motion Simulation in Conversation22
The Emotional Listener Portrait (ELP) model can not only automatically generate natural and diverse responses toward a given speaker via sampling from the learned distribution but also generate controllable responses with a predetermined attitude.
- TailorGAN: Making User-Defined Fashion Designs21
A novel self-supervised model to synthesize garment images with disentangled attributes (e.g., collar and sleeves) without paired data is proposed and can synthesize much better results than the state-of-the-art methods in both quantitative and qualitative comparisons.
- Deep Grouping Model for Unified Perceptual Parsing19
This work proposes a deep grouping model (DGM) that tightly marries the two types of representations and defines a bottom-up and a top-down process for feature exchanging and achieves state-of-the-art results while having a small computational overhead compared to other contextual-based segmentation models.
- 3D Human Avatar Digitization from a Single Image19
This paper proposes a pipeline that reconstructs 3D human shape avatar at a glance and simultaneously reconstructs the three-dimensional human geometry and whole body texture map with only a single RGB image as input.
- 17
- High-Quality Visually-Guided Sound Separation from Diverse Categories16
Compared to existing state-of-the-art discriminative audio-visual separation methods on the AVE and MUSIC datasets, results show that DAVIS outperforms other methods in separation quality, demonstrating the advantages of the framework for tackling the audio-visual source separation task.
- PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data16
Peanut’s human-AI collaborative pipeline separates the multi-modal task into two single- modal tasks, and utilizes state-of-the-art object detection and sound-tagging models to reduce the annotators’ effort to process each frame and the number of manually-annotated frames needed.
- Learning a Weakly-Supervised Video Actor-Action Segmentation Model With a Wise Selection16
A general Weakly-Supervised framework with a Wise Selection of training samples and model evaluation criterion (WS^2) is proposed, which achieves state-of-the-art performance on both weakly-supervised VOS and VAAS tasks and is on par with the best fully- supervised method on VAAS.
- Characteristics of and User Engagement With Antivaping Posts on Instagram: Observational Study15
This study identified the critical elements associated with high user engagement, which could be used to design antivaping posts to deliver health-related information more efficiently.
- GAN-EM: GAN Based EM Learning Framework15
The proposed GAN-EM achieves state-of-the-art clustering and semi-supervised classification results on MNIST, SVHN and CelebA, as well as comparable quality of generated images to other recently developed generative models.
- 15
- Speech-In-Noise Comprehension is Improved When Viewing a Deep-Neural-Network-Generated Talking Face14
The results of this study support the idea that a DNN-based model that generates a talking face from speech audio can meaningfully enhance comprehension in noisy environments, and has the potential to be used as a visual hearing aid.
- Space-Time Memory Network for Sounding Object Localization in Videos14
This work proposes a space-time memory network for sounding object localization in videos that can simultaneously learn spatio-temporal attention over both uni- modal and cross-modal representations from audio and visual modalities and outperforms recent state-of-the-art methods.
- 12
- Can multisensory training aid visual learning? A computational investigation12
Deep or artificial neural networks are used to address the question, Can multisensory training aid visual learning, and demonstrate that a network trained in a visual-haptic environment tends to learn visual representations containing useful abstractions and also learns representations that are less sensitive to imaging parameters.
- Scaling Concept With Text-Guided Diffusion Models11
ScalingConcept is introduced, a simple yet effective method to scale decomposed concepts up or down in real input without introducing new elements, and enables a variety of novel zero-shot applications across image and audio domains, including tasks such as canonical pose generation and generative sound highlighting or removal.
- Characterizing Anti-Vaping Posts for Effective Communication on Instagram Using Multimodal Deep Learning11
Key features identified from anti-vaping Instagram image posts will provide a potentially effective way to communicate with the public about the health effects of e-cigarette use to protect public health.
- 11
- 11
- 10
- 9
- 9
- 9
- 9
- Audio-Visual Event Localization in the Wild9
The extensive experiments support the following findings: modeling jointly over auditory and visual modalities outperforms modeling independently over them, audio-visual event localization in a noisy condition can still achieve promising results, the audio-guided visual attention can well capture semantic regions covering sounding objects and can even distinguish audio- visual unrelated videos.
- 9
- 8
- 8
- Cross-modal Contrastive Distillation for Instructional Activity Anticipation7
A novel cross-modal contrastive distillation (CCD) scheme is devised, which facilitates knowledge distillation between teacher and student in heterogeneous modalities with the proposed cross- modal distillation loss.
- 7
- 7
- 7
- 7
- GaussianStyle: Gaussian Head Avatar via StyleGAN6
The GaussianStyle preserves structural information, such as expressions and poses, using Gaussian points, while projecting the implicit volumetric representation into Style-GAN to capture high-frequency details and mitigate the over-smoothing commonly observed in neural texture rendering.
- FreSca: Scaling in Frequency Space Enhances Diffusion Models6
This work introduces FreSca, a straightforward method that applies guidance scaling independently to different frequency bands in the Fourier domain, and demonstrably enhances existing image editing methods without retraining.
- 6
- 6
- 5
- 5
- 5
- 5
- Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation5
AVSA-Sep successfully separates both sound types, with joint training and cross-modal alignment enhancing effectiveness, and includes a semantic parser for visible and invisible sounds and a separator for scene-informed separation.
- 5
- 5
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- 3
- 3
- 3
- 3
- Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA3
This work first model a confounding effect that causes language and vision bias simultaneously, then proposes a counterfactual inference to remove the influence of this effect, and concludes that the model trained in this strategy can concurrently and efficiently reduce vision and language bias.
- 3
- A Weakly Supervised Multi-task Ranking Framework for Actor–Action Semantic Segmentation3
A novel Schatten p-norm robust multi-task ranking model for weakly-supervised actor–action segmentation where only video-level tags are given for training samples that outperforms the state-of-the-art weakly supervised methods and performs as well as the top-performing fully supervised method.
- 3
- 3
- 2
- 2
- Efficiently Leveraging Linguistic Priors for Scene Text Spotting2
This paper proposes a method that leverages linguistic knowledge from a large text corpus to replace the traditional one-hot encoding used in auto-regressive scene text spotting and recognition models, which allows the model to capture the relationship between characters in the same word.
- 2
- 2
- 2
- 2
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- Co-Learn Sounding Object Visual Grounding and Visually Indicated Sound Separation in A Cycle1
A cyclic co-learning (CCoL) paradigm that can jointly learn sounding object visual grounding and visually indicated audio separation in a unified framework is proposed and achieves state-of-the-art performance on both sounding objectVisual grounding and visual indicated sound separation tasks.
- 1
- 1
- 1
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- Author Correction: Automated classification of big X-ray diffraction data using deep learning models–
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-16. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.