Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileWorks66 from public data
- Deconfounded Video Moment Retrieval with Causal Intervention248
A causality-inspired VMR framework is proposed that builds structural causal model to capture the true effect of query and video content on the prediction, and a Deconfounded Cross-modal Matching method is developed to remove the confounding effects of moment location.
- Boundary Proposal Network for Two-stage Natural Language Video Localization196
A novel Boundary Proposal Network (BPNet) is proposed, a universal two-stage framework that gets rid of the issues mentioned above and outperforms the state-of-the-art NLVL methods.
- Invariant Grounding for Video Question Answering128
A new learning framework is proposed, Invariant Grounding for VideoQA (IGV), to ground the question-critical scene, whose causal relations with answers are invariant across different interventions on the complement, which significantly improves the reasoning ability.
- 103
- NExT-Chat: An LMM for Chat, Detection and Segmentation96
This paper introduces a novel paradigm for object location modeling called pix2emb method, where the LMM is asked to output the location embeddings and then decode them with different decoders and demonstrates its capability of handling multiple tasks like visual grounding, region captioning, and grounded reasoning.
- 96
- Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching68
GeoText-1652 is introduced, a new natural language-guided geo-localization benchmark that extends the established University-1652 image dataset with spatial-aware text annotations, thereby establishing one-to-one correspondences between image, text, and bounding box elements.
- Content-Variant Reference Image Quality Assessment via Knowledge Distillation60
The content-variant reference method via knowledge distillation (CVRKD-IQA) is proposed, which uses non-aligned reference (NAR) images to introduce various prior distributions of high-quality images and can support more IQA applications with its robustness to content variations.
- Dysen-VDM: Empowering Dynamics-Aware Text-to-Video Diffusion with LLMs56
An innovative dynamic scene manager module, Inspired by human intuition, which includes an innovative dynamic scene manager (dubbed as Dysen) module, integrated into the backbone T2V DM for video generating, and realizes (nearly) human-level temporal dynamics understanding.
- Video Visual Relation Detection via Iterative Inference55
This work derives their model from the viewpoint of joint relation classification which is light-weight yet effective, and proposes a training approach to better learn the dependency knowledge from the likely correct triplet combinations.
- Online Distillation-enhanced Multi-modal Transformer for Sequential Recommendation45
This paper proposes a new model-agnostic framework for multi-modal sequential recommendation tasks, called Online Distillation-enhanced Multi- modal Transformer (ODMT), to enhance feature interaction and mutual learning among multi-source input (ID, text, and image), while avoiding conflicts among different features during training, thereby improving recommendation accuracy.
- 43
- 40
- Rethinking the Two-Stage Framework for Grounded Situation Recognition39
A novel SituFormer for GSR which consists of a Coarse-to-Fine Verb Model (CFVM) and a Transformer-based Noun Model (TNM), which is a transformer-based semantic role detection model, which detects all roles parallelly.
- 36
- 36
- Biased-Predicate Annotation Identification via Unbiased Visual Predicate Representation33
This work proposes a novel method that utilizes unbiased visual predicate representations for Biased-Annotation Identification (BAI) as a fundamental step for PSG/SGG tasks and shows great generalization and effectiveness on multiple datasets.
- Semantic Locality-Aware Deformable Network for Clothing Segmentation33
A semantic locality-aware segmentation model is proposed, which adaptively attaches an original clothing image with a semantically similar auxiliary exemplar by search to make the learning process of small sample learning more stable and tractable.
- Semantic Alignment for Multimodal Large Language Models31
Semantic Alignment for Multi-modal large language models (SAM) is introduced, which aims to enhance the preservation of linking information for coherent analysis and align the semantics of different images before feeding them into LLM.
- 29
- Turing Patterns for Multimedia: Reaction-Diffusion Multi-Modal Fusion for Language-Guided Video Moment Retrieval26
Reaction-Diffusion Multimodal Fusion (RDMF) is proposed, a novel framework that reimagines video-language alignment as a reaction-diffusion process, drawing on the principles of pattern formation introduced by Alan Turing, and represents a pioneering interdisciplinary approach.
- 26
- Mrtnet: Multi-Resolution Temporal Network for Video Sentence Grounding25
This work introduces MRTNet, a multi-resolution grounding network with four key components: a feature encoder, a Multi-Resolution Temporal (MRT) module, a Query-aware Attention (QAM) module, and a predictor.
- 25
- Multi-queue Momentum Contrast for Microvideo-Product Retrieval24
The microvideo-product retrieval task is formulated, which is the first attempt to explore the retrieval between the multi-modal and multi- modal instances, and a novel approach named Multi-Queue Momentum Contrast (MQMC) network is proposed for bidirectional retrieval.
- Conditional Hyper-Network for Blind Super-Resolution With Multiple Degradations23
A novel conditional hyper-network framework for super-resolution with multiple degradations (named CMDSR) is proposed, which helps the SR framework learn how to adapt to changes in the degradation distribution of input and demonstrates the effectiveness ofCMDSR over various blind, and even several non-blind methods.
- Partial Annotation-based Video Moment Retrieval via Iterative Learning22
A new setting of VMR is designed where users can easily point to small segments of non-controversy video moments and the proposed method can automatically fill in the remaining parts based on the video and query semantics, and a pretrained video action localization model is utilized to provide coarse guidance of potential video segments.
- VidVRD 202120
This paper makes a comprehensive and detailed introduction of this task, concludes the proposed algorithms in the last few years, and proposes future direction for research in this task.
- Context-Aware Deep Spatiotemporal Network for Hand Pose Estimation From Depth Images18
The proposed context-aware deep spatiotemporal network is able to learn the representations of the spatial information and the temporal structure from the image sequences and is capable of dynamically weighting different predictions to lay emphasis on sufficient context.
- 16
- DriveDiTFit: Fine-tuning Diffusion Transformers for Autonomous Driving Data Generation16
DriveDiTFit is proposed, a novel method for efficiently generating autonomous Driving data by Fine-tuning pre-trained Diffusion Transformers (DiTs) according to the discrepancy between the pre-trained source data and the target driving data.
- De-fine: De composing and Re fin ing Visual Programs with Auto-Feedback14
Inspired by benders decomposition, this work introduces De-fine, a training-free framework that automatically decomposes complex tasks into simpler subtasks and refines programs through auto-feedback, which can improve logical reasoning performance by integrating the strengths of multiple models.
- Weakly Supervised Video Moment Retrieval via Location-irrelevant Proposal Learning13
The proposed Location-irrelevant Proposal Learning (LPL) model is based on a pretrained video-text model with cross-modal prompt learning, together with different strategies to generate reasonable proposals with various lengths, which benefits from training with fine-grained video-text pairs.
- 12
- Domain-Wise Invariant Learning for Panoptic Scene Graph Generation12
This work proposes a novel framework to infer potentially biased annotations by measuring the predicate prediction risks within each subject-object pair (domain), and adaptively transfer the biased annotations to consistent ones by learning invariant predicate representation embeddings.
- 11
- FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents11
A comprehensive evaluation of state-of-the-art MLLMs is conducted and it is observed that no model surpasses 5% accuracy, underscoring the inherent difficulty of the task and revealing significant limitations in current models' visual layout reasoning and field-value alignment abilities.
- Toward Complex-query Referring Image Segmentation: A Novel Benchmark11
A new RIS benchmark with complex queries, namely RIS-CQ is proposed, of high quality and large scale, which challenges the existing RIS with enriched, specific, and informative queries, and enables a more realistic scenario of RIS research.
- Learning Style-Invariant Robust Representation for Generalizable Visual Instance Retrieval11
A novel Style-Invariant robust Representation Learning (SIRL) method is proposed for the Single Domain Generalization (SDG) based VIR task, which mainly aims to first diversify the training data with style augmentation, and then enforce the model to learn style-invariant features.
- Deep Multimodal Learning for Information Retrieval11
This workshop aims to provide a platform for discussion about multi-modal IR among scholars, practitioners, and other interested parties and show the great potential shown by multimodal-empowered IR.
- Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey10
This survey bridges the gap between traditional vector quantization and modern LLM applications, serving as a foundational reference for the development of efficient and generalizable multimodal systems.
- Fine-grained Structural Hallucination Detection for Unified Visual Comprehension and Generation in Multimodal LLM9
A dual-process approach utilizing a cross-graph siamese network to semantically differentiate the fine-grained discrepancies between the visual scene graph (VSG) and the textual scene graph (TSG) representations in both image-to-text (I2T) and text-to-image (T2I) processes is proposed.
- 9
- Backpropagation-Free Multi-modal On-Device Model Adaptation via Cloud-Device Collaboration8
A universal on-device Multi-modal Model Adaptation framework is introduced, revolutionizing on-device model adaptation by striking a balance between efficiency and effectiveness, and represents a pioneering solution for on-Device Multi-modal Model Adaptation (DMMA).
- SpeechEE: A Novel Benchmark for Speech Event Extraction8
This paper pioneers a SpeechEE, defined as detecting the event predicates and arguments from a given audio speech, and tailor an E2E SpeechEE system based on the encoder-decoder architecture, where a novel Shrinking Unit module and a retrieval-aided decoding mechanism are devised.
- In Defense of Clip-Based Video Relation Detection8
A Hierarchical Context Model (HCM) is proposed that enriches the object-based spatial context and relation-based temporal context based on clips and demonstrates that using clip tubelets can achieve superior performance compared to most video-based methods.
- I3: I ntent- I ntrospective Retrieval Conditioned on I nstructions7
I3 is introduced, a unified retrieval system that performs Intent-Introspective retrieval across various tasks, conditioned on Instructions without any task-specific training, embodying two key designs: progressive structure pruning and drawback extrapolation-based data refinement.
- 6
- Hierarchical Debiasing and Noisy Correction for Cross-domain Video Tube Retrieval6
The unsupervised domain adaptation model via Hierarchical dEbiAsing and noisy corRecTion (HEART) for cross-domain video tube retrieval is proposed, which contains two characteristic modules: Layered Feature Debiasing and Pseudo Label Refinement.
- 5
- Few-Shot Incremental Learning via Foreground Aggregation and Knowledge Transfer for Audio-Visual Semantic Segmentation5
Few-Shot Incremental Learning (FSIL) is introduced to the AVSS task, which seeks to seamlessly integrate new classes with limited incremental samples while preserving the knowledge of old classes.
- Grounding is All You Need? Dual Temporal Grounding for Video Dialog3
The Dual Temporal Grounding-enhanced Video Dialog model (DTGVD) is introduced, designed to bridge the gap between these two approaches to video dialog response generation and enables a more nuanced understanding of conversational dynamics.
- D-CAM: Learning Generalizable Weakly-Supervised Medical Image Segmentation from Domain-Invariant CAM3
- 3
- DGFamba: Learning Flow Factorized State Space for Visual Domain Generalization3
A novel Flow Factorized State Space model is proposed, dubbed as DGFamba, for visual domain generalization, which innovatively map the style-augmented and the original state embeddings by flow factorization to maintain domain consistency.
- Visuo-Tactile Class-Incremental Learning3
This article considers visuo-tactile correlations in VT-CIL and proposes a novel framework that efficiently incorporates the Visuo-Tactile Cross-Modal Pseudo-Label-Consistent (VT-CMPLC) constraint, Dual-Visuo-Tactile Exemplars (DVT-E), and the Dual-Visuo-Tactile-Compatible (DVT-C) constraint.
- 2
- The 1 st NIP@IR Workshop on New Interaction Paradigms for Information Retrieval in the Era of Generative AI1
This workshop will address key areas including system design, user engagement, and evaluation methodologies, and delve into broader concerns such as trust, transparency, and fairness, emphasizing the ethical implications of deploying generative AI in IR systems.
- 1
- Generation-Augmented Video Corpus Moment Retrieval–
This work proposes Video-GAR, a novel framework that reframes the conventional retrieval task from superficial matching to generative understanding, positing that the capability for query reconstruction evidences deep semantic comprehension.
- –
- –
- Introduction to the Special Issue on Deep Multimodal Generation and Retrieval–
This editorial introduces the Special Issue on Deep Multimodal Generation and Retrieval, providing a platform for researchers to share insights and advancements in this rapidly evolving domain and discusses ongoing challenges and future opportunities in this rapidly evolving domain.
- –
- The 2nd International Workshop on Deep Multi-modal Generation and Retrieval–
This workshop aims to encourage more explorations in Deep Multimodal Generation and Retrieval, providing a platform for researchers to share insights and advancements in this rapidly evolving domain.
- An Improved Multi-objective Visual Tracking Algorithm–
An improved real-time multi-target tracking algorithm based on the combination of improved YOLOV3-Tiny and deep-sort and using the effective combination of Kalman filter and Hungarian algorithm to prevent the phenomenon of over-fitting is proposed.
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-12. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.