Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Ping GuoSuggested from co-authorship
- Suggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works512 from public data
- Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images1,529
An end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image by progressively deforming an ellipsoid, leveraging perceptual features extracted from the input image.
- 1,485
- Two-dimensional materials for next-generation computing technologies1,044
The opportunities, progress and challenges of integrating two-dimensional materials with in-memory computing and transistor-based computing technologies, from the perspective of matrix and logic computing, are discussed.
- Evaluating bag-of-visual-words representations in scene classification962
This study provides an empirical basis for designing visual-word representations that are likely to produce superior classification performance and applies techniques used in text categorization to generate image representations that differ in the dimension, selection, and weighting of visual words.
- The THUMOS challenge on action recognition for videos “in the wild”918
The THUMOS benchmark is described in detail, the evaluation protocols used to quantify results in the two THUMOS tasks of action classification and temporal action detection are presented, and how well methods trained on trimmed videos generalize to untrimmed videos is evaluated.
- Towards optimal bag-of-features for object categorization and semantic video retrieval724
This paper evaluates various factors which govern the performance of Bag-of-features, and proposes a novel soft-weighting method to assess the significance of a visual word to an image and experimentally shows it can consistently offer better performance than other popular weighting methods.
- DSOD: Learning Deeply Supervised Object Detectors from Scratch646
Deeply Supervised Object Detector (DSOD), a framework that can learn object detectors from scratch following the single-shot detection (SSD) framework, and one of the key findings is that deep supervision, enabled by dense layer-wise connections, plays a critical role in learning a good detector.
- WildDeepfake609
WildDeepfake is a small dataset that can be used, in addition to existing datasets, to develop and test the effectiveness of deepfake detectors against real-world deepfakes, and proposes two (eg. 2D and 3D) Attention-based Deepfake Detection Networks (ADDNets) to leverage the attention masks on real/fake faces for improved detection.
- NAIS: Neural Attentive Item Similarity Model for Recommendation569
This work proposes a neural network model named Neural Attentive Item Similarity model (NAIS), which is the first attempt that designs neural network models for item-based CF, opening up new research possibilities for future developments of neural recommender systems.
- Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification479
This work proposes a hybrid deep learning framework for video classification, which is able to model static spatial information, short-term motion, as well as long-term temporal clues in the videos, and achieves very competitive performance on two popular and challenging benchmarks.
- Pose-Normalized Image Generation for Person Re-identification476
This work proposes a novel deep person image generation model for synthesizing realistic person images conditional on the pose based on a generative adversarial network designed specifically for pose normalization in re-id, thus termed pose-normalization GAN (PN-GAN).
- M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection436
A Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels, and learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block.
- Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks385
Through arming the DNN with better capability of harnessing both the feature and the class relationships, the proposed regularized DNN (rDNN) is more suitable for modeling video semantics.
- Consumer video understanding366
A new database, CCV, containing 9,317 web videos over 20 semantic categories, including events like "baseball" and "parade", scenes like "beach", and objects like "cat" is described and released, finding that humans are much better at understanding categories of nonrigid objects such as "cat", while current automatic techniques are relatively close to humans in recognizing categories that have distinctive background scenes or audio patterns.
- AdaViT: Adaptive Vision Transformers for Efficient Image Recognition355
AdaViT is introduced, an adaptive computation framework that learns to derive usage policies on which patches, self-attention heads and transformer blocks to use throughout the backbone on a per-input basis, aiming to improve inference efficiency of vision transformers with a minimal drop of accuracy for image recognition.
- Clean-Label Backdoor Attacks on Video Recognition Models335
This paper proposes the use of a universal adversarial trigger as the backdoor trigger to attack video recognition models, a situation where backdoor attacks are likely to be challenged by the above 4 strict conditions.
- Multi-scale Deep Learning Architectures for Person Re-identification319
A novel multi-scale deep learning model is proposed that is able to learn deep discriminative feature representations at different scales and automatically determine the most suitable scales for matching.
- A Survey on Video Diffusion Models318
This article presents a comprehensive review of video diffusion models in the AIGC era, and presents an overview of research on diffusion models in the video domain, categorizing the work into three key areas: video generation, video editing, and other video understanding tasks.
- Balanced Contrastive Learning for Long-Tailed Visual Recognition311
To correct the optimization behavior of SCL and further improve the performance of long-tailed visual recognition, a novel loss for balanced contrastive learning (BCL) is proposed that satisfies the condition of forming a regular simplex and assists the optimization of cross-entropy.
- 291
- Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation288
Hydra-MDP, a novel paradigm employing multiple teachers in a teacher-student model, which features a multi-head decoder to learn diverse trajectory candidates tailored to various evaluation metrics, achieves the 1-st place in the Navsim challenge.
- Recurrent Fusion Network for Image Captioning286
This paper proposes a novel recurrent fusion network (RFNet) for the image captioning task, which can exploit the interactions among the outputs of the image encoders and generate new compact and informative representations for the decoder.
- BEVT: BERT Pretraining of Video Transformers279
BEVT is introduced which decouples video representation learning into spatial representation learning and temporal dynamics learning and achieves state-of-the-art performance on three challenging video benchmarks where BEVT achieves very promising results.
- 270
- CNN-Based Chinese NER with Lexicon Rethinking265
This work proposes a faster alternative to Chinese NER: a convolutional neural network (CNN)-based method that incorporates lexicons using a rethinking mechanism that can model all the characters and potential words that match the sentence in parallel.
- Small footprint transistor architecture for photoswitching logic and in situ memory249
A transistor based on a two-dimensional material that can realize photoswitching logic (OR, AND) computing in a single cell and can change the logic behaviour is reported.
- Long-Term Cloth-Changing Person Re-identification248
This work introduces a shape embedding module as well as a cloth-elimination shape-distillation module aiming to eliminate the now unreliable clothing appearance features and focus on the body shape information.
- ObjectFormer for Image Manipulation Detection and Localization240
This paper proposes ObjectFormer, a method to detect and localize image manipulations that outperforms state-of-the-art tampering detection and localization methods, and uses a set of learnable object prototypes as mid-level representations to model the object-level consistencies among different regions.
- Cross-Domain Contrastive Learning for Unsupervised Domain Adaptation238
This work builds upon contrastive self-supervised learning to align features so as to reduce the domain discrepancy between training and testing sets, and introduces a simple yet effective framework CDCL, for domain alignment.
- VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks231
VLABench is an open-source benchmark for evaluating universal LCM task learning and provides high-quality training data collected via an automated framework incorporating heuristic skills and prior information to support the downstream finetuning.
- Trajectory-Based Modeling of Human Actions with Motion Reference Points224
This paper proposes a simple representation specifically aimed at the modeling of human action recognition in videos that operates on top of visual codewords derived from local patch trajectories, and therefore does not require accurate foreground-background separation, which is typically a necessary step to model object relationships.
- Semantic Proposal for Activity Localization in Videos via Sentence Query220
This paper proposes a novel Semantic Activity Proposal (SAP) which integrates the semantic information of sentence queries into the proposal generation process to get discriminative activity proposals and evaluates the algorithm on the TACoS dataset and the Charades-STA dataset.
- High-level event recognition in unconstrained videos202
While the existing solutions vary, common key modules are identified and detailed descriptions along with some insights for each are provided, including extraction and representation of low-level features across different modalities, classification strategies, fusion techniques, etc.
- 194
- 192
- Multi-Stream Multi-Class Fusion of Deep Networks for Video Classification188
A multi-stream framework that is able to exploit multimodal features that are more comprehensive than those previously attempted and produces significantly better results than the state of the arts on two popular benchmarks, 92.2% on UCF-101 (without using audio) and 84.9% on Columbia Consumer Videos.
- Recent Advances in Zero-Shot Recognition: Toward Data-Efficient Understanding of Visual Content186
This article provides a comprehensive review of existing zero-shot recognition techniques covering various aspects ranging from representations of models, data sets, and evaluation settings and highlights the limitations of existing approaches.
- Learning to Score Figure Skating Sport Videos181
A deep architecture that includes two complementary components, i.e., Self-Attentive L STM and Multi-scale Convolutional Skip LSTM can efficiently learn the local and global sequential information in each video.
- Secrets of RLHF in Large Language Models Part II: Reward Modeling179
This report introduces contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses, thereby improving model generalization and employs meta-learning to enable the reward model to maintain the ability to differentiate subtle differences in out-of-distribution samples.
- Towards Transferable Adversarial Attacks on Vision Transformers175
A dual attack framework is introduced, which contains a Pay No Attention (PNA) attack and a PatchOut attack, to improve the transferability of adversarial samples across different ViTs, and it is shown that skipping the gradients of attention during backpropagation can generate adversarial examples with high transferability.
- Black-box Adversarial Attacks on Video Recognition Models175
V-BAD is a promising new tool to evaluate and improve the robustness of video recognition models to black-box adversarial attacks, and can craft both untargeted and targeted attacks to fool two state-of-the-art deepVideo recognition models.
- 173
- 164
- Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders160
A suite of 256 SAEs, trained on each layer and sublayer of the Llama-3.1-8B-Base model, with 32K and 128K features are introduced and modifications to a state-of-the-art SAE variant, Top-K SAEs, are evaluated across multiple dimensions.
- Exploring Inter-feature and Inter-class Relationships with Deep Neural Networks for Video Classification154
A novel unified framework that jointly learns feature relationships and exploits the class relationships for improved video classification performance is proposed and demonstrates that the proposed framework exhibits superior performance over several state-of-the-art approaches.
- To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning153
This work introduces a fine-grained visual instruction dataset, LVIS-Instruct4V, which contains 220K visually aligned and context-aware instructions produced by prompting the powerful GPT-4V with images from LVIS, and demonstrates that high-quality visual instructional data could improve the performance of LLaVA-1.5, a state-of-the-art large multimodal model, across a wide spectrum of benchmarks.
- MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detection152
A novel framework with better utilization of the depth information and fine-grained cross-modal interaction between LiDAR and camera is proposed, which consists of two important components and achieves state-of-the-art results on both 3D object detection and tracking tasks without using test-time-augmentation and ensemble techniques.
- 152
- Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better150
The effectiveness of the RSLAD approach over existing adversarial training and distillation methods in improving the robustness of small models against state-of-the-art attacks including the AutoAttack is empirically demonstrated.
- 150
- DiffusionAD: Norm-Guided One-Step Denoising Diffusion for Anomaly Detection142
DiffusionAD, a novel anomaly detection pipeline comprising a reconstruction sub-network and a segmentation sub-network, is introduced and a norm-guided paradigm is proposed to integrate the benefits of multiple noise scales, enhancing the fidelity of reconstructions.
- SVFormer: Semi-supervised Video Transformer for Action Recognition142
SVFormer is introduced, which adopts a steady pseudo-labeling framework (i.e., EMA-Teacher) to cope with unlabeled video samples, and a novel augmentation strategy, Tube Token-Mix, tailored for video data where video clips are mixed via a mask with consistent masked tokens over the temporal axis.
- Weakly Supervised Dense Video Captioning142
This paper focuses on a novel and challenging vision task, dense video captioning, which aims to automatically describe a video clip with multiple informative and diverse caption sentences, and proposes lexical fully convolutional neural networks with weakly supervised multi-instance multi-label learning to weakly link video regions with lexical labels.
- Predicting Emotions in User-Generated Videos141
Results of a comprehensive set of experiments indicate that combining multiple types of features---such as the joint use of the audio and visual clues---is important, and attribute features such as those containing sentiment-level semantics are very effective.
- DB-LSTM: Densely-connected Bi-directional LSTM for human action recognition140
A Densely-connected Bi-directional LSTM (DB-LSTM) network is proposed to model the visual and temporal associations in both forward and backward directions to boost the effectiveness and robustness of modeling long-range action recognition.
- A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment139
This paper introduces the concept of "full-stack" safety to systematically consider safety issues throughout the entire process of LLM training, deployment, and eventual commercialization, and represents the first safety survey to encompass the entire lifecycle of LLMs.
- Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning135
This paper proposes masked video distillation (MVD), a simple yet effective two-stage masked feature modeling framework for video representation learning that demonstrates that video transformers pre-trained with spatial-temporal co-teaching outperform models distilled with a single teacher on a multitude of video datasets.
- Motion Guided Spatial Attention for Video Captioning134
The proposed MGSA exploits the motion between video frames by learning spatial attention from stacked optical flow images with a custom CNN and designed a Gated Attention Recurrent Unit (GARU) to adaptively incorporate previous attention maps.
- Prototypical Residual Networks for Anomaly Detection and Localization129
A framework called Prototypical Residual Network (PRN), which learns feature residuals of varying scales and sizes between anomalous and normal patterns to accurately reconstruct the segmentation maps of anomalous regions, is proposed.
- SimDA: Simple Diffusion Adapter for Efficient Video Generation128
A Simple Dif-fusion Adapter (SimDA) is proposed that fine-tunes only 24M out of I.IB parameters of a strong T2I model, adapting it to video generation in a parameter-efficient way and could minimize the training effort with extremely few tunable parameters for model adaptation.
- Evaluating Two-Stream CNN for Video Classification126
An in-depth study is conducted to investigate important implementation options that may affect the performance of deep nets on video classification, and very competitive results are attained on two popular video classification benchmarks.
- Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition124
A depth guided Adaptive Meta-Fusion Network for few-shot video recognition which is termed as AMeFu-Net, which is trained in the meta-learning way and demonstrates the effectiveness of the model on several action recognition benchmarks.
- 123
- Modeling Multimodal Clues in a Hybrid Deep Learning Framework for Video Classification123
A novel hybrid deep learning framework is introduced that integrates useful clues from multiple modalities, including static spatial appearance information, motion patterns within a short time window, audio information, as well as long-range temporal dynamics.
- Trainable Undersampling for Class-Imbalance Learning116
A meta-learning method built on the undersampling to parametrize the data sampler and train it to optimize the classification performance over the evaluation metric through reinforcement learning, which solves the non-differentiable optimization problem for training the data Sampler via reinforcement learning.
- Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos114
This work proposes a novel method for learning pairwise modality interactions in order to better exploit complementary information for each pair of modalities in videos and thus improve performances on both tasks.
- Image Block Augmentation for One-Shot Learning114
This paper proposes a Self-Training Jigsaw Augmentation (Self-Jig) method for one-shot learning, which provides a very useful way of directly generating massive amounts of training images for novel classes.
- AgentGym: Evolving Large Language Model-based Agents across Diverse Environments113
This paper proposes AgentGym, a new framework featuring a variety of environments and tasks for broad, real-time, uni-format, and concurrent agent exploration, and proposes a novel method, AgentEvol, to investigate the potential of agent self-evolution beyond previously seen data across tasks and environments.
- Columbia-UCF TRECVID2010 multimedia event detection: Combining multiple modalities, contextual concepts, and temporal matching113
It is important to combine strong complementary features from multiple modalities for multimedia event detection, and cross-frame matching is helpful in coping with temporal order variation.
- StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learning110
A novel model-agnostic meta Style Adversarial training (StyleAdv) method together with a novel style adversarial attack method is proposed for CD-FSL, which boosts the generalization ability for novel target datasets.
- CDistNet: Perceiving Multi-domain Character Distance for Robust Text Recognition108
This paper proposes a novel module called multi-domain character distance perception (MDCDP) to establish a visually and semantically related position embedding and develops CDistNet that stacks multiple MDCDPs to guide a gradually precise distance modeling.
- Reconstructive Neuron Pruning for Backdoor Defense107
This paper proposes a novel defense called Reconstructive Neuron Pruning (RNP) to expose and prune backdoor neurons via an unlearning and then recovering process and shows that such an asymmetric process on only a few clean samples can effectively expose andPrune the backdoor neurons implanted by a wide range of attacks.
- A Study of Multi-Task and Region-Wise Deep Learning for Food Ingredient Recognition106
An insightful analysis of three compelling issues in ingredient recognition, which involve recognition in either image-level or region level, pooling in either single or multiple image scales, and learning ineither single or multi-task manner are provided.
- Efficient Video Transformers with Spatial-Temporal Token Selection103
STTS is a token selection framework that dynamically selects a few informative tokens in both temporal and spatial dimensions conditioned on input video samples that achieves similar results while requiring 20% less computation.
- Object Detection from Scratch with Deep Supervision103
Deeply Supervised Object Detectors (DSOD), an object detection framework that can be trained from scratch, is proposed based on the single-shot detection framework (SSD), and achieves consistently better results than the state-of-the-art methods with much more compact models.
- Video event detection using motion relativity and visual relatedness102
A new motion feature, namely Expanded Relative Motion Histogram of Bag-of-Visual-Words (ERMH-BoW) to employ motion relativity and visual relatedness for event detection and to alleviate the visual word correlation problem in BoW is proposed.
- Generalized Meta-FDMixup: Cross-Domain Few-Shot Learning Guided by Labeled Target Data101
A novel Generalized Meta-learning based Feature-Disentangled Mixup network, namely GMeta-FDMixup is proposed to address Cross-Domain Few-Shot Learning (CD-FSL), and a novel feature disentanglement module is contributed to narrow the domain gap explicitly.
- Unsupervised Image-to-Image Translation with Stacked Cycle-Consistent Adversarial Networks101
Novel Stacked Cycle-Consistent Adversarial Networks (SCANs) are proposed by decomposing a single translation into multi-stage transformations, which not only boost the image translation quality but also enable higher resolution image-to-image translation in a coarse- to-fine fashion.
- 99
- Cross-Domain Sentiment Classification with Target Domain Specific Information98
This work proposes a method to simultaneously extract domain specific and invariant representations and train a classifier on each of the representation, respectively, and introduces a few target domain labeled data for learning domain-specific information.
- 97
- 94
- SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL93
This work presents SimpleAR, a vanilla autoregressive visual generation framework without complex architecure modifications that with only 0.5B parameters, can generate 1024x1024 resolution images with high fidelity, and achieve competitive results on challenging text-to-image benchmarks.
- DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs93
Surprisingly, this simple method greatly enhances the power of LMMs to model interactions among visual tokens across layers but with minimal additional cost, and DeepStack rivals closely to the counterparts that use the full context length.
- 93
- 89
- Heuristic Black-Box Adversarial Attacks on Video Recognition Models88
A heuristic black-box adversarial attack model that generates adversarial perturbations only on the selected frames and regions is proposed that can significantly reduce the computation cost and lead to more than 28% reduction in query numbers for the untargeted attack on both datasets.
- A relative similarity based method for interactive patient risk prediction86
An interactive patient risk prediction method is proposed, which actively queries medical experts with the relative similarity of patients, and makes several interesting discoveries including that querying relative similarities is effective in patient risk Prediction, and sometimes can even yield better prediction accuracy than asking for absolute questions.
- Columbia University/VIREO-CityU/IRIT TRECVID2008 High-Level Feature Extraction and Interactive Video Search86
A_CU-run6: local feature alone – average fusion of 3 SVM classification results for each concept using various feature representation choices.
- Sketch-BERT: Learning Sketch Bidirectional Encoder Representation From Transformers by Self-Supervised Learning of Sketch Gestalt85
This work presents a model of learning Sketch Bidirectional Encoder Representation from Transformer (Sketch-BERT), and generalizes BERT to sketch domain, with the novel proposed components and pre-training algorithms, including the newly designed sketch embedding networks, and the self-supervised learning of sketch gestalt.
- Understanding and Predicting Interestingness of Videos85
A pilot study on the understanding of human perception of video interestingness, and demonstrates a simple computational method to identify more interesting videos.
- 85
- MotionEditor: Editing Video Motion via Content-Aware Diffusion84
MotionEditor is the first to use diffusion models specifically for video motion editing, considering the origin dynamic background and camera movement, and incorporates a novel content-aware motion adapter into ControlNet to capture temporal motion correspondence.
- White-box Multimodal Jailbreaks Against Large Vision-Language Models82
A universal attack strategy that jointly attacks both text and image modalities to exploit a broader spectrum of vulnerability within VLMs and can effectively jailbreak MiniGPT-4 with a 96% success rate, highlighting the fragility of VLMs and the exigency for new alignment strategies.
- 82
- Social Anchor-Unit Graph Regularized Tensor Completion for Large-Scale Image Retagging81
This work proposes a novel Social anchor-Unit GrAph Regularized Tensor Completion (SUGAR-TC) method to efficiently refine the tags of social images, which is insensitive to the scale of data.
- 81
- VCDB: A Large-Scale Database for Partial Copy Detection in Videos81
A large-scale video copy database (VCDB) with over 100,000 Web videos, containing more than 9,000 copied segment pairs found through careful manual annotation is introduced.
- 81
- Brain state decoding for rapid image retrieval81
This paper proposes a new paradigm to explore and combine the strengths of machine vision and automatic content analytics, and demonstrates significant performance gains of the proposed system with multiple and diverse image classes over several data sets.
- Fast tracking of near-duplicate keyframes in broadcast domain with transitivity propagation80
The symmetric and transitive nature of near-duplicate for the effective detection and fast tracking of NDK pairs based upon the matching of local keypoints in frames is explored.
- ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System79
The vision for multimodal and versatile video understanding is presented and a prototype system, built upon a tracklet-centric paradigm, which treats tracklets as the basic video unit and employs various Video Foundation Models to annotate their properties e.g., appearance, motion, etc.
- Towards Bridging Event Captioner and Sentence Localizer for Weakly Supervised Dense Event Captioning78
- Visual Co-Occurrence Alignment Learning for Weakly-Supervised Video Moment Retrieval78
Through extensive experiments, it is demonstrated that the proposed visual co-occurrence alignment learning method outperforms the cross-modal alignment learning counterpart and achieves promising results for video moment retrieval.
- Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety76
This survey provides a systematic review of current safety research on large models, covering Vision Foundation Models, Large Language Models, Vision-Language Pre-training models, Vision-Language Models, Diffusion Models, and large-model-powered Agents, and presents a comprehensive taxonomy of safety threats to these models.
- 76
- Adaptively Weighted Multi-task Deep Network for Person Attribute Classification75
A novel validation loss trend algorithm is proposed to dynamically and adaptively update the weight for learning each task in the training process to solve the problems of multi-task deep convolutional neural network for person attribute analysis.
- AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning74
AgentGym-RL, a new framework to train LLM agents for multi-turn interactive decision-making through RL, features a modular and decoupled architecture, ensuring high flexibility and extensibility, and proposes ScalingInter-RL, a training approach designed for exploration-exploitation balance and stable RL optimization.
- CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation74
The Layout Designer is introduced, which taps into the potential of large language models in layout planning, transforming them into experts in layout generation and optimization and form CreatiLayout-a systematic solution that integrates the layout model, dataset, and planner for creative layout-to-image generation.
- MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes74
This work presents MOSEv2, a significantly more challenging dataset designed to further advance VOS methods under real-world conditions, and benchmark 20 representative VOS methods under 5 different settings, highlighting that current VOS methods still fall short under real-world complexities.
- 74
- A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models73
A comprehensive review of multimodal-guided image editing techniques that leverage T2I diffusion models, and proposes a unified framework to formalize the editing process, categorizing it into two primary algorithm families.
- Enhancing the Self-Universality for Transferable Targeted Attacks73
A novel transfer-based targeted attack method that optimizes the adversarial perturbations without any extra training efforts for auxiliary networks on training data is proposed, named as Self-Universality (SU) attack.
- Implicit Temporal Modeling with Learnable Alignment for Video Recognition72
This paper proposes a novel Implicit Learnable Alignment (ILA) method, which minimizes the temporal modeling effort while achieving incredibly high performance and allows eliminating the costly or insufficient temporal self-attention in video.
- 72
- Fudan-Huawei at MediaEval 2015: Detecting Violent Scenes and Affective Impact in Movies with Deep Learning72
This work trains a Convolutional Neural Network model with a subset of ImageNet classes selected particularly for violence detection, and adopts a specially designed two-stream CNN framework to extract features on both static frames and motion optical ows.
- MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation69
This paper proposes a large-scale multi-modal dataset for referring motion expression video segmentation, focusing on segmenting and tracking target objects in videos based on language description of objects’ motions, and introduces MeViS, a dataset containing 33,072 human-annotated motion expressions in both text and audio.
- 69
- 69
- In-memory computing to break the memory wall*68
The state-of-the-art memory device technologies potential for in-memory computing are reviewed, their versatile applications in neural network, stochastic generation, and hybrid precision digital computing are summarized, and promising solutions for unprecedented computing tasks are discussed.
- The MediaEval 2014 Affect Task: Violent Scenes Detection67
The MediaEval 2014 Affect Task: Violent Scenes Detection, which is running for the fourth year, originates from a use case at Technicolorthat aims to help users find suitable contents from a movie database.
- Semantic context transfer across heterogeneous sources for domain adaptive video search67
Extensive experiments on 120 textual queries over TRECVID 2005-2008 data sets demonstrate the effectiveness of semantic context transfer for domain adaptive video search, and exploit the context information associated with Flickr images to estimate query-detector similarity.
- 66
- 65
- Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation64
This paper proposes a framework called Reuse and Diffuse to produce more frames following the frames already generated by an LDM, and proposes a set of strategies for composing video-text data that involve diverse content from multiple existing datasets including video datasets for action recognition and image-text datasets.
- Cross-Modal Transferable Adversarial Attacks from Images to Videos64
This paper proposes a simple yet effective cross-modal attack method, named as Image To Video (I2V) attack, which generates adversarial frames by minimizing the cosine similarity between features of pretrained image models from adversarial and benign examples, then combines the generated adversarial Frames to perform black-box attacks on video recognition models.
- FoodLMM: A Versatile Food Assistant Using Large Multi-Modal Model63
This paper proposes FoodLMM, a versatile food assistant based on LMMs with various capabilities, including food recognition, ingredient recognition, recipe generation, nutrition estimation, food segmentation, and multi-round conversation, to facilitate FoodLMM in dealing with tasks beyond pure text output.
- Look Before You Match: Instance Understanding Matters in Video Object Segmentation61
It is argued that instance understanding matters in VOS, and integrating it with memory-based matching can enjoy the synergy, which is intuitively sensible from the definition of VOS task, i.e., identifying and segmenting object instances within the video.
- Multi-task Deep Neural Network for Joint Face Recognition and Facial Attribute Prediction61
This paper for the first time, advocate a multi-task deep neural network for jointly learning face recognition and facial attribute prediction tasks and can achieve the best performance on the small protocol among all the publicly released methods.
- Visual word proximity and linguistics for semantic video indexing and near-duplicate retrieval61
Experimental results indicate that soft-weighting is superior to other popular weighting schemes such as term frequency (TF) weighting in large-scale video database and CEMD shows excellent performance compared to cosine similarity in near-duplicate retrieval.
- ResFormer: Scaling ViTs with Multi-Resolution Training60
ResFormer, a framework that is built upon the seminal idea of multi-resolution training for improved performance on a wide spectrum of, mostly unseen, testing resolutions, is introduced and can be easily extended to semantic segmentation, object detection and video action recognition.
- Semantic Feature Augmentation in Few-shot Learning.60
A novel auto-encoder network dual TriNet, which projects multi-layer visual features of deep CNNs into the semantic space, and the augmented instance representation is projected back into the image feature spaces by the decoder TriNet.
- Keyframe Retrieval by Keypoints: Can Point-to-Point Matching Help?60
It is demonstrated that P2P constraint can significantly boost the performance of NDK retrieval, while showing competitive accuracy in concept classification of broadcast domain.
- AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments57
A GENT G YM is proposed, a framework featuring 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction that takes the initial step to develop LLM-based agents that can handle diverse tasks via methods like self-improvement or reinforcement learning.
- 57
- SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognition56
SVTRv2, a CTC model endowed with the ability to handle text irregularities and model linguistic context, is proposed, and a feature rearrangement module is introduced to ensure that visual features accommodate the requirement of CTC, thus alleviating the alignment puzzle.
- 56
- 55
- Semi-supervised Vision Transformers55
A joint semi-supervised learning framework, Semiformer, which contains a transformer stream, a convolutional stream and a carefully designed fusion module for knowledge sharing between these streams, which is compatible with most modern transformer and Convolutional neural architectures.
- Deep Learning for Video Captioning: A Review55
The problem of video captioning is formulated, state-of-the-art methods categorized by their emphasis on vision or language are reviewed, and followed by a summary of standard datasets and representative approaches are reviewed.
- TC-Net for iSBIR55
A Triplet Classification Network (TC-Net) for iSBIR is presented which is composed of two major components: triplet Siamese network, and auxiliary classification loss which can break the limitations existed in previous works.
- OmniTracker: Unifying Visual Object Tracking by Tracking-With-Detection54
A novel tracking-with-detection paradigm is proposed, where tracking supplements appearance priors for detection and detection provides tracking with candidate bounding boxes for the association, and a unified tracking model, OmniTracker, is presented to resolve all the tracking tasks with a fully shared network architecture, model weights, and inference pipeline.
- MOSS: An Open Conversational Large Language Model54
The development of MOSS is reported, an open-sourced conversational LLM that contains 16 B parameters and can perform a variety of instructions in multi-turn interactions with humans and establishes a complete technical roadmap for large language models from pre-training, supervised fine-tuning to alignment.
- Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision54
This paper delves into a two-player paradigm that separates the roles of reasoning and critique models, where the critique model provides step-level feedback to supervise the reasoning (actor) model during both test-time and train-time, and proposes a critique-in-the-loop self-improvement method.
- Video Moment Retrieval from Text Queries via Single Frame Annotation54
This paper proposes a new paradigm called ViGA, which cuts the input video into clips and contrasts between clips and queries, and achieves better results than the state-of-the-art weakly supervised methods by a large margin, even comparable to fully supervised methods in some cases.
- VideoLT: Large-scale Long-tailed Video Recognition54
FrameStack is proposed, a simple yet effective method for long-tailed video recognition that performs sampling at the frame-level in order to balance class distributions, and the sampling ratio is dynamically determined using knowledge derived from the network during training.
- Hierarchical Visual-Textual Graph for Temporal Activity Localization via Language54
A novel TALL method which builds a Hierarchical Visual-Textual Graph to model interactions between the objects and words as well as among the objects to jointly understand the video contents and the language is proposed.
- Matching User Photos to Online Products with Robust Deep Features54
Experiments on two challenging real-world datasets demonstrate that both the robust contrastive loss and the multi-task fine-tuning approach are effective, leading to very promising results with a time cost suitable for real-time retrieval.
- Towards textually describing complex video contents with audio-visual concept classifiers54
This paper explores the technical feasibility of the challenging issue of precisely recounting video contents using cutting-edge automatic recognition techniques and finds that the state-of-the-art visual and audio concept classification, although far from perfect, is able to provide very useful clues indicating what is happening in the videos.
- Video Emotion Recognition with Transferred Deep Feature Encodings53
A novel auxiliary Image Transfer Encoding process is proposed to efficiently encode and generate video representation and thoroughly investigate different configurations of convolutional neural networks.
- StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation52
StableAvatar is presented, the first end-to-end video diffusion transformer that synthesizes infinite-length high-quality videos without post-processing and introduces a novel Time-step-aware Audio Adapter that prevents error accumulation via time-step-aware modulation.
- Boosting the Transferability of Video Adversarial Examples via Temporal Translation51
A temporal translation attack method, which optimizes the adversarial perturbations over a set of temporal translated video clips to boost the transferability of video adversarial examples for black-box attacks on video recognition models.
- Emotion in Context51
The proposed framework can realize recognition in real-time and achieves state-of-the-art performance on two challenging emotion recognition benchmarks, 50.6 on VideoEmotion and 51.8 on Ekman.
- 50
- 50
- Unlearnable Clusters: Towards Label-Agnostic Unlearnable Examples49
This work presents a novel technique called Unlearnable Clusters (UCs) to generate label-agnostic unlearnable examples with cluster-wise perturbations and proposes to leverage Vision-and-Language Pre-trained Models (VLPMs) like CLIP as the surrogate model to improve the transferability of the crafted UCs to diverse domains.
- Lost in binarization48
A novel approach that facilitates query-adaptive ranking for the images with equal Hamming distance by firstly offline learning bit weights of the binary codes for a diverse set of predefined semantic concept classes and then rapidly computed by evaluating the proximity between a query and the concept categories.
- Context Perception Parallel Decoder for Scene Text Recognition47
A Context Perception Parallel Decoder (CPPD) is proposed to perceive the related context and predict the character sequence in a PD pass, which benefits the decoder to focus accurately on characters with the attention mechanism, thereby improving the recognition accuracy.
- Learning Hybrid Part Filters for Scene Recognition47
It is demonstrated that Hybrid-Parts outperforms recent state-of-the-arts benchmarks, and combining it with standard low-level features such as the GIST descriptor can lead to further improvements.
- Harnessing Synthesized Abstraction Images to Improve Facial Attribute Recognition46
This paper builds a dual-path facial attribute recognition network to utilize features from the original face images and facial abstraction images to improve facial attributes recognition, especially the attributes located on small face regions.
- Discovering joint audio–visual codewords for video event detection46
This paper proposes a new representation—called bi-modal words—to explore representative joint audio–visual patterns, and finds that average pooling is particularly suitable for bi-modal representation, and using multiple kernel learning to combine multi-modal representations at various granularities is helpful.
- Embodied One-Shot Video Recognition45
A novel setting is introduced, termed as embodied agents based one- shot learning, which leverages synthetic videos produced in a virtual environment to understand realistic videos of target classes and proposes a general video segment augmentation method, which significantly facilitates a variety of one-shot learning tasks.
- OmniViD: A Generative Framework for Universal Video Understanding44
This work seeks to unify the output space of video understanding tasks by using languages as labels and additionally introducing time and box tokens, which enables a variety of video tasks to be formulated as video-grounded token generation.
- 44
- A Coarse-to-Fine Framework for Resource Efficient Video Recognition43
LiteEval is a coarse-to-fine framework that dynamically allocates computation on a per-video basis, and can be deployed in both online and offline settings, and adaptively determines on the fly when to read in more discriminative yet computationally expensive features.
- Instruction-Guided Scene Text Recognition42
A novel instruction-guided scene text recognition (IGTR) paradigm that formulates STR as an instruction learning problem and understands text images by predicting character attributes, e.g., character frequency, position, etc is proposed.
- Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data42
Open-VCLIP++ is presented, a simple yet effective framework that adapts CLIP to a strong zero-shot video classifier, capable of identifying novel actions and events during testing, and introduces Interpolated Weight Optimization, a technique that leverages the advantages of weight interpolation during both training and testing.
- Video Relation Detection via Multiple Hypothesis Association42
This paper proposes a novel relation association method called Multiple Hypothesis Association (MHA), which maintains multiple possible relation hypothesis during the association process in order to tolerate and handle the inaccurate or missing problem in the former steps and generate more accurate video relations.
- Non-local NetVLAD Encoding for Video Classification42
This paper describes the solution for the YouTube-8M video understanding challenge organized by Google AI and fuses six different sub-models into one single computational graph, which is categorized into three families, the most effective family is the model with non-local operations following the NetVLAD encoding.
- CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design41
CreatiDesign, a systematic solution for automated graphic design covering both model architecture and dataset construction, designs a unified multi-condition driven architecture that enables flexible and precise integration of heterogeneous design elements with minimal architectural modifications to the base diffusion model.
- 41
- Embodied AI: From LLMs to World Models [Feature]40
This paper comprehensively explores the literature in embodied AI from basics to advances, covering both LLM driven and WM driven works, and meticulously delineates their indispensable roles in end-to-end embodied cognition and physical laws-driven embodied interactions.
- Deep Ranking for Image Zero-Shot Multi-Label Classification40
This work investigates and formalizes a flexible framework consisting of two components, i.e., visual-semantic embedding and zero-shot multi-label prediction, and presents a deep regression model to project the visual features into the semantic space, which explicitly exploits the correlations in the intermediate semantic layer of word vectors and makes label prediction possible.
- 40
- 40
- 40
- Ideator: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves39
This paper proposes IDEATOR, a novel jailbreak method that autonomously generates malicious image-text pairs for black-box jailbreak attacks and introduces the VLJailbreakBench, a safety benchmark comprising 3,654 multimodal jailbreak samples.
- Wave-SAN: Wavelet based Style Augmentation Network for Cross-Domain Few-Shot Learning39
This paper studies the problem of CD-FSL by spanning the style distributions of the source dataset, and proposes a novel Style Augmentation (StyleAug) module and a Self-Supervised Learning (SSL) module to ensure the predictions of style-augmented images are semantically similar to the unchanged ones.
- Multi-modal cooking workflow construction for food recipes39
MM-ReS is built, the first large-scale dataset for cooking workflow construction, consisting of 9,850 recipes with human-labeled workflow graphs and a neural encoder-decoder model that utilizes both visual and textual information to construct the cooking workflow, which achieved over 20% performance gain over existing hand-crafted baselines.
- Recurrent Memory Reasoning Network for Expert Finding in Community Question Answering39
A novel \textslrecurrent memory reasoning network (RMRN) is proposed to perform expert finding, which focuses on different parts of a question, and accordingly retrieves information from the histories of the candidate expert.
- Sketch Recognition with Deep Visual-Sequential Fusion Model39
A deep end-to-end network for sketch recognition, named Deep Visual-Sequential Fusion model (DVSF) is proposed to model the visual and sequential patterns of the strokes, which outperforms the state-of-the-art approaches.
- 39
- 38
- Locate Before Answering: Answer Guided Question Localization for Video Question Answering37
LocAns is proposed, a novel approach that integrates a question localization module and an answer prediction module into an end-to-end model and achieves state-of-the-art performance on three modern long-term VideoQA datasets, NExT-QA, ActivityNet-QA, and AGQA.
- FT-TDR: Frequency-Guided Transformer and Top-Down Refinement Network for Blind Face Inpainting37
A transformer-based network is used to detect the corrupted regions to be inpainted as masks by modeling the relation among different patches and a top-down refinement network is proposed to hierarchically restore features at different levels and generate contents that are semantically consistent with the unmasked face regions.
- 37
- Dense Dilated Network for Few Shot Action Recognition36
A novel neural network architecture to simultaneously capture local and long-term spatial temporal information is introduced with the blocks being composed of densely-connected dilated convolutions layers.
- Beauty is here36
This paper demonstrates a computational approach to automatically evaluate the aesthetics of videos, with particular emphasis on identifying beautiful scenes, and shows that combining multiple features is helpful and very promising results can be obtained using the noisy but annotation-free training data.
- BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks35
This paper proposes a novel blue-team method BlueSuffix that defends target VLMs against jailbreak attacks without compromising its performance under black-box setting and empirically shows that BlueSuffix outperforms the baseline defenses by a significant margin.
- MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion35
The proposed MotionFollower is a lightweight score-guided diffusion model for video motion editing that achieves an approximately 80% reduction in GPU memory while delivering superior motion editing performance and exclusively supporting large camera movements and actions.
- A Multi-Task Neural Approach for Emotion Attribution, Classification, and Summarization35
A new neural approach is proposed, Bi-stream Emotion Attribution-Classification Network (BEAC-Net), to solve three related emotion analysis tasks: emotion recognition, emotion attribution, and emotion-oriented summarization, in a single integrated framework.
- Binary Optimized Hashing35
Binary optimized hashing (BOH) is proposed, in which it is proved that if the loss function is Lipschitz continuous, the binary optimization problem can be relaxed to a bound-constrained continuous optimization problem.
- Human2Robot: Learning Robot Actions from Paired Human-Robot Videos34
This work introduces H&R, a novel third-person dataset containing 2,600 episodes of precisely synchronized human and robot motions, collected using a VR teleoperation system and presents Human2Robot, a framework designed to leverage this data to learn a rich and implicit representation of robot dynamics.
- 34
- 34
- Joint audio-visual bi-modal codewords for video event detection34
This paper proposes a joint audio-visual bi- modal representation, called bi-modal words, and finds that average pooling is the most suitable strategy for bi-Modal feature generation.
- SUPER34
It is shown that, by choosing suitable features, classifiers, and fusion strategies, recognition speed can be greatly improved with minor performance degradation, and the suggested SUPER framework is 220-fold faster than the baseline approach with merely 3.8% drop in recognition accuracy.
- On the sampling of web images for learning visual concept classifiers34
The impact of exploiting the tagged images towards concept learning is studied, investigating the issue of how the quality of pseudo training images affects concept detection performance, and a simple approach is proposed, named semantic field, for predicting the relevance between a target concept and the tag list associated with the images.
- 34
- 34
- 33
- Detection Hub: Unifying Object Detection Datasets via Query Adaptation on Language Embedding33
A new design is addressed by a new design that is dataset-aware and category-aligned that mitigates the dataset inconsistency but also provides coherent guidance for the detector to learn across multiple datasets.
- Hot Topic-Aware Retweet Prediction with Masked Self-attentive Model33
This work proposes a novel masked self-attentive model that incorporates the posting histories of users with external memory and utilize a hierarchical attention mechanism to construct the users' interests and can be jointly hot-topic aware and user interests aware to make a final prediction.
- Which Looks Like Which: Exploring Inter-class Relationships in Fine-Grained Visual Categorization33
This paper treats the training of each category classifier as a single learning task, and formulates a generic multiple task learning (MTL) framework to train multiple classifiers simultaneously, which is able to automatically discover both clusters of similar categories and outliers.
- 32
- 32
- Two-stage Visual Cues Enhancement Network for Referring Image Segmentation31
Through the two-stage enhancement, the proposed TV-Net enjoys better performances in learning fine-grained matching behaviors between the natural language expression and image, especially when the visual information of the referent is inadequate, thus produces better segmentation results.
- 30
- Attacking Video Recognition Models with Bullet-Screen Comments30
This paper introduces a novel adversarial attack, the bullet-screen comment (BSC) attack, which attacks video recognition models with BSCs, generated with a Reinforcement Learning (RL) framework.
- TGDM: Target Guided Dynamic Mixup for Cross-Domain Few-Shot Learning30
This paper proposes a novel target guided dynamic mixup (TGDM) framework that leverages the target data to guide the generation of mixed images via dynamic mix up and introduces an intermediate domain generated by mixing images in the source and the target domain.
- 30
- Take Goods from Shelves: A Dataset for Class-Incremental Object Detection30
A new hierarchical large-scale object detection dataset, called Take Goods from Shelves (TGFS), containing 38K images of 24 fine-grained and 3 coarse classes is proposed and a preliminary method for solving the goods-adding problem, called Faster R-CNN Class-incremental Object Detector (FCIOD), is described and evaluated.
- Aid: Adapting Image2video Diffusion Models for Instruction-Guided Video Prediction29
The Multi-Modal Large Language Model (MLLM) is introduced to predict future video states based on initial frames and text instructions and a dual query transformer (DQFormer) architecture is designed, which integrates the instructions and frames into the conditional embeddings for future frame prediction.
- 29
- 29
- MouSi: Poly-Visual-Expert Vision-Language Models28
This paper proposes the use of ensemble experts technique to synergizes the capabilities of individual visual encoders, including those skilled in image-text matching, OCR, image segmentation, etc, while bridging the gap between image encoders and pre-trained LLMs.
- Learning to Generate and Edit Hairstyles28
A new large-scale hairstyle dataset called Hairstyle30k is contributed, which is composed of 30k images containing 64 different types of hairstyles and a novel GAN model termed Hairstyles GAN (H-GAN) which can be learned efficiently is proposed.
- Exploring inter-concept relationship with context space for semantic video indexing28
This paper proposes the construction of a context space and the exploration of the space for CBCF, under which the significance of a concept/detector can be adapted when applied to a target domain different from where the detector is being developed.
- MagDiff: Multi-alignment Diffusion for High-Fidelity Video Generation and Editing27
This work is the first to propose a unified Multi-alignment Diffusion, dubbed as MagDiff, for both tasks of high-fidelity video generation and editing, and Experimental results on four benchmarks suggest that the method outperforms the previous method on each task.
- Doubly Abductive Counterfactual Inference for Text-Based Image Editing27
A Doubly Abductive Counterfactual inference framework that can support a wide spectrum of user editing intents, including addition, removal, manipulation, replacement, style transfer, and facial change, which are extensively validated in both qualitative and quantitative evaluations.
- ME-D2N: Multi-Expert Domain Decompositional Network for Cross-Domain Few-Shot Learning27
To solve the data imbalance problem between the source data with sufficient examples and the auxiliary target data with limited examples, a novel Multi-Expert Domain Decompositional Network (ME-D2N) is built under the umbrella of multi-expert learning.
- 26
- Experimenting VIREO-374: Bag-of-Visual-Words and Visual-Based Ontology for Semantic Video Indexing and search.26
This paper presents the approaches and results of high-level feature extraction and automatic video search in TRECVID-2007, and explores the upper limit of bag-of-visualwords (BoW) approach based upon local appearance features.
- Shortcuts Everywhere and Nowhere: Exploring Multi-Trigger Backdoor Attacks25
This study explores the concept of Multi-Trigger Backdoor Attacks (MTBAs), where multiple adversaries leverage different types of triggers to poison the same dataset, and demonstrates that multiple triggers can coexist, overwrite, or cross-activate one another.
- LRANet: Towards Accurate and Efficient Scene Text Detection with Low-Rank Approximation Network25
This work proposes a novel parameterized text shape method based on low-rank approximation that achieves consistency, compactness, simplicity, and robustness in shape representation, and implements an accurate and efficient arbitrary-shaped text detector named LRANet.
- VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models25
This work is the first to propose Video Instruction Diffusion (VIDiff), a unified foundation model designed for a wide range of video tasks, which can edit and translate the desired results within seconds based on user instructions.
- Beyond Semantic Search: What You Observe May Not Be What You Think.25
This paper presents the approaches and results of the four TRECVID 2008 tasks the authors participated in: high-level feature extraction, automatic video search, video copy detection, and rushes summarization.
- Vocabulary-Informed Zero-Shot and Open-Set Learning24
A weighted maximum margin framework for semantic manifold-based recognition that incorporates distance constraints from (both supervised and unsupervised) vocabulary atoms that ensures that labeled samples are projected closer to their correct prototypes, in the embedding space, than to others.
- 24
- RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models23
To mitigate the conflicts arising from multi-task supervision during fine-tuning of LMMs, a novel Linear Rectification Mixture of Diverse Experts (RoDE) approach is introduced, which utilizes a diverse array of experts to address tasks of varying complexity, thereby facilitating the coordination of trainable parameters.
- 4D Gaussian Splatting: Modeling Dynamic Scenes with Native 4D Primitives23
This paper reformulates the reconstruction of a time-varying 3D scene as approximating its underlying spatiotemporal 4D volume by optimizing a collection of native 4D primitives, i.e., 4D Gaussians, with explicit geometry and appearance modeling.
- Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection23
This work proposes RichSem, a simple but effective method, which is robust to learn rich semantics from coarse locations without the need of accurate bounding boxes and achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors.
- 23
- Regional Gating Neural Networks for Multi-label Image Classification23
RGNN is an end-to-end deep learning framework that can automatically select contextual region features with specially designed gate units, which are then fused for classification and results show that RGNN is superior to existing state-of-the-art methods.
- Bag-of-visual-words expansion using visual relatedness for video indexing23
A novel BoW expansion method is proposed to alleviate the effect of visual word correlation problem by diffusing the weights of visual words in BoW based on visual word relatedness, which is rigorously defined within a visual ontology.
- 22
- Fudan-NJUST at MediaEval 2014: Violent Scenes Detection Using Deep Neural Networks.22
Results indicate that using multiple features is still very helpful, and more importantly, the proposed regularized DNN offers significantly better results than the popular SVM.
- EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmark21
EgoNight, the first comprehensive benchmark for nighttime egocentric vision, with visual question answering (VQA) as the core task, is presented, and EgoNight-VQA provides a strong foundation for advancing application-driven egocentric vision research and for developing models that generalize across illumination domains.
- Identity-Driven Multimedia Forgery Detection via Reference Assistance21
This work proposes an identity-driven multimedia forgery dataset, IDForge, which contains 249,138 video shots sourced from 324 wild videos of 54 celebrities collected from the Internet, and proposes the Reference-assisted Multimodal Forgery Detection Network (R-MFDN), aiming at the detection of deepfake videos.
- On the Importance of Spatial Relations for Few-shot Action Recognition21
It is found that the spatial misalignment between objects also occurs in videos, notably more common than the temporal inconsistency, and a novel Spatial Alignment Cross Transformer (SA-CT) which learns to re-adjust the spatial relations and incorporates the temporal information is contributed.
- GCMA: Generative Cross-Modal Transferable Adversarial Attacks from Images to Videos21
This paper investigates cross-modal transferable attacks from image domains to video domains with the generator-oriented approach, i.e., crafting adversarial perturbations for each frame of video clips with the perturbation generator trained in the ImageNet domain to attack target video models.
- Self-Supervised Learning for Semi-Supervised Temporal Language Grounding21
The results demonstrate that the proposed proposed Temporal Language Grounding can achieve competitive performance compared to fully-supervised state-of-the-art methods while only requiring a small portion of temporal annotations.
- Flexible multi-task learning with latent task grouping21
A flexible multi-task learning framework to identify latent grouping structures under agnostic settings, where the prior of the latent subspace is unknown to the learner, and provides proofs of theoretical guarantee on learning performance.
- 20
- Imbalanced Gradients: A New Cause of Overestimated Adversarial Robustness20
12 state-of-the-art defense models are examined, and it is found that models exploiting label smoothing easily cause imbalanced gradients, and on which attacks can decrease their PGD robustness by over 23%.
- 20
- 19
- From Canteen Food to Daily Meals: Generalizing Food Recognition to More Practical Scenarios19
Two new benchmarks, namely DailyFood-172 and DailyFood-16, are presented, specifically designed to curate food images from everyday meals, to evaluate the transferability of approaches from the well-curated food image domain to the everyday-life food image domain.
- MRN: Multiplexed Routing Network for Incremental Multilingual Text Recognition19
A Multiplexed Routing Network (MRN) is proposed, which outperforms existing general-purpose IL methods by large margins, with average accuracy improvements ranging from 10.3% to 35.8% under different settings.
- Visual Relations Augmented Cross-modal Retrieval19
A novel framework that enhanced visual representation with relations between components is introduced, and visual relations are represented by the scene graph extracted from an image, then encoded by the graph convolutional neural networks for learning visual relational features.
- Sparse Temporal Causal Convolution for Efficient Action Modeling19
This work demonstrates that the explicit temporal feature interactions can be largely reduced without any degradation in spatio-temporal convolutional networks, and uses causal convolutions for temporal feature learning.
- Exploiting Objects with LSTMs for Video Categorization19
This paper proposes to leverage high-level semantic features to open the "black box" of the state-of-the-art temporal model, Long Short Term Memory (LSTM), with an aim to understand what is learned.
- 18
- 18
- Frame-Transformer Emotion Classification Network18
A new architecture--Frame-Transformer Emotion Classification Network (FT-EC-net) is proposed to solve three highly correlated emotion analysis tasks: emotion recognition, emotion attribution and emotion-oriented summarization.
- 18
- Fudan at MediaEval 2013: Violent Scenes Detection Using Motion Features and Part-Level Attributes.18
It is concluded that a successful violent scenes detection system should use truly multimodal features, ranging from motion-based to static visual descriptors, as well as audio and attribute features.
- 17
- 17
- Generalizing Face Forgery Detection via Uncertainty Learning17
This paper extends the Transformer model in a probabilistic manner by modeling dependencies between patches as Gaussian random variables and introduces a Patch Selection Module that can efficiently and accurately identify discriminative regions with high-uncertainty information, which are further utilized for final classification.
- Mix-DANN and Dynamic-Modal-Distillation for Video Domain Adaptation17
This paper proposes Mix-Domain-Adversarial Neural Network and Dynamic-Modal-Distillation (MD-DMD), a novel multi-modal adversarial learning framework for unsupervised video domain adaptation that incorporates the temporal information between source and target domains, as well as the diversity of adaptability between modalities.
- Comp-GAN17
A novel end-to-end Compositional Generative Adversarial Network that is able to synthesize new face images with specified poses and desired facial expressions and which can dynamically change the expression and pose of faces according to the input images while keeping the identity information.
- DeepProduct17
Extensive experiments on challenging real-world datasets demonstrate that both the robust contrastive loss and the multitask fine-tuning scheme are effective, leading to very promising results with a time cost suitable for mobile product search scenarios.
- Name-Face Association in Web Videos: A Large-Scale Dataset, Baselines, and Open Issues17
A Web video dataset of celebrities, named WebV-Cele, for name-face association, consisting of 75 073 Internet videos of over 4 000 hours, covering 2 427 celebrities and 649 001 faces is introduced, believed to be the most comprehensive dataset for this problem.
- Real-time summarization of user-generated videos based on semantic recognition17
This paper introduces an approach for UGV summarization based on semantic recognition and shows that this approach clearly outperforms several alternative methods and is highly efficient.
- Autonomous trust construction in multi-agent systems—a graph theory methodology17
The paper presents an autonomous trust construction model based on graph theory methodology that adopts the graph to describe the trust information, and uses the graph combination and path searching to construct the trust relation.
- 16
- Look Before You Decide: Prompting Active Deduction of MLLMs for Assumptive Reasoning16
This paper curates a Multimodal Assumptive Assumptive Reas oning Benchmark (MARS-Bench) and finds that most prevalent MLLMs can be easily fooled by the introduction of a presupposition into the question, whereas such presuppositions appear naive to human reasoning.
- Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentation16
OmniAVS is proposed, a new dataset containing 2,104 videos and 61,095 multimodal referring expressions and Omnimodal Instructed Segmentation Assistant (OISA) is introduced, to address the challenges of multimodal reasoning and finegrained understanding of audiovisual content in OmniAVS.
- Highly Transferable Diffusion-based Unrestricted Adversarial Attack on Pre-trained Vision-Language Models16
A Multimodal Diffusion-based Attack (MDA), which conducts adversarial attacks against VLMs using Stable Diffusion, demonstrating that the adversarial examples generated by MDA are highly transferable across different VLMs on different downstream tasks, surpassing state-of-the-art methods by a large margin.
- GeoGS3D: Single-view 3D Reconstruction via Geometric-aware Diffusion Model and Gaussian Splatting16
FDGaussian is introduced, a novel two-stage framework for single-image 3D reconstruction that generates images with high consistency across different views and reconstructs high-quality 3D objects, both qualitatively and quantitatively.
- Imbalanced gradients: a subtle cause of overestimated adversarial robustness16
A margin decomposition (MD) attack that decomposes a margin loss into individual terms and then explores the attackability of these terms separately via a two-stage process to exploit imbalanced gradients.
- 16
- 16
- STNMamba: Mamba-Based Spatial-Temporal Normality Learning for Video Anomaly Detection15
This work proposes a lightweight and effective Mamba-based network named STNMamba, which incorporates carefully designed Mamba modules to enhance the learning of spatial-temporal normality and achieves competitive performance with fewer parameters and lower computational costs than existing methods.
- REDUCIO! Generating 1K Video Within 16 Seconds Using Extremely Compressed Motion Latents15
It is argued that videos contain significantly more redundant information than images, allowing them to be encoded with very few motion latents, allowing them to be encoded with very few motion latents in Reducio-VAE.
- 15
- 15
- 15
- GPU-based MapReduce for large-scale near-duplicate video retrieval15
Experimental results on the benchmark CC_WEB_VIDEO NDV dataset demonstrate that the proposed framework can significantly speed up processing huge amounts of video repositories.
- Retrieval Augmented Recipe Generation14
This work introduces Stochastic Diversified Retrieval Augmentation to retrieve recipes semantically related to the image from an existing datastore as a supplement, integrating them into the prompt to add diverse and rich context to the input image.
- Fighting Malicious Media Data: A Survey on Tampering Detection and Deepfake Detection14
A comprehensive review of the current media TD approaches is provided and the challenges and trends in this field are discussed, to discuss the challenges and trends in this field for future research.
- AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuning14
A novel Adversarial Contrastive Prompt Tuning (ACPT) method is proposed to robustly fine-tune the CLIP image encoder to extract similar embeddings for any two intermediate adversarial queries and a detection framework AdvQDet is introduced that can detect 7 state-of-the-art query-based attacks with >99% detection rate within 5 shots.
- Feature Deformation Meta-Networks in Image Captioning of Novel Objects14
The feature deformation meta-networks (FDM-net), which is trained on source data, and learn to adapt to the novel object features detected by the auxiliary detection model, and the results show the effectiveness of the FDM-nets.
- 14
- Generating Keyword Queries for Natural Language Queries to Alleviate Lexical Chasm Problem14
This work formulated the task as a translation problem to convert natural language queries into keyword queries as a novel encoder-decoder method with two decoders: the pointer decoder firstly extracts query terms directly from the source text via copying mechanism, then the generator decoder generates query terms using two attention modules simultaneously considering the sourceText and extracted query terms.
- HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policies13
HiMoE-VLA, a VLA framework built around a Hierarchical Mixture-of-Experts (HiMoE) action module, turns the negative transfer observed in strong baselines into positive transfer under controlled heterogeneous co-training.
- AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models13
BackdoorVLA is introduced, a targeted backdoor attack that compels a VLA to execute an attacker-specified long-horizon action sequence whenever a trigger is present, providing a standardized framework for evaluating VLA vulnerabilities and demonstrating the potential for precise adversarial manipulation.
- 13
- Person-level Action Recognition in Complex Events via TSD-TSM Networks13
This paper proposes to conduct straddling expansion and region squaring on the original bounding box of a person proposal to widen the potential space of motion and interaction and lead to a square box for ROI detection.
- Web video categorization using category-predictive classifiers and category-specific concept classifiers13
A novel three-stage framework is proposed for Web video classification using category-predictive classifiers and category-specific concept classifiers, which integrates contextual features and concept-level semantics induced from visual content.
- A Bayesian Hashing approach and its application to face recognition13
A novel method called Bayesian Hashing is introduced, which learns an optimal Hamming embedding to encode high-dimensional features to binary bits, and its application to the challenging problem of face recognition is discussed.
- Special issue on Multimedia Event Detection13
Although the storage and dissemination capacity of the network has grown exponentially, the development of automatic tools to search and retrieve this data has not kept pace, and by large, manual annotation and categorization is used for video search.
- Adaptive Split-Fusion Transformer12
A new hybrid named Adaptive Split-Fusion Transformer (ASF-former) that treats convolutional and attention branches differently with adaptive weights is proposed that outperforms its CNN, transformer, and hybrid counterparts in terms of accuracy.
- Adaptive Temporal Grouping for Black-box Adversarial Attacks on Videos12
The adaptive temporal grouping (ATG) method, which groups video frames by the similarity of their features extracted from the ImageNet-pretrained image model, helps any black-box attack methods to optimize the adversarial perturbations over key-frames instead of all frames, where the estimated gradient of key-frame is shared with other frames in each group.
- Visual Content Recognition by Exploiting Semantic Feature Map with Attention and Multi-task Learning12
This article introduces a novel method to learn the Semantic Feature Map (SFM) with attention-based deep neural networks for image and video classification in an end-to-end manner, aiming to explicitly model the spatial object contexts within the images.
- Learning Semantic Feature Map for Visual Content Recognition12
This paper proposes to learn Semantic Feature Map by deep neural networks to model the spatial object contexts for better understanding of image and video contents and introduces a multi-task learning framework to train the model in an end-to-end manner.
- 12
- 11
- FreeEnhance: Tuning-Free Image Enhancement via Content-Consistent Noising-and-Denoising Process11
This paper proposes a novel framework, namely FreeEnhance, for content-consistent image enhancement using the off-the-shelf image diffusion models, and outperforms the state-of-the-art image enhancement models in terms of quantitative metrics and human preference.
- Relation Triplet Construction for Cross-modal Text-to-Video Retrieval11
A Multi-Granularity Matching (MGM) framework that considers both fine-grained relation triplet matching and coarse- grained global semantic matching for text-to-video retrieval is proposed, representing videos as SVO triplet tracklets.
- 11
- 11
- 11
- 10
- Decoder Pre-Training with only Text for Scene Text Recognition10
A novel method named Decoder Pre-training with only text for STR (DPTR), which treats text embeddings produced by the CLIP text encoder as pseudo visual embeddings and uses them to pre-train the decoder, and introduces a Feature Merge Unit that guides the extracted visual embeddings focusing on the character foreground within the text image, thereby enabling the pre-trained decoder to work more efficiently and accurately.
- 10
- 10
- Learning to Rank Patches for Unbiased Image Redundancy Reduction10
This work proposes a self-supervised framework for image redundancy reduction called Learning to Rank Patches (LTRP), and implements LTRP via two steps: inferring the semantic density score of each patch by quantifying variation between reconstructions with and without this patch, and learning to rank the patches with the pseudo score.
- Navigating Weight Prediction with Diet Diary10
A novel task of weight prediction with a dietary diary that aims to leverage historical food intake and weight to predict future weights is proposed, and a model-agnostic time series forecasting framework is proposed to tackle this task.
- 10
- OmniTracker: Unifying Object Tracking by Tracking-with-Detection10
This work proposes a novel tracking-with-detection paradigm, where tracking supplements appearance priors for detection and detection provides tracking with candidate bounding boxes for association and achieves on-par or even better results than both task-specific and unified tracking models.
- Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization10
Open-VCLIP is introduced, a simple yet effective approach that transforms CLIP into strong zero-shot video classifiers that can recognize unseen actions and events at test time and proposes Interpolated Weight Optimization, which utilizes the benefit of weight interpolation in both training and test time.
- 10
- 10
- TC-GAN10
The experimental results demonstrate that the proposed novel triangle cycle-consistent generative adversarial networks outperforms the state-of-the-art methods in the preservation of facial identity and expression features during face frontalization.
- 10
- 10
- The Shanghai-Hongkong Team at MediaEval2012: Violent Scene Detection Using Trajectory-based Features10
The central component of the system is a set of features derived from the appearance and motion of local patch trajectories, which use these features and SVM classier as the baseline approach and add in a few other components to further improve the performance.
- 10
- Constructing agents blackboard communication architecture based on graph theory10
This paper presents a model for constructing agent blackboard communication architecture based on graph theory, which computes the location of central blackboard or sub-blackboards based on median location method, and compute the communication topology among sub- Blackboard based on Steiner Tree method.
- Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn Dialogue9
Ask-to-Clarify is proposed, a unified end-to-end framework that seamlessly integrates multi-turn disambiguation dialogue with low-level visuomotor control, eliminating the reliance on high-level action primitives or external planners.
- DAVID-XR1: Detecting AI-Generated Videos with Explainable Reasoning9
DAMID-X is introduced, the first dataset to pair AI-generated videos with detailed defect-level, temporal-spatial annotations and written rationales, and DAVID-XR1, a video-language model designed to deliver an interpretable chain of visual reasoning-including defect categorization, temporal-spatial localization, and natural language explanations.
- Deeper Insights into the Robustness of ViTs towards Common Corruptions9
This paper investigates how CNN-like architectural designs and CNN-based data augmentation strategies impact on ViTs’ robustness towards common corruptions through an extensive and rigorous benchmarking, and introduces a novel conditional method enabling input-varied augmentations from two angles.
- HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognition9
Hierarchical and Conditional Modality Selection (HCMS) is introduced, a simple yet efficient multimodal learning framework for efficient video recognition that dynamically decides on-the-fly whether to use computationally expensive modalities on a per-input basis.
- 9
- 9
- On Stochastic Primal-Dual Hybrid Gradient Approach for Compositely Regularized Minimization9
A Stochastic PDHG (SPDHG) algorithm with either uniformly or non-uniformly averaged iterates that outperforms other competing algorithms on a wide spectrum of regularized stochastic minimization problems.
- A novel defense model for dynamic topology network based on mobile agent9
A novel defense model for the dynamic topology network, which includes three modules: network topology discovery, adaptive agents re-configuration mechanism and active defense, based on mobile agent technology.
- 8
- PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer8
A Parameter-Efficient Cross-Task Prompt (PECTP) framework with Prompt Retention Module (PRM) and classifier Head Retention Module (HRM) and HRM to inherit old knowledge in the previously learned classifier heads to facilitate the cross-task prompts’ generalization ability.
- Text-Driven Video Prediction8
To investigate the capability of text in causal inference for progressive motion information, the TVP framework contains a Text Inference Module (TIM), producing step-wise embeddings to regulate motion inference for subsequent frames, and a refinement mechanism incorporating global motion semantics guarantees coherent generation.
- ModelLock: Locking Your Model With a Spell8
A diffusion-based framework dubbed ModelLock is proposed that explores text-guided image editing to transform the private finetuning data into unique styles or blend new objects into the background to open up a new direction for intellectual property protection of private models.
- Learning to Separate Domains in Generalized Zero-Shot and Open Set Learning: a probabilistic perspective8
A probabilistic way of directly estimating and fine-tuning the decision boundary between seen and novel/unseen classes is proposed and the uncertain domain is newly introduced in this framework to adopt those instances whose domain cannot be predicted confidently.
- BigVid at MediaEval 2016: Predicting Interestingness in Images and Videos.8
Results show that the visual features and high-level attributes are complementary to each other, which is interesting and potential useful in the MediaEval 2016 Predicting Media Interestingness Task.
- A Framework of Video Coding for Compressing Near-Duplicate Videos8
A novel video coding framework is proposed to effectively compress NDVs by making full use of the relevance among them and Experimental results demonstrate that a significant storage saving can be achieved by the proposed NDV coding framework.
- Strong geometrical consistency in large scale partial-duplicate image search8
A new geometry verification method named as Strong Geometry Consistency (SGC), which uses the orientation, scale and location information of the local feature points to accurately and quickly remove the false matches, and a simple scale weighting strategy, which gives feature points with larger scales greater weights.
- Mixture of Experts for Stellar Data Classification8
Real world data experimental results show that the mixture of experts has a good generalization ability and the obtained correct classification rate is higher than that of using a single neural network.
- Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy7
Thinking with Deltas is proposed, a framework driven by a differential Visual Reasoning Policy that inherently bolsters visual understanding capabilities and significantly outperforms state-of-the-art methods on both general and medical benchmarks, without requiring external annotations or auxiliary tools.
- ArcFlow: Unleashing 2-Step Text-to-Image Generation via High-Precision Non-Linear Flow Distillation7
This work proposes ArcFlow, a few-step distillation framework that explicitly employs non-linear flow trajectories to approximate pre-trained teacher trajectories and implements ArcFlow via trajectory distillation on pre-trained teacher models using lightweight adapters to ensure fast, stable convergence while preserving generative diversity and quality.
- OmniGen-AR: AutoRegressive Any-to-Image Generation7
This work presents OmniGen-AR, a unified autoregressive framework for Any-to-Image generation that supports a broad spectrum of conditional inputs within a single model, and introduces Disentangled Causal Attention (DCA), which separates the full-sequence causal mask into condition causal attention and content causal attention.
- INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning7
Experimental results show that the proposed Inst-IT models not only achieve outstanding performance on Inst-IT Bench and other instance understanding benchmarks, but also demonstrate significant improvements across various generic image and video understanding benchmarks, highlighting that the method not only boosts instance-level understanding but also strengthens the overall capabilities of generic image and video comprehension.
- ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoning7
ControlThinker, a novel framework that employs a "comprehend-then-generate" paradigm that effectively mitigates the semantic gap between raw text prompts and target images, resulting in improved visual quality and semantic consistency across a wide range of benchmarks.
- 7
- Unified View Empirical Study for Large Pretrained Model on Cross-Domain Few-Shot Learning7
This study presents an exhaustive empirical assessment of the Contrastive Language–Image Pre-Training model within the CD-FSL task, and establishes a straightforward baseline model, E-base, based on the authors' empirical analysis, underscoring the importance of this investigation.
- 7
- LoRA of Change: Learning to Generate LoRA for the Editing Instruction From a Single Before-After Image Pair6
The LoRA of Change of Change framework for image editing with visual instructions is proposed, enabling large-scale training with paired data alone and introducing the LoRA Reverse optimization technique, enabling large-scale training with paired data alone.
- 6
- 6
- BadPatch: Diffusion-Based Generation of Physical Adversarial Patches6
This work proposes BadPatch, a novel diffusion-based framework for generating customizable and naturalistic adversarial patches that achieves attack performance comparable to state-of-the-art non-naturalistic patches while maintaining a natural appearance.
- 6
- SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Images6
Experimental results show that SparseGrasp significantly outperforms state-of-the-art methods in terms of both speed and adaptability, providing a robust solution for multi-turn grasping in changeable environment.
- Video Mobile-Former: Video Recognition with Efficient Global Spatial-temporal Modeling6
Video Mobile-Former is the first Transformer-based video model which constrains the computational budget within 1G FLOPs, and significantly improves the video recognition performance of alternative lightweight baselines, and outperforms other efficient CNN-based models at the low FLOP regime from 500M to 6G totalFLOPs on various video recognition tasks.
- Bag of Tricks for Building an Accurate and Slim Object Detector for Embedded Applications6
A bag of tricks that improve the detection performance for a specified on-road application, under the premise of ensuring that it does not increase the computational cost of YOLOv5s are explored.
- Enhancement of refresh time in quasi-nonvolatile memory by the density of states engineering6
This work provides a new idea to enhance the refresh time of quasi-nonvolatile memory by the density of states engineering and demonstrates great application potential for high-speed and low-power memory technology.
- 6
- Towards Optimal CNN Descriptors for Large-Scale Image Retrieval6
This paper presents a unified implementation of modern global-CNN-based retrieval systems, break such a system into six major components, and investigates each part individually as well as globally when considering different configurations, and introduces a novel joint loss function with learnable parameter for fine-tuning for retrieval tasks.
- Learning part-based mid-level representation for visual recognition6
This paper proposes a mid-level image representation for visual recognition, where an image is represented based upon the response maps of local part filters, and leverages sparse coding to accelerate the feature extraction process, which is ten times faster without significantly compromising the recognition accuracy.
- Adaptive Proximal Average Approximation for Composite Convex Minimization6
Preliminary experimental results on overlapping group Lasso and graph-guided fused Lasso problems indicate that the proposed method is almost five times faster than the state-of-the-art PA-APG method and therefore suitable for higher-precision required optimization.
- Modeling Local Interest Points for Semantic Detection and Video Search at TRECVID 2006.6
In high-level feature extraction, LIPs with both local description and spatial distribution are explored with both local description and spatial distribution for characterizing and sketching semantic concepts respectively for high-level feature extraction and automatic video search.
- Steering the Verifiability of Multimodal AI Hallucinations5
It is revealed that obvious and elusive hallucinations elicit different intervention probes, allowing for fine-grained control over the model's verifiability, and an activation-space intervention method is proposed that learns separate probes for obvious and elusive hallucinations.
- LRANet++: Low-Rank Approximation Network for Accurate and Efficient Text Spotting5
A novel parameterized text shape representation based on low-rank approximation for precise detection and a triple assignment detection head for fast inference is proposed and integrated into an end-to-end text spotting framework, termed LRANet++, capable of accurately and efficiently spotting arbitrary-shaped text.
- Achieving More with Less: Additive Prompt Tuning for Rehearsal-Free Class-Incremental Learning5
This work presents a novel prompt-based CIL paradigm that trains a single set of shared prompts across all tasks and directly modifies the CLS token's attention computation by adding the prompts to it, offering a more efficient yet powerful solution for rehearsal-free class-incremental learning.
- 5
- GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models5
GenBreak is a framework that fine-tunes a red-team large language model (LLM) to systematically explore underlying vulnerabilities in T2I generators, and guides the LLM to craft adversarial prompts that enhance both evasion capability and image toxicity, while maintaining semantic coherence and diversity.
- You Only Estimate Once: Unified, One-stage, Real-Time Category-Level Articulated Object 6D Pose Estimation for Robotic Grasping5
YOEO is proposed, a single-stage method that simultaneously outputs instance segmentation and NPCS representations in an end-to-end manner and deploys a synthetically-trained model in a real-world setting, providing real-time visual feedback at 200Hz, enabling a physical Kinova robot to interact with unseen articulated objects.
- Domain Expansion and Boundary Growth for Open-Set Single-Source Domain Generalization5
This article proposes a novel learning approach based on domain expansion and boundary growth to expand the scarce source samples and enlarge the boundaries across the known classes that indirectly broaden the boundary between the known and unknown classes.
- 5
- 5
- EnJa: Ensemble Jailbreak on Large Language Models5
A novel EnJa attack to hide harmful instructions using prompt-level jailbreak, boost the attack success rate using a gradient-based attack, and connect the two types of jailbreak attacks via a template-based connector are proposed.
- Ingredient-enriched Recipe Generation from Cooking Videos5
A framework that introduces ingredient recognition module which uses the copy mechanism to fuse the predicted ingredient information into the generated sentence is proposed which achieves the promising performances on both YouCookII and Cooking-COIN datasets.
- Transferability Estimation Based On Principal Gradient Expectation5
Principal Gradient Expectation (PGE), a simple yet effective method for assessing transferability, is proposed and extensive experiments show that the proposed metric is superior to state-of-the-art methods on all properties.
- Reformulating natural language queries using sequence-to-sequence models5
Dear editor, In recent years, reformulating natural language queries has attracted significant amount of attention from both industry and academia, however, there is a lexical chasm betweennatural language queries and relevant documents.
- Instance Image Retrieval with Generative Adversarial Training5
Experiments on four widely used databases show that adversarial training really works for instance level image retrieval and the proposed ILRGAN can get promising retrieval performances.
- [Reproduction,genotype identification and evaluation of APP/PS1 transgenic mice].5
The transgenic mouse is an effective tool for the research and prevention of AD and presents typical symptoms and behaviors of Alzheimer's disease.
- 5
- Large scale semantic concept detection, fusion, and selection for domain adaptive video search5
This thesis presents approaches to handle three correlated issues that are critical to this problem: how to construct an effective feature representation for semantic concept detection, how to exploit semantic context to improve the detection of these concepts, and how to select the most suitable concept detectors to answer user queries.
- BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos4
It is demonstrated that training existing video artifact detection models and multi-modal large language models (MLLMs) on BrokenVideos substantially enhances their ability to localize corrupted regions within generated content.
- From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning4
The Dual Low-Rank Adaptation (Dual-LoRA), a holistic-to-local framework that enhances the adapter's capacity to address data conflict through dual structural optimization, and introduces Visual Cue Enhancement (VCE), a multi-level local feature aggregation module designed to enrich the visionlanguage projection with local details.
- T2UE: Generating Unlearnable Examples from Text Descriptions4
Text-to-Unlearnable Example (T2UE), a novel framework that enables users to generate UEs using only text descriptions, demonstrates the feasibility of ''zero-contact data protection', where personal data can be safeguarded based solely on their textual descriptions, eliminating the need for direct data exposure.
- MDiff4STR: Mask Diffusion Model for Scene Text Recognition4
MDiff4STR, a Mask Diffusion model enhanced with two key improvement strategies tailored for STR consistently outperforms popular STR models, surpassing state-of-the-art ARMs in accuracy, while maintaining fast inference with only three denoising steps.
- ReToMe-VA: Recursive Token Merging for Video Diffusion-based Unrestricted Adversarial Attack4
The Recursive Token Merging for Video Diffusion-based Unrestricted Adversarial Attack (ReToMe-VA), which is the first framework to generate imperceptible adversarial video clips with higher transferability and introduces a Recursive Token Merging (ReToMe) mechanism, resulting in temporally consistent adversarial videos.
- 4
- A Multimodal Framework for Video Ads Understanding4
A multimodal system to improve the ability of structured analysis of advertising video content, which achieved a score of 0.2470 measured in consideration of localization and prediction accuracy, ranking fourth in the 2021 TAAC final leaderboard.
- Stacked multichannel autoencoder – an efficient way of learning from synthetic data4
This paper investigates and formalizes a general framework – Stacked Multichannel Autoencoder (SMCAE) that enables bridging the synthetic gap and learning from synthetic data more efficiently and shows that this SMCAE can not only transform and use synthetic data on a challenging face-sketch recognition task, but that it can also help simulate real images which can be used for training classifiers for recognition.
- LSVC20174
This workshop and challenge aims at exploring new challenges and approaches for large-scale video classification with large number of classes from open source videos in a realistic setting, based upon an extension of Fudan-Columbia Video Dataset (FCVID).
- Optimal Bayesian hashing for efficient face recognition4
This paper proposes a novel method called Bayesian Hashing to learn an optimal Hamming embedding of high-dimensional features, with a focus on the challenging application of face recognition.
- Multiple task learning with flexible structure regularization4
This work directly imposes a joint ?
- Organizing Video Search Results to Adapted Semantic Hierarchies for Topic-based Browsing4
This paper introduces an approach to organize video search results to an adapted semantic hierarchy by starting from the Wikipedia hierarchies and adjusting the structures according to the characteristics of the returned videos from a search engine.
- Placing Videos on a Semantic Hierarchy for Search Result Navigation4
This article proposes to organize video search results in a highly structured way by placing videos on a semantic hierarchy that accurately organizes various facets of a given search topic.
- 4
- VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models3
Experiments on RoboTwin, RLBench, and real-world manipulation tasks show that VLA-Pro consistently improves cross-task generalization across multiple backbones, achieving up to a 207% relative improvement in simulation and increasing real-world success rate from 5.8% to 65.0%.
- NAP-Tuning: Neural Augmented Prompt Tuning for Adversarially Robust Vision-Language Models3
This work highlights the efficacy of internal feature-level intervention in prompt tuning for adversarial robustness, moving beyond input-side alignment approaches to create an adaptive defense mechanism that can identify and rectify adversarial perturbations across embedding spaces.
- Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation3
This work proposes a semantically grounded framework that predicts and validates actions and ingredients as internal context for instruction generation and shows state-of-the-art performance and markedly improved semantic fidelity.
- Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval3
This work designs a pose-aware spatial memory mechanism that organizes history frames according to their already estimated camera poses, enabling location-aware retrieval and proposes Segment Sampling so that the retrieval spans distinct relevant segments rather than a single high-similarity region.
- 3
- FaceA-Net: Facial Attribute-Driven ID Preserving Image Generation Network3
This work proposes an attribute-driven training method that establishes global-level and local-level tasks to learn the global face feature and fine-grained attribute features, respectively and introduces a novel ID-context decoupling framework that decouples image context generation from human ID generation, ensuring the quality of contextual content as well as facilitating the learning of ID information.
- 3
- 3
- Can Action be Imitated? Learn to Reconstruct and Transfer Human Dynamics from Videos3
This paper introduces a novel task, dubbedmeshbased action imitation, to enable an arbitrary target human mesh to perform the same action shown on the video demonstration, using a novel Mesh-based Video Action Imitation (M-VAI) method.
- 3
- VSCC'20173
A brief summary of the first workshop on visual analysis for smart and connected communities (VSCC'2017), which is held in conjunction with the ACM Conference on Multimedia, 2017 is presented.
- 3
- 3
- Spectral Analysis and Recognition Using Multi-scale Features and Neural Networks3
A novel spectral analysis and classification technique, which is based on multi-scale feature extraction and neural networks, and whose classification results are much better than the best results obtained by principle component analysis feature extraction method.
- 2
- RAG-6DPose: Retrieval-Augmented 6D Pose Estimation via Leveraging CAD as Knowledge Base2
RAG-6DPose is presented, a retrieval-augmented approach that leverages 3D CAD models as a knowledge base by integrating both visual and geometric cues, demonstrating the effectiveness and robustness of the approach, particularly in handling occlusions and novel viewpoints.
- What Do Visual Models Look At? Dilated Attention for Targeted Transferable Attacks2
The Dilated Attention (DA) attack is proposed, which integrates attention maximization loss and dynamic linear augmentation to improve targeted transferability and hopes it can serve as a foundational attack, guiding future research endeavors in the domain of targeted transferable attacks.
- VCE: Safe Autoregressive Image Generation via Visual Contrast Exploitation2
This work proposes Visual Contrast Exploitation (VCE), a novel framework comprising an innovative contrastive image pair construction paradigm that precisely decouples unsafe concepts from their associated content semantics, and a sophisticated DPO-based training approach that enhances the model's ability to identify and leverage visual contrastive features from image pairs, enabling precise concept erasure.
- 2
- 2
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection2
A novel personalized vision-language model (VLM) that integrates low-level visual artifact analysis and high-level semantic inconsistency detection and employs a lightweight method that dynamically encodes identity-specific information into specialized identifier tokens, enabling the model to learn distinct identity characteristics while maintaining robust generalization capabilities.
- 2
- EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models2
This work proposes EAGLE, a novel MLLM that empowers comprehension of arbitrary referring visual prompts with less training efforts than existing approaches, and embeds referring visual prompts as spatial concepts conveying specific spatial areas comprehensible to the MLLM, with the semantic comprehension of these regions originating from the MLLM itself.
- Expose Before You Defend: Unifying and Enhancing Backdoor Defenses via Exposed Models2
The results demonstrate the importance of backdoor exposure for backdoor defense, showing that the exposed models can significantly benefit a range of downstream defense tasks, including backdoor label detection, backdoor trigger recovery, backdoor model detection, and backdoor removal.
- Prompting Large Language Models to Reformulate Queries for Moment Localization2
Inspired by the recent success of large language models, especially their ability of understanding and generating complex natural language contents, in this extended abstract, early attempts are made at reformulating the moment queries into a set of instructions using largelanguage models and making them more friendly to the localization models.
- 2
- Composite Binary Decomposition Networks2
The composite binary decomposition networks (CBDNet) are proposed, which first compose real-valued tensor of each layer with a limited number of binary tensors, and then decompose some conditioned binary tensor into two low-rank binary Tensors, so that the number of parameters and operations are greatly reduced comparing to the original ones.
- Iterative object and part transfer for fine-grained recognition2
This paper proposes a nonparametric data-driven method for object and part localization that achieves better results than many state-of-the-art approaches, including a few using oracle (manually annotated) bounding boxes in the test images.
- NTT-Fudan Team @ TRECVID 2015: Multimedia Event Detection.2
An overview of the developed system is presented, highlighting on the selection and fusion of multiple classification models from a wide range of feature representations to improve the performance.
- ASM'152
ASM workshop provides a forum to present and discuss the emerging field of affect and sentiment analysis in multimedia and focuses mainly on multimedia content analysis and its applications for automatic sentiment and affective analysis.
- 2
- Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models1
Spatiotemporal sycophancy is identified, a failure mode in which Vid-LLMs retract initially correct, visually grounded judgments and conform to misleading user feedback under negation-based gaslighting.
- SpatialImaginer: Towards Adaptive Visual Imagination for Spatial Reasoning1
SpatialImaginer is proposed, a unified multimodal generation framework that integrates textual reasoning with visual imagination and adopts a divide-and-conquer strategy, using text chain-of-thought for high-level semantic planning and the visual imagination for geometry-sensitive state transformation and consistency preservation.
- 1
- 1
- Brain3D: Generating 3D Objects from fMRI1
A novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject who was presented with a 2D image, and yields as output the corresponding 3D object images, aligning remarkably with the established discoveries in neuroscience.
- 1
- Optimizing Cross-Client Domain Coverage for Federated Instruction Tuning of Large Language Models1
This work empirically establishes that cross-client domain coverage, rather than data heterogeneity, is the pivotal factor in FedDIT dynamics and introduces FedDCA, an algorithm that explicitly maximizes this coverage through diversity-oriented client center selection and retrieval-based augmentation, constructing diverse, non-redundant cross-client instruction sets.
- 1
- Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric1
A novel metric, the Model Utilization Index (MUI), is proposed, which introduces mechanism interpretability techniques to complement traditional performance metrics and deduces a common trend observed in popular LLMs, which is termed the “Utility Law.
- EditMaster: Bridging Text instruction and Visual Example for Multimodal guided Image Editing1
The EditMaster framework is proposed, a unified framework that integrates text and visual controls through multimodal instruction learning, enabling precise image manipulation with bidirectional consistency and setting a new benchmark for this field.
- 1
- OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks1
OmniGenBench is introduced, a novel and comprehensive benchmark meticulously designed to assess the instruction-following abilities of state-of-the-art LMMs across both perception-centric and cognition-centric dimensions.
- FedAPT: Federated Adversarial Prompt Tuning for Vision-Language Models1
This work introduces Federated Adversarial Prompt Tuning (FedAPT), a novel method designed to enhance the adversarial robustness of FPT, and proposes a class-aware prompt generator that generates visual prompts from text prompts.
- 1
- Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers1
Downstream Transfer Attack (DTA) is proposed, a novel attack method that leverages a pre-trained ViT model to craft the adversarial example and then applies the adversarial example to attack a fine-tuned version of the model on a downstream dataset.
- 1
- Adaptive Retention & Correction: Test-Time Training for Continual Learning1
This study introduces a simple Out-of-Task Detection method, OTD, and proposes an Adaptive Retention mechanism for dynamically tuning the classifier layer on past task data; an Adaptive Correction mechanism for revising predictions when the model classifies data from previous tasks into classes from the current task.
- AutoTVG: A New Vision-language Pre-training Paradigm for Temporal Video Grounding1
Experimental results on Charades-STA and ActivityNet Captions show that AutoTVG achieves highly competitive performance with in-distribution methods under out-of-distribution testing, and is superior to existing pre-training frameworks with much less training data.
- V3Det Challenge 2024 on Vast Vocabulary and Open Vocabulary Object Detection: Methods and Results1
The V3Det Challenge 2024 aims to push the boundaries of object detection research and encourage innovation in this field, and aims to inspire future research directions in vast vocabulary and open-vocabulary object detection, driving progress in this field.
- 1
- 1
- Extreme vocabulary learning1
A novel visual-semantic embedding framework — extreme vocabulary learning (EVoL) is proposed; the EVoL embeds the visual features into semantic space in a probabilistic way and adopts the vast open vocabulary in the semantic space to help further constraint the margin and coverage of training classes.
- 1
- 1
- Learning to score and summarize figure skating sport videos.1
A reinforcement learning based video summarization algorithm is proposed here; and the experiments show better performance than the other baseline video summarizing algorithms.
- Left-Right Skip-DenseNets for Coarse-to-Fine Object Categorization.1
The left-right skip-densenets for coarse-to-fine object categorization can enable both coarse and fine-grained classification in a single framework and the layer-skipping mechanism which learns a gating network to predict whether skip some layers in the testing stage is proposed.
- A fast video event recognition system and its application to video search1
This work presents a fast video event recognition system, which requires just a few seconds to process a general YouTube video with a few minutes of duration, and demonstrates the technical feasibility of using event recognition results as the sole clue for video search.
- On the pooling of positive examples with ontology for visual concept learning1
In this paper, ontology knowledge is incorporated for semantic pooling of positive examples from ontologically neighboring concepts, which effectively widens the coverage of the positive samples with visually more diversified content, which is important for learning a good concept classifier.
- [Advances of nutrigenomics].1
The main research fields, advances and the trend in the future of nutrigenomics are discussed.
- –
- –
- PixelSmile: Toward Fine-Grained Facial Expression Editing–
PixelSmile is proposed, a diffusion framework that disentangles expression semantics via fully symmetric joint training that achieves superior disentanglement and robust identity preservation, confirming its effectiveness for continuous, controllable, and fine-grained expression editing, while naturally supporting smooth expression blending.
- –
- UniCam: Taming Unified Diffusion Models in Noise Space for Camera-controllable Video Rendering–
The UniCam framework is proposed, a unified framework that introduces a temporally coherent stochastic representation, termed CameraNoise, warped from camera intrinsic and extrinsic parameters, which significantly outperforms prior methods in both fidelity and controllability.
- –
- –
- –
- –
- Multi-Prompt Progressive Alignment for Multi-Source Unsupervised Domain Adaptation–
This work proposes a progressive alignment strategy for adapting CLIP to unlabeled downstream task that effectively mitigates confirmation bias and promotes a more robust convergence, allowing for the learning of genuinely domain-invariant features.
- –
- Repeating Words for Video-Language Retrieval with Coarse-to-Fine Objectives–
It is observed that the repetition of keywords in the original captions can enhance retrieval performance and improve alignment between video and text, and a novel and effective inference pipeline is proposed that incorporates a voting mechanism and a new Matching Entropy metric.
- –
- –
- –
- –
- –
- –
- –
- –
- COSMO-RL: Towards Trustworthy LMRMs via Joint Safety and Stability–
COSMO-RL is presented, a mixed reinforcement learning framework that trains reasoning oriented LMRMs under multimodal, multitask, and multiobjective signals, and the resulting model, COSMO-R1, improves safety while maintaining-and often improving multimodal reasoning and instruction following.
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- FedEGG: Federated Learning with Explicit Global Guidance–
A novel FL algorithm FedWalk is proposed that leverages an ex-ternal easy-to-converge task at the server side as a leash task to guide the local training of the clients.
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- Research and application of SVM analysis model for spectral fingerprint of mixed gas–
Experimental results show that under the condition of optimizing the model’s parameters, selecting the fingerprint features of the spectral data reasonably and constructing a new spectral data sample with the fingerprint characteristics of the mixed gas component can help eliminate the measurement cross sensitivity caused by the cross overlap of the absorption spectra of the Mixed gas component.
- –
- Large-scale video semantic recognition based on consistency of segment-level and video-level predictions–
This paper proposes a video semantic recognition algorithm based on the consistency of video- and segment-level predictions that achieves 82.62% mean average precision on the video segment semantic recognition task using the large-scale video dataset YouTube-8M.
- FDU Participation in TRECVID 2019 VTT Task.–
The system design of the FDU team in the TRECVID 2019 VTT task adopts temporal concept prediction as an auxiliary task to assist caption generation and demonstrates the effectiveness of the designed auxiliary task as well as the whole captioning system.
- –
- [Establishment of pharyngeal 3-dimensional finite element model of patients with isolated cleft palate].–
Finite element model of pharyngeal cavity has good biomechanical characteristics and geometric similarity and can be used in isolated cleft palate with preoperative biomechanicals analysis, for repairing and functional reconstructive surgery to provide ideal biomechanICAL model predicts.
- –
- –
- –
- Fudan at TRECVID 2015: Adaptive Feature Fusion for Multimedia Event Detection in Videos.–
An overview of the MED system is presented, focusing on combining multiple feature representations to improve the performance, and a simple yet effective fusion method is adopted to generate the predictions.
- –
- –
- –
- –
- –
- Registration of intraoperative optical image sequence–
An improved aTPS (iaTPS) algorithm to deal with the elasticity of the cortical warping is proposed, and the comparison experiment on the intraoperative optical data of human brain has shown that the new algorithm can offer better registration accuracy than the aTPS algorithm.
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-14. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.