Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileWorks96 from public data
- LAVT: Language-Aware Vision Transformer for Referring Image Segmentation569
This work shows that significantly better cross-modal alignments can be achieved through the early fusion of linguistic and visual features in intermediate layers of a vision Transformer encoder network.
- Is synthetic data from generative models ready for image recognition?443
This work extensively study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks, and focuses on two perspectives: synthetic data for improving classification models in data-scarce settings (i.e. zero-shot and few-shot), and syntheticData for large-scale model pre-training for transfer learning.
- Memory in the Age of AI Agents288
This work aims to provide an up-to-date landscape of current agent memory research, clearly delineating the scope of agent memory and distinguishing it from related concepts such as LLM memory, retrieval augmented generation (RAG), and context engineering, and articulate a forward-looking perspective on emerging research frontiers.
- A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models254
This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models, image-text matching models, and text- to-image generation models.
- Stable Virtual Camera: Generative View Synthesis with Diffusion Models208
Stable Virtual Camera (Seva), a generalist diffusion model that creates novel views of a scene, given any number of input views and target cameras, is presented, which outperforms existing methods across different datasets and settings.
- Deep FusionNet for Point Cloud Semantic Segmentation196
A deep fusion network architecture (FusionNet) with a unique voxel-based \mini-PointNet" point cloud representation and a new feature aggregation module (fusion module) for large-scale 3D semantic segmentation.
- Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learning191
A neural collapse inspired framework for FSCIL that holds the neural collapse optimality and does not break the feature-classifier alignment in an incremental fashion and experiments demonstrate that the proposed framework outperforms the state-of-the-art performances.
- TransMix: Attend to Mix for Vision Transformers140
TransMix is proposed, which mixes labels based on the attention maps of Vision Transformers, which can consistently improve various ViT-based models at scales on ImageNet classification.
- Towards Interpreting Visual Information Processing in Vision-Language Models129
This work studies the processing of visual tokens in the language model component of LLaVA, a prominent VLM, and finds that the model extracts object information from these refined representations at the last token position for prediction, mirroring the process in text-only language models for factual association tasks.
- SegPGD: An Effective and Efficient Adversarial Attack for Evaluating and Boosting Segmentation Robustness114
This work proposes an effective and efficient segmentation attack method, dubbed SegPGD, and proposes to apply it as the underlying attack method for segmentation adversarial training, which can boost the robustness of segmentation models.
- Vision-Language Models Do Not Understand Negation92
A data-centric approach wherein CLIP models are finetune on large-scale synthetic datasets containing millions of negated captions is explored, which can result in a 10% increase in recall on negated queries and a 28% boost in accuracy on multiple-choice questions with negated captions.
- Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation90
This work proposes a novel self-supervised approach to find interpretable latent directions for a given concept in an interpretable latent space of diffusion models as seman-tic concepts, and discovers vectors related to inappropriate concepts.
- Joint Optimisation for Object Class Segmentation and Dense Stereo Reconstruction86
This work provides a principled energy minimisation framework that unifies the two problems of dense stereo reconstruction and object class segmentation and demonstrates that, by resolving ambiguities in real world data, joint optimisation of the two problems substantially improves performance.
- A Survey on Transferability of Adversarial Examples across Deep Neural Networks76
This survey explores the landscape of the adversarial transferability of adversarial examples and categorizes existing methodologies to enhance adversarial transferability, and discusses the fundamental principles guiding each approach.
- HFS: Hierarchical Feature Selection for Efficient Image Segmentation69
This paper proposes a real-time system, Hierarchical Feature Selection (HFS), that performs image segmentation at a speed of 50 frames-per-second, and can be used in a variety computer vision tasks that are built on top ofimage segmentation and superpixel extraction.
- Open Problems in Machine Unlearning for AI Safety63
This paper identifies key limitations that prevent unlearning from serving as a comprehensive solution for AI safety, particularly in managing dual-use knowledge in sensitive domains like cybersecurity and chemical, biological, radiological, and nuclear safety.
- Real-Fake: Effective Training Data Synthesis Through Distribution Matching53
This work analyzes the principles underlying training data synthesis for supervised learning and elucidates a principled theoretical framework from the distribution-matching perspective that explicates the mechanisms governing synthesis efficacy.
- Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models48
The robustness of 11 widely-used adaptation methods across 4 vision-language datasets under multimodal corruptions is assessed and it is indicated that increasing the number of adaptation data and parameters does not guarantee enhanced robustness; instead it results in even lower robustness.
- Video Motion Transfer with Diffusion Transformers44
This work proposes DiTFlow, a method for transferring the motion of a reference video to a newly synthesized one, designed specifically for Diffusion Transformers, and guides the latent denoising process in an optimization-based, training-free, manner.
- Interleaving Reasoning for Better Text-to-Image Generation40
This work introduces Interleaving Reasoning Generation (IRG), a framework that alternates between text-based thinking and image synthesis: the model first produces a text-based thinking to guide an initial image, then reflects on the result to refine fine-grained details, visual quality, and aesthetics while preserving semantics.
- Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models40
An Audio Dialogue Understanding Benchmark (ADU-Bench) is proposed, which consists of 4 benchmark datasets that assess the open-ended audio dialogue ability for LALMs in 3 general scenarios, 12 skills, 9 multilingual languages, and 4 categories of ambiguity handling.
- Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?37
GPT4 and GPT-4V demonstrate better robustness against jailbreak attacks compared to open-source LLMs and MLLMs, and Llama2 and Qwen-VL-Chat are more robust compared to other open-source models.
- Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?33
A simple yet effective approach is proposed, termed Mixed Modality In-Context Example Selection (MMICES), which considers both visual and language modalities when selecting demos, which revealed that multimodal ICL is predominantly driven by the textual content whereas the visual information in the demos has little influence.
- Mixture of Experts Made Intrinsically Interpretable30
MoE-X, a Mixture-of-Experts (MoE) language model designed to be interpretable, achieves performance comparable to dense models while significantly improving interpretability, with interpretability surpassing even sparse autoencoder (SAE)-based approaches.
- DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks29
This work proposes a Multi-Agent Hierarchical Task Generation (MAHTG) system that extracts research-worthy inspirations from seminar transcripts and translates research-worthy inspirations into high-quality research tasks, ensuring the traceability of research task formulation while filtering noise.
- ToolTweak: An Attack on Tool Selection in LLM-based Agents29
This paper presents ToolTweak, a lightweight automatic attack that increases selection rates from a baseline of around 20% to as high as 81%, with strong transferability between open-source and closed-source models and evaluation of two defenses: paraphrasing and perplexity filtering.
- RoDLA: Benchmarking the Robustness of Document Layout Analysis Models24
This work is the first to introduce a robustness benchmark for DLA models, which includes 450K document images of three datasets, and introduces a self-titled model, i.e., Robust Document Layout Analyzer (RoDLA), which improves attention mechanisms to boost extraction of robust features.
- Not Just Pretty Pictures: Toward Interventional Data Augmentation Using Text-to-Image Generators24
It is shown that recent Text-to-Image (T2I) generators’ ability to simulate image interventions via natural-language prompts can be leveraged to train more robust models, offering a more interpretable and controllable alternative to traditional augmentation methods.
- Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image21
Evaluating the adversarial robustness of MLLMs when employing Chain-of-Thought reasoning finds that CoT marginally improves adversarial robustness against existing attack methods, and introduces a novel stop-reasoning attack technique that effectively bypasses the CoT-induced robustness enhancements.
- Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samples21
This paper investigates this vulnerability for MLLMs, particularly image-based and video-based ones, and aims to induce high energy-latency cost during inference by crafting an imperceptible perturbation.
- Code2World: A GUI World Model via Renderable Code Generation19
This work proposes Code2World, a vision-language coder that simulates the next visual state via renderable code generation that significantly enhances downstream navigation success rates in a flexible manner and achieves the top-performing next UI prediction, rivaling the competitive GPT-5 and Gemini-3-Pro-Image.
- Beyond Linear Probes: Dynamic Safety Monitoring for Language Models19
Tuncated Polynomial Classifiers are introduced, a natural extension of linear probes for dynamic activation monitoring that compete with or outperform MLP-based probe baselines of the same size for harmful prompt classification, all the while being more interpretable than their black-box counterparts.
- Which Model Generated This Image? A Model-Agnostic Approach for Origin Attribution18
This work proposes OCC-CLIP, a CLIP-based framework for few-shot one-class classification, enabling the identification of an image's source model, even among multiple candidates, and proves the effectiveness of the OCC-CLIP framework.
- CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios18
- 18
- AlignGuard: Scalable Safety Alignment for Text-to-Image Generation14
This work introduces SafetyDPO, a method for safety alignment of T2I models through Direct Preference Optimization (DPO), which consistently outperforms the state-of-the-art on many benchmarks and establishes new practices for safety alignment in T2I networks.
- 13
- Label Delay in Online Continual Learning13
A new continual learning framework with explicit modeling of the label delay between data and label streams over time steps is introduced, showing experimentally that the method is the least affected by the labeldelay factor and in some cases successfully recovers the accuracy of the non-delayed counterpart.
- As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?12
This work proposes a simple yet effective adversarial attack strategy termed Patch Representation Misalignment (PRM), Solely based on open-sourced CLIP vision encoders, that produces adversaries that simultaneously fool more than 20 downstream models spanning 4 common vision-language tasks.
- PVUW 2024 Challenge on Complex Video Understanding: Methods and Results12
Two new tracks are added, Complex Video Object Segmentation Track based on MOSE dataset and Motion Expression guided Video Segmentation track based on MeViS dataset, to foster the development of a more comprehensive and robust pixel-level understanding of video scenes in complex environments and realistic scenarios.
- 12
- 12
- Influencer Backdoor Attack on Semantic Segmentation11
This work explores backdoor attacks on segmentation models to misclassify all pixels of a victim class by injecting a specific trigger on non-victim pixels during inferences, which is dubbed Influencer Backdoor Attack (IBA).
- Linear Complexity Self-Attention with $3^{\text{rd}}$ Order Polynomials11
Two modules are proposed that can be used as ”drop-in” replacements for more-complex non-local and self-attention layers in state-of-the-art CNNs and ViT architectures and achieve comparable, if not better, performance across a wide range of computer vision tasks while keeping a complexity equivalent to a standard linear layer.
- Towards Robust Prompts on Vision-Language Models10
This work proposes robust prompt learning by integrating multiple-scale image features into the prompt, which improves both types of robustness of existing in-context and prompt learning approaches.
- Localizing Events in Videos with Multimodal Queries9
ICQ is introduced, a new benchmark designed for localizing events in videos with MQs, alongside an evaluation dataset ICQ-Highlight, and 3 Multimodal Query Adaptation methods and a novel Surrogate Fine-Tuning strategy are proposed, serving as strong baseline methods.
- Learning Camera Movement Control from Real-World Drone Videos9
DGFormer, an auto-regressive transformer that leverages the camera path and images from all past frames to predict camera movement in the next frame is introduced, which effectively learns to perform challenging camera movements such as navigating through obstacles, maintaining low altitude to increase perceived speed, and orbiting towers and buildings.
- TDGNet: Hallucination Detection in Diffusion Language Models via Temporal Dynamic Graphs8
TDGNet is introduced, a temporal dynamic graph framework that formulates hallucination detection as learning over evolving token-level attention graphs and shows consistent AUROC improvements over output-based, latent-based, and static-graph baselines, with single-pass inference and modest overhead.
- DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer8
DreamBeast significantly enhances the quality of generated 3D creatures with user-specified part compositions while reducing computational overhead, as demonstrated by extensive quantitative and qualitative evaluations.
- LUMix: Improving Mixup by Better Modelling Label Uncertainty8
This paper proposes LUMix, which models such uncertainty by adding label perturbation during training and can be universally applied to any deep networks, with minimal computational cost.
- A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization7
This method integrates a risk-neutral policy with an adjustable policy to form a risk-averse CVaR policy and reveals that it excels in identifying risk-averse CVaR policies in some Mujoco environments where the traditional CVaR-PG fails to learn a reasonable policy.
- 6
- 6
- General Adversarial Defense Against Black-box Attacks via Pixel Level and Feature Level Distribution Alignments6
This paper uses Deep Generative Networks with a novel training mechanism to eliminate the distribution gap between adversarial and clean samples in feature space of the target DNNs, and proposes a more effective pixel level training constraint to make this achievable, thus enhancing robustness on adversarial samples.
- Unsupervised Contrastive Domain Adaptation for Semantic Segmentation6
A label expansion approach that is able to discover samples from hard classes during the adaptation process to further boost performance and consistently outperforms state-of-the-art methods for domain adaptation.
- Olympus: A Universal Task Router for Computer Vision Tasks5
Olympus is introduced, a new approach that transforms Multimodal Large Language Models (MLLMs) into a unified framework capable of handling a wide array of computer vision tasks, and easily integrates with existing MLLMs, expanding their capabilities with comparable performance.
- Permission Manifests for Web Agents4
The manifest establishes a compliance framework that enables beneficial agent interactions while respecting site owners'preferences, and complementing data use initiatives such as AIPref.
- Reimagining Safety Alignment with An Image4
Magic Image, an optimization-driven visual prompt framework that enhances security while reducing over-refusal by optimizing image prompts using harmful/benign samples, enables a single model to adapt to different value systems and better align with given safety preferences without parameter updates.
- AttributionLab: Faithfulness of Feature Attribution Under Controllable Environments4
This paper designs the network and manually setting its weights, along with designing data, and serves as a sanity check for faithfulness: If an attribution method is not faithful in a controlled environment, it can be unreliable in the wild.
- Task-Agnostic Robust Representation Learning4
This paper derives an upper bound on the adversarial loss of a prediction model (which is based on the learned representation) on any downstream task, using its loss on the clean data and a robustness regularizer to make the downstream prediction model more robust.
- Collaborative Quantization Embeddings for Intra-Subject Prostate MR Image Registration4
A hierarchical quantization method is proposed, discretizing the learned feature vectors using a jointly-trained dictionary with a constrained size, in order to improve the generalisation of the registration networks.
- StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction3
StraTA is a simple framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning (RL) and trains strategy generation and action execution jointly with a hierarchical GRPO-style rollout design, further enhanced by diverse strategy rollout and critical self-judgment.
- Towards Label-Free Biological Reasoning Synthetic Dataset Creation via Uncertainty Filtering3
Applied to biological perturbation prediction, it is shown that the filtered subset has higher accuracy, and that supervised fine-tuning on uncertainty-filtered data outperforms unfiltered synthetic data, narrows the gap to ground-truth training, and surpasses strong LRM baselines.
- 3
- 3
- 3
- AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction2
AnnoCaseLaw is introduced, a first-of-its-kind dataset of 471 meticulously annotated U.S. Appeals Court negligence cases enriched with comprehensive, expert-labeled annotations that highlight key components of judicial decision making, along with relevant legal concepts.
- 2
- AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment2
This paper presents AnimateZoo, a zero-shot diffusion-based video generator to address this challenging cross-species animation issue, aiming to accurately produce animal animations while preserving the background, and introduces two high-quality animal video datasets featuring a wide variety of species.
- 2
- 2
- ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation1
This work proposes a novel hybrid approach called Compositional Simulation, which combines classical simulation and neural simulation to generate accurate action-video pairs while maintaining real-world consistency, and significantly reduces the sim2real domain gap.
- SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents1
SeClaw is introduced, a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents and provides a practical foundation for measuring, diagnosing, and comparing security failures in autonomous LLM agents.
- Reliable and Responsible Foundation Models: A Comprehensive Survey1
Critical issues, including bias and fairness, security and privacy, uncertainty, explainability, and distribution shift are explored, as well as methods like alignment and Artificial Intelligence-Generated Content (AIGC) detection.
- An MRP Formulation for Supervised Learning: Generalized Temporal Difference Learning Models1
This study demonstrates that SL can be reformulated as a problem of interconnected data modeled by an MRP, effectively solved using TD learning, and establishes conditions where TD outperforms ordinary least squares (OLS), and establishes convergence guarantees for the generalized TD algorithm.
- Diversified Dynamic Routing for Vision Tasks1
The model is explicitly trained to solve the challenge of finding relevant partitioning of the data and assigning the correct experts in an unsupervised approach and shows improved performance over several baselines on semantic segmentation and object detection and instance segmentation on MS-COCO.
- Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting1
A weakly supervised pre-training method that can acquire effective scene text representations by jointly learning and aligning visual and textual information and outperforms existing pre-training techniques consistently across multiple public datasets.
- –
- –
- –
- –
- Towards Understanding Multimodal Fine-Tuning: Spatial Features–
This work uses stage-wise model diffing, a technique that isolates representational changes introduced during multimodal fine-tuning, to reveal how a language model learns to "see", and shows when and where spatially grounded multimodal features arise.
- –
- Measuring Black-Box Confidence via Reasoning Trajectories: Geometry, Coverage, and Verbalization–
A black-box trajectory-confidence score is proposed: a CoT is embedded as a sliding-window trajectory and its convergence to external answer anchors with a one-parameter softmax, and the method needs no logits, hidden states, or supervised calibrators.
- Gradient Residual Connections–
This work studies how gradient information can be leveraged to improve neural network's ability to approximate high-frequency functions, and proposes a gradient-based residual connection as a complement to the standard identity skip connection used in residual networks.
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-12. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.