Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Volker TrespSuggested from co-authorship
- Philip H. S. TorrSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works108 from public data
- MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models328
Analysis across 12 state-of-the-art models reveals that MLLMs are susceptible to breaches instigated by the approach, even when the equipped LLMs have been safety-aligned, and proposes a straightforward yet effective prompting strategy to enhance the resilience of MLLMs against these types of attacks.
- A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models254
This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models, image-text matching models, and text- to-image generation models.
- 184
- A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment139
This paper introduces the concept of "full-stack" safety to systematically consider safety issues throughout the entire process of LLM training, deployment, and eventual commercialization, and represents the first safety survey to encompass the entire lifecycle of LLMs.
- Improved Techniques for Optimization-Based Jailbreaking on Large Language Models134
Improved techniques for optimization-based jailbreaks like GCG are presented and demonstrated that they can help GCG outperform state-of-the-art jailbreaking attacks and achieve nearly 100% attack success rate.
- Understanding Individual Decisions of CNNs via Contrastive Backpropagation108
Contrastive Layer-wise Relevance Propagation is proposed, which is capable of producing instance-specific, class-discriminative, pixel-wise explanations and both qualitative and quantitative evaluations show that the CLRP generates better explanations than the LRP.
- 103
- Backdoor Defense via Adaptively Splitting Poisoned Dataset98
This work argues that the core of training-time defense is to select poisoned samples and to handle them properly, and proposes an adaptively splitting dataset-based defense (ASD), which successfully defends against backdoor attacks during training.
- Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images94
It is found that high energy-latency cost during inference of VLMs can be manipulated by maximizing the length of generated sequences by proposing verbose images, with the goal of crafting an imperceptible perturbation to induce VLMs to generate long sentences during inference.
- Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation90
This work proposes a novel self-supervised approach to find interpretable latent directions for a given concept in an interpretable latent space of diffusion models as seman-tic concepts, and discovers vectors related to inappropriate concepts.
- 84
- Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety76
This survey provides a systematic review of current safety research on large models, covering Vision Foundation Models, Large Language Models, Vision-Language Pre-training models, Vision-Language Models, Diffusion Models, and large-model-powered Agents, and presents a comprehensive taxonomy of safety threats to these models.
- A Survey on Transferability of Adversarial Examples across Deep Neural Networks76
This survey explores the landscape of the adversarial transferability of adversarial examples and categorizes existing methodologies to enhance adversarial transferability, and discusses the fundamental principles guiding each approach.
- Improving the Robustness of Capsule Networks to Image Affine Transformations67
It is revealed that the routing procedure contributes neither to the generalization ability nor to the affine robustness of the CapsNets, and an affine CapsNet (Aff-CapsNets) is proposed, which are more robust to affine transformations.
- An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models61
This work proposes the Cross-Prompt Attack (CroPA), a method that updates the visual adversarial perturbation with learnable prompts, which are designed to counteract the misleading effects of the adversarial image.
- 59
- Hide in Thicket: Generating Imperceptible and Rational Adversarial Perturbations on 3D Point Clouds54
A novel shape-based adversarial attack method, HiT-ADV, which initially conducts a two-stage search for attack regions based on saliency and imperceptibility scores, and then adds deformation perturbations in each attack region using Gaussian kernel functions, which is extendable to physical attack.
- FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentation53
The proposed Federated Representation Augmentation (FRAug) substantially outperforms the current state-of-the-art FL methods for feature distribution shifts, including PartialFed and FedBN.
- When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models49
This survey provides a comprehensive overview of the methodologies enabling LLMs to process, understand, and generate 3D data, and examines their integration with LLMs for tasks such as 3D scene understanding, captioning, question-answering, and dialogue.
- Interpretable Graph Capsule Networks for Object Recognition45
This work proposes interpretable Graph Capsule Networks (GraCapsNets), where the routing part is replaced with a multi-head attention-based Graph Pooling approach, and individual classification explanations can be created effectively and efficiently.
- OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization44
The proposed method formulates the features of image and text sets as two distinct distributions and employs optimal transport theory to determine the most efficient mapping between them, which informs the generation of adversarial examples to effectively counteract the overfitting issues.
- Capsule Network is Not More Robust than Convolutional Network42
This paper carefully examines the special designs in CapsNet that differ from that of a ConvNet commonly used for image classification, and proposes enhanced ConvNets simply by introducing the essential components behind the CapsNet’s success.
- Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?37
GPT4 and GPT-4V demonstrate better robustness against jailbreak attacks compared to open-source LLMs and MLLMs, and Llama2 and Qwen-VL-Chat are more robust compared to other open-source models.
- Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?33
A simple yet effective approach is proposed, termed Mixed Modality In-Context Example Selection (MMICES), which considers both visual and language modalities when selecting demos, which revealed that multimodal ICL is predominantly driven by the textual content whereas the visual information in the demos has little influence.
- 27
- FedBiP: Heterogeneous One-Shot Federated Learning with Personalized Latent Diffusion Models26
Federated Bi-Level Personalization (FedBiP) is proposed, which personalizes the pretrained LDM at both instance-level and concept-level and is the first approach to simultaneously address feature space heterogeneity and client data scarcity in OSFL.
- 24
- Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image21
Evaluating the adversarial robustness of MLLMs when employing Chain-of-Thought reasoning finds that CoT marginally improves adversarial robustness against existing attack methods, and introduces a novel stop-reasoning attack technique that effectively bypasses the CoT-induced robustness enhancements.
- Energy-Latency Manipulation of Multi-modal Large Language Models via Verbose Samples21
This paper investigates this vulnerability for MLLMs, particularly image-based and video-based ones, and aims to induce high energy-latency cost during inference by crafting an imperceptible perturbation.
- Revisiting and Exploring Efficient Fast Adversarial Training via LAW: Lipschitz Regularization and Auto Weight Averaging21
An effective FGSM-based fast adversarial training method equipped with Lipschitz regularization and Auto Weight averaging, abbreviated as FGSM-LAW is proposed, demonstrating the superiority of this method over state-of-the-art fast adversarial training methods and the advanced standard adversarial training methods.
- Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Models20
This work shows that certain special characters or their combinations with English letters are stronger memory triggers, leading to more severe data leakage in large language models, and proposes a simple but effective Special Characters Attack (SCA) to induce training data leakage.
- Fast Propagation Is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworks20
This work proposes to exploit the interior building blocks of the model to improve efficiency and achieves better model robustness by dynamically sample lightweight subnetworks as a surrogate model during training.
- 20
- 20
- 19
- Which Model Generated This Image? A Model-Agnostic Approach for Origin Attribution18
This work proposes OCC-CLIP, a CLIP-based framework for few-shot one-class classification, enabling the identification of an image's source model, even among multiple candidates, and proves the effectiveness of the OCC-CLIP framework.
- 15
- True Multimodal In-Context Learning Needs Attention to the Visual Context14
DARA is introduced, an efficient fine-tuning strategy that encourages models to attend to the visual context by rebalancing attention across visual and textual tokens and presents TrueMICL, an MICL-dedicated dataset that explicitly requires the integration of multimodal information-particularly visual content-for correct task completion.
- AlignGuard: Scalable Safety Alignment for Text-to-Image Generation14
This work introduces SafetyDPO, a method for safety alignment of T2I models through Direct Preference Optimization (DPO), which consistently outperforms the state-of-the-art on many benchmarks and establishes new practices for safety alignment in T2I networks.
- Learning Visual Prompts for Guiding the Attention of Vision Transformers14
The proposed optimization-based visual prompting strategy is learned in a self-supervised manner without requiring annotations and without fine-tuning the vision transformer.
- Reliable Evaluation of Adversarial Transferability13
Contrary to the prevailing belief, the reevaluation revealed that the adversarial transferability across these diverse network types is notably diminished, and there is no single AE that can be transferred to all popular models.
- 13
- As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?12
This work proposes a simple yet effective adversarial attack strategy termed Patch Representation Misalignment (PRM), Solely based on open-sourced CLIP vision encoders, that produces adversaries that simultaneously fool more than 20 downstream models spanning 4 common vision-language tasks.
- 11
- Influencer Backdoor Attack on Semantic Segmentation11
This work explores backdoor attacks on segmentation models to misclassify all pixels of a victim class by injecting a specific trigger on non-victim pixels during inferences, which is dubbed Influencer Backdoor Attack (IBA).
- 11
- 11
- 11
- Towards Robust Prompts on Vision-Language Models10
This work proposes robust prompt learning by integrating multiple-scale image features into the prompt, which improves both types of robustness of existing in-context and prompt learning approaches.
- Localizing Events in Videos with Multimodal Queries9
ICQ is introduced, a new benchmark designed for localizing events in videos with MQs, alongside an evaluation dataset ICQ-Highlight, and 3 Multimodal Query Adaptation methods and a novel Surrogate Fine-Tuning strategy are proposed, serving as strong baseline methods.
- 9
- TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models8
Extensive experiments on various open source D-LLMs demonstrate that TraceDet consistently improves hallucination detection, achieving an average gain in AUROC of 15.2% compared to baselines.
- 8
- 7
- 7
- 7
- 6
- 6
- 6
- 6
- 6
- 5
- 5
- 5
- 5
- Reimagining Safety Alignment with An Image4
Magic Image, an optimization-driven visual prompt framework that enhances security while reducing over-refusal by optimizing image prompts using harmful/benign samples, enables a single model to adapt to different value systems and better align with given safety preferences without parameter updates.
- 4
- 4
- 4
- 4
- 3
- Boosting Fair Classifier Generalization through Adaptive Priority Reweighing3
A novel adaptive reweighing method that prioritizes samples closer to the decision boundary and assigns a higher weight to improve the generalizability of fair classifiers to eliminate the impact of the distribution shifts between training and test data on model generalizability.
- 3
- 3
- 3
- 3
- 3
- 2
- 2
- 2
- 2
- Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attacks2
This work first taxonomize the stochastic defense strategies against QBBA, and proposes to explore non-additive randomness in models to defend against it, focusing on underexplored Vision Transformers based on their flexible architectures.
- The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Models1
Experiments show that TraceLock improves the quality-step tradeoff over heuristic and learned baselines, with particularly stable behavior under cross-setting deployment, suggesting that frozen diffusion language models expose a learnable space of commitment trajectories beyond confidence-based decoding.
- 1
- 1
- 1
- 1
- XAI for In-hospital Mortality Prediction via Multimodal ICU Data1
An eXplainable Multimodal Mortality Predictor (X-MMP) approaching an efficient, explainable AI solution for predicting in-hospital mortality via multimodal ICU data, and introduces an explainable method, namely Layer-Wise Propagation to Transformer as a proper extension of the LRP method to Transformers.
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-13. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.