Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Alan YuilleSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works118 from public data
- Improving Transferability of Adversarial Examples With Input Diversity1,524
This work proposes to improve the transferability of adversarial examples by creating diverse input patterns by applying random transformations to the input images at each iteration, and shows that the proposed attack method can generate adversarialExamples that transfer much better to different networks than existing baselines.
- Adversarial Examples for Semantic Segmentation and Object Detection1,057
This paper proposes a novel algorithm named Dense Adversary Generation (DAG), which applies to the state-of-the-art networks for segmentation and detection, and finds that the adversarial perturbations can be transferred across networks with different training data, based on different architectures, and even for different recognition tasks.
- Feature Denoising for Improving Adversarial Robustness1,031
It is suggested that adversarial perturbations on images lead to noise in the features constructed by these networks, and new network architectures are developed that increase adversarial robustness by performing feature denoising.
- Adversarial Examples Improve Image Recognition634
This work proposes AdvProp, an enhanced adversarial training scheme which treats adversarial examples as additional examples, to prevent overfitting, and shows that AdvProp improves a wide range of models on various image recognition tasks and performs better when the models are bigger.
- 381
- Adversarial Attacks and Defences Competition364
In this chapter, the structure and organization of the NIPS 2017 competition is described and the solutions developed by several of the top-placing teams are described.
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning267
This approach introduces an experience-based distillation mechanism to build a hierarchical skill library SkillBank, an adaptive retrieval strategy for general and task-specific heuristics, and a recursive evolution mechanism that allows the skill library to co-evolve with the agent's policy during reinforcement learning.
- 236
- Single-Shot Object Detection with Enriched Semantics228
A novel single shot object detection network named Detection with Enriched Semantics (DES) is proposed, to enrich the semantics of object detection features within a typical deep detector, by a semantic segmentation branch and a global activation module.
- Universal Physical Camouflage Attacks on Object Detectors226
This paper proposes to learn an adversarial pattern to effectively attack all instances belonging to the same object category, referred to as Universal Physical Camouflage Attack (UPC), which crafts camouflage by jointly fooling the region proposal network, as well as misleading the classifier and the regressor to output errors.
- HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing218
This study introduces HQ-Edit, a high-quality instruction-based image editing dataset with around 200,000 edits, and proposes two evaluation metrics, Alignment and Coherence, to quantitatively assess the quality of image edit pairs using GPT-4V and DALL-E 3.
- 201
- 181
- Learning Transferable Adversarial Examples via Ghost Networks171
Ghost Networks is proposed to improve the transferability of adversarial examples by reproducing the NeurIPS 2017 adversarial competition, and outperforms the No.1 attack submission by a large margin, demonstrating its effectiveness and efficiency.
- SimpleMem: Efficient Lifelong Memory for LLM Agents149
SimpleMem is introduced, an efficient memory framework based on semantic lossless compression that consistently outperforms baseline approaches in accuracy, retrieval efficiency, and inference cost, demonstrating a superior balance between performance and efficiency.
- PatchAttack: A Black-Box Texture-Based Attack with Reinforcement Learning142
The proposed PatchAttack is query efficient and can break models for both targeted and non-targeted attacks, and circumvents state-of-the-art adversarial defense methods successfully.
- How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs116
This work introduces a comprehensive safety evaluation suite, covering both out-of-distribution (OOD) generalization and adversarial robustness, and proposes a straightforward attack strategy for misleading VLLMs to produce visual-unrelated responses.
- MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine108
This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with multigranular annotations for more than 65 diseases, and proposes LLaVA-Tri by pretraining LLaVA on MedTrinity-25M, surpassing representative SOTA multimodal large language models.
- Audio-Visual LLM for Video Understanding100
This paper introduces Audio-Visual LLM, a novel Multimodal Large Language Model designed for holistic video understanding through integrated visual and auditory inputs, using uniquely designed modality-specific tokens to selectively activate the corresponding visual and auditory encoders.
- VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models99
This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs), which categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis.
- Diffusion Models as Masked Autoencoders98
This work revisits generatively pre-training visual representations in light of recent interest in denoising diffusion models and condition diffusion models on masked input and formulate diffusion models as masked autoencoders (DiffMAE).
- 90
- What If We Recaption Billions of Web Images with LLaMA-3?89
The empirical results confirm that this enhanced dataset, Recap-DataComp-1B, offers substantial benefits in training advanced vision-language models, and the generated images exhibit a significant improvement in alignment with users' text instructions, especially in following complex queries.
- 89
- DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge Distillation88
This work proposes to boost the representation learning of a multi-camera BEV based student detector by training it to imitate the features of a well-trained LiDAR based teacher detector, leading to the state-of-the-art performance on the popular benchmark nuScenes.
- Robust and Accurate Object Detection via Adversarial Learning83
This work augments the fine-tuning stage for object detectors by exploring adversarial examples, which can be viewed as a model-dependent data augmentation, and dynamically selects the stronger adversarial images sourced from a detector’s classification and localization branches and evolves with the detector to ensure the augmentation policy stays current and relevant.
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning78
This paper introduces an efficient strategy to transform Large Language Models (LLMs) into Multi-Modal Large Language Models (MLLMs) by conceptualizing this transformation as a domain adaptation process, i.e., transitioning from text understanding to embracing multiple modalities, and demonstrates that selectively tuning only with conversational data can improve efficiency further.
- Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety76
This survey provides a systematic review of current safety research on large models, covering Vision Foundation Models, Large Language Models, Vision-Language Pre-training models, Vision-Language Models, Diffusion Models, and large-model-powered Agents, and presents a comprehensive taxonomy of safety threats to these models.
- Masked Autoencoders Enable Efficient Knowledge Distillers75
The approach is simple: in addition to optimizing the pixel reconstruction loss on masked inputs, it minimize the distance between the intermediate feature map of the teacher model and that of the student model, which leads to a computationally efficient knowledge distillation framework.
- GPT-IMAGE-EDIT-1.5M: A Million-Scale, GPT-Generated Image Dataset71
This paper introduces GPT-IMAGE-EDIT-1.5M, a publicly available, large-scale image-editing corpus containing more than 1.5 million high-quality triplets (instruction, source image, edited image), and systematically construct this dataset by leveraging the versatile capabilities of GPT-4o to unify and refine three popular image-editing datasets.
- 64
- Can CNNs Be More Robust Than Transformers?64
This paper examines the design of Transformers to build pure CNN architectures without any attention-like operations that are as robust as, or even more robust than, Transformers, and leads to three highly effective architecture designs for boosting robustness, yet simple enough to be implemented in several lines of code.
- Intriguing properties of adversarial training.64
This paper provides the first rigorous study on diagnosing elements of adversarial training, which reveals two intriguing properties of normalization and the role of network capacity, and finds that enforcing BNs to behave consistently at training and testing can further enhance robustness.
- Mamba-Reg: Vision Mamba Also Needs Registers62
To better cope with Mamba blocks’ uni-directional inference paradigm, two key modifications are introduced: evenly inserting registers throughout the input token sequence, and recycling registers for final decision predictions.
- Regional Homogeneity: Towards Learning Transferable Universal Adversarial Perturbations Against Defenses59
It is shown that a simple universal perturbation can fool a series of state-of-the-art defenses, and it is verified that regionally homogeneous perturbations can well transfer across different vision tasks.
- Revisiting Adversarial Training at Scale55
This paper introduces an efficient and effective training strategy to enable adversarial training with giant models and web-scale data at an affordable computing cost and posits AdvXL as a pioneering approach, charting a new trajectory for the efficient training of robust visual representations at significantly larger scales.
- Neural Architecture Search for Lightweight Non-Local Networks55
This work proposes a Lightweight Non-Local (LightNL) block by squeezing the transformation operations and incorporating compact features, and proposes an efficient neural architecture search algorithm to learn an optimal configuration of LightNL blocks in an end-to-end manner.
- Unleashing the Power of Visual Prompting At the Pixel Level54
A simple and effective visual prompting method for adapting pre-trained models to downstream recognition tasks, which sets a new record of 82.8% average accuracy across 12 popular classification datasets, substantially surpassing the prior art by +5.6%.
- A Simple Data Mixing Prior for Improving Self-Supervised Learning46
This paper proposes SDMP, short for Simple Data Mixing Prior, to capture this straightforward yet essential prior, and is the first method that successfully leverages data mixing to improve (rather than hurt) the performance of Vision Transformers in the self-supervised setting.
- 46
- One-Pixel Shortcut: on the Learning Preference of Deep Neural Networks44
This paper introduces an unlearnable dataset called CIFar-10-S, which is indistinguishable from CIFAR-10 by humans but induces the trained model to extremely low accuracy, and perturbs in-class images at the same position to the same target value that could mostly and stably deviate from all the original images.
- AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation42
This paper studies the vulnerabilities of transformer-based Large Language Models to jailbreaking attacks, focusing specifically on the optimization-based Greedy Coordinate Gradient (GCG) strategy, and introduces an enhanced method that manipulates models' attention scores to facilitate LLM jailbreaking, which is named AttnGCG.
- Generative Image Layer Decomposition with Visual Effects39
LayerDecomp is proposed, a generative framework for image layer decomposition which outputs photorealistic clean backgrounds and high-quality transparent foregrounds with faithfully preserved visual effects and proposes a consistency loss which enforces the model to learn accurate representations for the transparent foreground layer when ground-truth annotations are not available.
- Detecting Semantic Parts on Partially Occluded Objects38
Experiments show that the algorithm outperforms several competitors in semantic part detection when occlusions are present, and uses a simple voting method, based on log-likelihood ratio tests and spatial constraints, to combine the evidence of local cues.
- Visual concepts and compositional voting35
This work uses visual concepts as building blocks for a simple pattern theoretical model, which is significantly better than discriminative methods like SVM and deep networks trained specifically for semantic part detection, and shows that compositional voting outperforms even deep networks when the amount of occlusion becomes large.
- CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a \$10,000 Budget; An Extra \$4,000 Unlocks 81.8% Accuracy33
This work explores CLIPA at scale, extending the experiments up to the H/14 model with ~13B image-text pairs seen during training, and finds this inverse scaling law is also applicable in the finetuning stage, enabling further reduction in computational needs.
- M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation32
This paper shows that this scale-wise autoregressive framework VAR can be effectively decoupled into intra-scale modeling, which captures local spatial dependencies within each scale, and inter-scale modeling, which models cross-scale relationships progressively from coarse-to-fine scales.
- $\texttt{Complex-Edit}$: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmark30
A ``curse of synthetic data'' is observed: when synthetic data is involved in model training, the edited images from such models tend to appear increasingly synthetic as the complexity of the editing instructions rises -- a phenomenon that intriguingly also manifests in the latest GPT-4o outputs.
- 30
- Autoregressive Pretraining with Mamba in Vision29
This paper shows that Mamba's visual capability can be significantly enhanced through autoregressive pretraining, a direction not previously explored, and successfully unlocks its scaling potential to large and even huge model sizes.
- L2B: Learning to Bootstrap Robust Models for Combating Label Noise27
A simple and effective method, named Learning to Bootstrap (L2B), which enables models to bootstrap themselves using their own predictions without being adversely affected by erroneous pseudo-labels, by dynamically adjusting the importance weight between real observed and generated labels.
- SPFormer: Enhancing Vision Transformer with Superpixel Representation26
The superpixel structure offers a window into the model's internal processes, providing valuable insights that enhance the model's interpretability and robustness, and significantly improves SPFormer's robustness, particularly in challenging scenarios such as image rotations and occlusions.
- CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions25
A simple yet effective designs to better leverage richly described synthetic captions, incorporating an autoregressive captioner to mimic the recaptioning process and significantly improves zero-shot performance in cross-modal retrieval tasks.
- On the Adversarial Robustness of Camera-based 3D Object Detection25
This study conducts the first comprehensive investigation of the robustness of leading camera-based 3D object detection approaches under various adversarial conditions, and finds bird's-eye-view-based representations exhibit stronger robustness against localization attacks.
- 23
- Practical Disruption of Image Translation Deepfake Networks23
This work proposes Leaking Transferable Perturbations (LTP), an algorithm that significantly reduces the number of queries needed to disrupt an image translation network by dynamically re-purposing previous disruptions into new query efficient disruptions.
- Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-Training22
MixCon3D, a simple yet effective method aiming to sculpt holistic 3D representation in contrastive language-image-3D pre-training, surpassing the previous state-of-the-art performance on the challenging 1,156- category Objaverse-LVIS dataset by 5.7%.
- Rejuvenating image-GPT as Strong Visual Representation Learners21
This paper enhances image-GPT (iGPT), one of the pioneering works that introduce autoregressive pretraining to predict the next pixels for visual representation learning, by shifting the prediction target from raw pixels to semantic tokens, enabling a higher-level understanding of visual content.
- Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics20
Preliminary results suggest that visual instruction tuning, a prevailing strategy for transitioning LLMs into MLLMs, unexpectedly and interestingly helps models attain both improved truthfulness and ethical alignment in the pure NLP context.
- SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-training20
This paper develops SMAUG, an efficient pre-training framework for video-language models, and introduces a space-time token sparsification module, which leverages context information to further select only "important" spatial regions and temporal frames for pre-training.
- Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Images19
Calibrating Concepts and Operations (CCO) is proposed, which enables neural symbolic models to capture underlying data characteristics and to reason with hierarchical importance, and introduces an executor with learnable concept embedding magnitudes for handling distribution imbalance, and an operation calibrator for highlighting important operations and suppressing redundant ones.
- AHELM: A Holistic Evaluation of Audio-Language Models18
AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE, which evaluates the ALMs on avoiding stereotypes, and CoRe-Bench, which measures reasoning over conversational audio through inferential multi-turn question answering -- to holistically measure the performance of ALMs across 10 aspects.
- OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning17
OpenVision is presented, a fullyopen, cost-effective family of vision encoders that match or surpass the performance of OpenAI's CLIP when integrated into multimodal frameworks like LLaVA.
- Simulated Adversarial Testing of Face Recognition Models17
This work proposes a framework for learning how to test machine learning algorithms using simulators in an adversarial manner in order to find weaknesses in the model before deploying it in critical scenarios, and presents a method to find adversarial regions as opposed to the typical adversarial points found in the adversarial example literature.
- Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains16
This work investigates step-by-step reasoning in the medical and mathematical domains by explicitly decomposing the thinking trajectories into two parts: knowledge and reasoning and introduces a fine-grained evaluation framework that judges the correctness of knowledge used and the quality of reasoning.
- 16
- ViT-5: Vision Transformers for The Mid-2020s14
With a design aligned with contemporary foundation-model practices, ViT-5 offers a simple drop-in upgrade over vanilla ViT for mid-2020s vision backbones and serves as a stronger backbone for generative modeling.
- Benchmarking Robustness in Neural Radiance Fields14
This work provides the first comprehensive analysis of the robustness of NeRF-based novel view synthesis algorithms in the presence of different types of corruptions and reveals that standard data augmentation techniques do not help the robustness of NeRF-based models.
- 14
- CHAI: Command Hijacking against embodied AI13
This paper introduces CHAI (Command Hijacking against embodied AI), a physical environment indirect prompt injection attack that exploits the multimodal language interpretation abilities of AI models and underscores the urgent need for defenses that extend beyond traditional adversarial robustness.
- 13
- UPC: Learning Universal Physical Camouflage Attacks on Object Detectors.13
This paper proposes to learn an adversarial pattern to effectively attack all instances belonging to the same object category, referred to as Universal Physical Camouflage Attack (UPC), which crafts camouflage by jointly fooling the region proposal network, as well as misleading the classifier and the regressor to output errors.
- Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness12
This paper performs large-scale adversarial vision-language pre-training from scratch using web-scale data and introduces a novel ``double visual defense" to enhance this robustness of vision-language models against adversarial visual perturbations.
- Scaling (Down) CLIP: A Comprehensive Analysis of Data, Architecture, and Training Strategies12
This paper investigates the performance of the Contrastive Language-Image Pre-training (CLIP) when scaled down to limited computation budgets and provides practical insights into how to effectively train and deploy CLIP models, making them more accessible and affordable for practical use in various applications.
- 12
- ViP: Unified Certified Detection and Recovery for Patch Attack with Vision Transformers12
This paper provides the very first study on developing certified detection against the dual patch attack, in which the attacker is allowed to adversarially manipulate pixels in two different regions.
- OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning11
A simplification on OpenVision's architecture and loss design is provided, retaining only the captioning loss as a purely generative training signal, and competitively matches the original model's performance on a broad set of multimodal benchmarks while substantially cutting both training time and memory consumption.
- Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency11
The Adventurer series models are introduced, where images are treated as sequences of patch tokens and employ uni-directional language models to learn visual representations and this architecture can benefit future explorations in modeling long sequences for high-resolution or fine-grained images.
- 11
- AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability10
This paper comprehensively build AQA-Bench with three different algorithms, namely binary search, depth-first search, and breadth-first search, and to evaluate the sequential reasoning ability of 14 different LLMs, revealing several interesting findings.
- Fast AdvProp10
This paper introduces Fast AdvProp, which aggressively revamps AdvProp's costly training components, rendering the method nearly as cheap as the vanilla training, and shows that it is able to further model performance on a spectrum of visual benchmarks, without incurring extra training cost.
- Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails9
These findings demonstrate that alignment of LLM agents is not a static property but a fragile and dynamic one, vulnerable to feedback-driven decay during deployment, and that alignment benefits erode rapidly under self-evolution, with initially aligned models converging toward unaligned states.
- DeepVoting: An Explainable Framework for Semantic Part Detection under Partial Occlusion.9
A novel detection framework, named DeepVoting, which accumulates local visual cues, called visual concepts (VC), to locate the semantic parts of an object, and shows significantly better performance on semantic part detection in occlusion scenarios, compared with Faster-RCNN.
- Spiral RoPE: Rotate Your Rotary Positional Embeddings in the 2D Plane8
Spiral RoPE is proposed, a simple yet effective extension that enables multi-directional positional encoding by partitioning embedding channels into multiple groups associated with uniformly distributed directions, allowing spatial relationships to be encoded beyond the horizontal and vertical axes.
- 8
- Bag of Tricks for FGSM Adversarial Training7
Surprisingly, it is found that simple tricks, i.e., masking partial pixels, setting a large convolution stride and smooth activation functions, or regularizing the weights of the first convolutional layer, can effectively tackle the overfitting issue.
- VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation6
VLAA-GUI, a modular GUI agentic framework built around three integrated components that guide the system on when to Stop, Recover, and Search, achieves top performance on two benchmarks with Linux and Windows tasks.
- 6
- Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learning6
A new Prompt-Based Aggregation (PBA) is proposed to defend against the NAW attack in federated VLN, which provides the server with a ”prompt” of the vision-and-language alignment variance between the benign and malicious clients so that they can be distinguished during training.
- OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation5
A family of advanced vision encoder that learns a single, unified visual representation that can serve both image understanding and image generation, and provides empirical evidence that generation and understanding are mutually beneficial in the architecture.
- 5
- ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning5
A new self-supervised video representation learning framework, ARVideo, which autoregressively predicts the next video token in a tailored sequence order, and adopts a randomized spatiotemporal prediction order to facilitate learning from multi-dimensional data.
- 5
- 4
- FedConv: Enhancing Convolutional Neural Networks for Handling Data Heterogeneity in Federated Learning4
Findings indicate that with strategic architectural modifications, pure CNNs can achieve a level of robustness that either matches or even exceeds that of ViTs when handling heterogeneous data clients in FL.
- 4
- Prediction‐based MAC‐layer sensing in cognitive radio networks4
Simulation results show that the proposed prediction-based sensing scheduling algorithm could achieve a good tradeoff between network throughput and fairness among SUs.
- 3
- 3
- 3
- Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models3
A comparative analysis of the visual representations in existing vision-and-language models and vision-only models by probing on a broad range of tasks is conducted to shed light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.
- 3
- 2
- 2
- 2
- Reliable and Responsible Foundation Models: A Comprehensive Survey1
Critical issues, including bias and fairness, security and privacy, uncertainty, explainability, and distribution shift are explored, as well as methods like alignment and Artificial Intelligence-Generated Content (AIGC) detection.
- 1
- 1
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-15. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.