Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileWorks69 from public data
- UNITER: UNiversal Image-TExt Representation Learning2,731
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning561
To enable large-scale training, VILLA adopts the "free" adversarial training strategy, and combines it with KL-divergence-based regularization to promote higher invariance in the embedding space.
- UNITER: Learning UNiversal Image-TExt Representations479
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- StoryGAN: A Sequential Conditional GAN for Story Visualization312
A new story-to-image-sequence generation model, StoryGAN, based on the sequential conditional GAN framework is proposed, which outperformed state-of-the-art models in image quality, contextual consistency metrics, and human evaluation.
- Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning301
This work introduces adversarial training into self-supervision, to provide general-purpose robust Pretrained models for the first time, and finds these robust pretrained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine- Tuning.
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models144
VALUE (Vision-And-Language Understanding Evaluation), a set of meticulously designed probing tasks generalizable to standard pre-trained V+L models, aiming to decipher the inner workings of multimodal pre-training.
- A Survey of Attacks on Large Vision–Language Models: Resources, Advances, and Future Trends130
This article provides a comprehensive review of the various forms of existing LVLM attacks, and systematically review the development of LVLM attack methods, such as adversarial attacks that manipulate model outputs, jailbreak attacks that exploit model vulnerabilities for unauthorized actions, and data poisoning that affects model training.
- UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training114
UC2, the first machine translation-augmented framework for cross-lingual cross-modal representation learning, is introduced, to tackle the scarcity problem of multilingual captions for image datasets and facilitate the learning of a joint embedding space of images and all languages of interest.
- Sequential Attention GAN for Interactive Image Editing114
A novel Sequential Attention Generative Adversarial Network (SeqAttnGAN) is proposed, which applies a neural state tracker to encode the previous image and the textual description in each turn of the sequence, and uses a GAN framework to generate a modified version of the image that is consistent with the preceding images and coherent with the description.
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog111
This paper presents a new model for visual dialog, Recurrent Dual Attention Network (ReDAN), using multi-step reasoning to answer a series of questions about an image, and demonstrates that ReDAN can locate context-relevant visual and textual clues via iterative refinement, which can lead to the correct answer step-by-step.
- Fine-grained Iterative Attention Network for Temporal Language Localization in Videos106
A Fine-grained Iterative Attention Network (FIAN) that consists of an iterative attention module for bilateral query-video in-formation extraction and a content-oriented localization strategy instead of applying recent anchor-based localization is proposed.
- Memory-Guided Semantic Learning Network for Temporal Sentence Grounding99
A memory-augmented network, called Memory-Guided Semantic Learning Network (MGSL-Net), that learns and memorizes the rarely appeared content in TSG task and substantially improves the accuracy not only on the entire dataset but also on the rare cases.
- Meta Module Network for Compositional Visual Reasoning85
This work proposes Meta Module Network (MMN), a more powerful NMN architecture centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically, inheriting the strong explainability and compositionality of NMN.
- FreeLB: Enhanced Adversarial Training for Language Understanding82
A novel adversarial training algorithm - FreeLB - is proposed, that promotes higher robustness and invariance in the embedding space, by adding adversarial perturbations to word embeddings and minimizing the resultant adversarial risk inside different regions around input samples.
- Violin: A Large-Scale Dataset for Video-and-Language Inference81
A new large-scale dataset, named Violin (VIdeO-and-Language INference), is introduced for this task, which consists of 95,322 video-hypothesis pairs from 15,887 video clips, spanning over 582 hours of video.
- What Makes A Good Story? Designing Composite Rewards for Visual Storytelling76
This paper proposes three assessment criteria: relevance, coherence and expressiveness, which are observed through empirical analysis could constitute a “high-quality” story to the human eye and proposes a reinforcement learning framework, ReCo-RL, with reward functions designed to capture the essence of these quality criteria.
- 72
- Unsupervised Temporal Video Grounding with Deep Semantic Clustering62
This paper proposes a novel Deep Semantic Clustering Network (DSCNet) to leverage all semantic information from the whole query set to compose the possible activity in each video for grounding and demonstrates that the DSCNet achieves competitive performance, and even outperforms most weakly-supervised approaches.
- 58
- Benchmarking all-atom biomolecular structure prediction with FoldBench54
FoldBench is introduced, an extensive benchmark dataset consisting of 1522 biological assemblies categorized into nine distinct prediction tasks, providing crucial insights and benchmarks to inform future model development and refinement, and highlighting significant advancements yet revealing persistent limitations within the field.
- 51
- Adversarial Category Alignment Network for Cross-domain Sentiment Classification50
An adversarial category alignment network (ACAN) is proposed, which attempts to enhance category consistency between the source domain and the target domain, and increases the discrepancy of two polarity classifiers to provide diverse views.
- BachGAN: High-Resolution Image Synthesis From Salient Object Layout44
The proposed Background Hallucination Generative Adversarial Network (BachGAN), which leverages a background retrieval module to first select a set of segmentation maps from a large candidate pool, then encodes these candidate layouts via a background fusion module to hallucinate a suitable background for the given objects.
- Deep Co-Attention Network for Multi-View Subspace Learning39
A deep co-attention network for multi-view subspace learning is proposed, which aims to extract both the common information and the complementary information in an adversarial setting and provide robust interpretations behind the prediction to the end-users via the co-Attention mechanism.
- 39
- Contrastively Smoothed Class Alignment for Unsupervised Domain Adaptation36
A novel Contrastively Smoothed Class Alignment (CoSCA) model is proposed, that explicitly incorporates both intra- and inter-class domain discrepancy to better align ambiguous target samples with the source domain.
- Contextual Text Style Transfer36
A Context-Aware Style Transfer (CAST) model is proposed, which uses two separate encoders for each input sentence and its surrounding context and a classifier is further trained to ensure contextual consistency of the generated sentence.
- LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training35
This study thoroughly investigates the sparsity of the dense LLaMA model by constructing MoE for both the attention and MLP modules in the transformer blocks and designs a two-stage post-training strategy to enhance model performance.
- Cryo-EM reveals mechanisms of natural RNA multivalency33
Cryo-electron microscopy is used to determine structures from four RNA families to reveal a plethora of structural motifs that contribute to RNA multivalency, including kissing loop, palindromic base-pairing, A-stacking, metal ion coordination, pseudoknot and minor-groove interactions.
- 30
- Discourse-Aware Neural Extractive Model for Text Summarization.25
A graph-based discourse-aware neural summarization model that outperforms state-of-the-art methods by a significant margin and provides a fine-grained granularity for extractive selection, which helps reduce redundancy in extracted summaries.
- Cross-Thought for Sentence Encoder Pre-training20
This paper proposes Cross-Thought, a novel approach to pre-training sequence encoder, which is instrumental in building reusable sequence embeddings for large-scale NLP tasks such as question answering and textual entailment tasks.
- 20
- 19
- 16
- Credit Risk Identification in Supply Chains Using Generative Adversarial Networks13
Experimental results demonstrate that the GAN-based model outperforms traditional methods, including logistic regression, decision trees, and neural networks, achieving superior accuracy, recall, and F1 scores.
- A fast and robust face recognition approach combining Gabor learned dictionaries and collaborative representation10
Experiments on two benchmark face datasets show that the proposed method can achieve high classification accuracy and is comparatively low in terms of time-consumption, compared to the CRC-RLS algorithm.
- IBM Research and Columbia University TRECVID-2013 Multimedia Event Detection (MED), Multimedia Event Recounting (MER), Surveillance Event Detection (SED), and Semantic Indexing (SIN) Systems.9
The current IBM-Columbia MER system exploits nine observations about human cognition, language, and visual perception in order to produce an effective video recounting of an event, and achieves better results on the retrospective and interactive tasks than last year.
- 8
- 8
- 7
- 7
- 7
- MDF-Net: multilayer feature dynamic interactive fusion network for few-shot fine-grained image classification5
The multilayer feature dynamic interaction fusion network (MDF-Net) adaptively fuses shallow local details with deep semantic features using cross-scale covariance correlation computation and residual-guided bidirectional attention propagation to address the imbalance between deep and shallow representations.
- Improved local-feature-based few-shot learning with Sinkhorn metrics5
The Sinkhorn algorithm is exploited to solve the ROT problem, which is efficient for the end-to-end training of local-feature-based FSL and achieves a series of state-of-the-arts on multiple datasets in both the single-domain and cross-domain FSL scenarios.
- 5
- Adaptive Learning Rates with Maximum Variation Averaging4
An adaptive learning rate rule is proposed in which the running mean squared gradient is replaced by a weighted mean, with weights chosen to maximize the estimated variance of each coordinate, which gives a worst-case estimate for the local gradient variance.
- 4
- 4
- MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients3
An adaptive learning rate principle is proposed, in which the running mean of squared gradient in Adam is replaced by a weighted mean, with weights chosen to maximize the estimated variance of each coordinate, which results in a faster adaptation to the local gradient variance, which leads to more desirable empirical convergence behaviors than Adam.
- 3
- 2
- Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures1
A comprehensive, reproducible benchmark that offers diverse distilled checkpoints and an extensible framework to facilitate future research, and establishes a comprehensive, reproducible benchmark that offers diverse distilled checkpoints and an extensible framework to facilitate future research.
- 1
- INSET: Sentence Infilling with Inter-sentential Generative Pre-training1
This paper presents a framework to decouple this challenge and address these three aspects of missing sentence generation, leveraging the power of existing large-scale pre-trained models such as BERT and GPT-2.
- Longitudinal planning for personalized health management using daily behavioral data1
An integrated framework that unifies dynamic modeling, sparse learning, dictionary learning and matrix completion to translate users’ behavioral data into personalized dynamic system models and use them as constraints for deriving deeply personalized longitudinal health plans is developed.
- 1
- Bridging Disentanglement with Independence and Conditional Independence via Mutual Information for Representation Learning.1
The proposed encoder, called Split Encoder, can be applied into models that penalize total correlation, and shows significant improvement in unsupervised learning of disentangled representations and reconstructions.
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-15. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.