Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Lawrence CarinSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works164 from public data
- UNITER: UNiversal Image-TExt Representation Learning2,731
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2,008
An Attentional Generative Adversarial Network that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation and for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.
- Patient Knowledge Distillation for BERT Model Compression1,021
This work proposes a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student), which translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy.
- Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling836
Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that CLIPBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end- to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full- length videos, proving the proverbial less-is-more principle.
- GIT: A Generative Image-to-text Transformer for Vision and Language808
This paper design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering and establishes new state of the arts on 12 challenging benchmarks with a large margin.
- Ferret: Refer and Ground Anything Anywhere at Any Granularity625
The introduction of Ferret, a new Multimodal Large Language Model capable of understanding spatial referring of any shape or granularity within an image and accurately grounding open-vocabulary descriptions, and a novel and powerful hybrid region representation that integrates discrete coordinates and continuous features jointly to represent a region in the image.
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training573
HELP, a novel framework for large-scale video+language omni-representation learning that achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains is presented.
- 563
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning561
To enable large-scale training, VILLA adopts the "free" adversarial training strategy, and combines it with KL-divergence-based regularization to promote higher invariance in the embedding space.
- An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA559
This work proposes PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA, and treats GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge.
- An Empirical Study of Training End-to-End Vision-and-Language Transformers482
Meter, a Multimodal End-to-end TransformER framework, is presented, through which it is investigated how to design and pre-train a fully transformer-based VL model in an end- to-end manner and provides insights on how to train a performant VL transformer.
- UNITER: Learning UNiversal Image-TExt Representations479
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- Semantic Compositional Networks for Visual Captioning453
Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics.
- Generalized Decoding for Pixel, Image, and Language404
X-Decoder is the first work that provides a unified way to support all types of image segmentation and a variety of vision-language (VL) tasks with strong transferability to a wide range of downstream tasks in both zero-shot and finetuning settings.
- Relation-Aware Graph Attention Network for Visual Question Answering400
A Relation-aware Graph Attention Network (ReGAT), which encodes each image into a graph and models multi-type inter-object relations via a graph attention mechanism, to learn question-adaptive relation representations.
- Prompting GPT-3 To Be Reliable389
This systematic empirical study sheds new insights on the reliability of prompting LLMs, but more importantly, its prompting strategies can help practitioners more reliably use LLMs like GPT-3.
- SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning356
SwinBERT is an end-to-end transformer-based model for video captioning, which takes video frame patches directly as inputs, and outputs a natural language description, and adopts a video transformer to encode spatial-temporal representations that can adapt to variable lengths of video input without dedicated design for different frame rates.
- Scaling Up Vision-Language Pretraining for Image Captioning342
LEMON O, a LargE-scale iMage captiONer, is presented, and the first empirical study on the scaling behavior of VLP for image captioning is provided, and it is shown LEMON can generate captions with long-tail vi-sual concepts when used in a zero-shot manner.
- StoryGAN: A Sequential Conditional GAN for Story Visualization312
A new story-to-image-sequence generation model, StoryGAN, based on the sequential conditional GAN framework is proposed, which outperformed state-of-the-art models in image quality, contextual consistency metrics, and human evaluation.
- Discourse-Aware Neural Extractive Text Summarization306
DiscoBert extracts sub-sentential discourse units (instead of sentences) as candidates for extractive selection on a finer granularity and outperforms state-of-the-art methods by a significant margin on popular summarization benchmarks compared to other BERT-base models.
- 302
- 268
- 223
- ReCo: Region-Controlled Text-to-Image Generation220
The proposed model, dubbed as ReCo (Region-Controlled T2I), enables the region control for arbitrary objects described by open-ended regional texts rather than by object labels from a constrained category set, and can better control the object count, spatial relationship, and region attributes such as color/size, with the free-form regional description.
- Guiding Instruction-based Image Editing via Multimodal Large Language Models212
This work investigates how MLLMs facilitate edit instructions and presents MLLM-Guided Image Editing (MGIE), which learns to derive expressive instructions and provides explicit guidance and can lead to a notable improvement in automatic metrics and human evaluation while maintaining competitive inference efficiency.
- Hierarchical Graph Network for Multi-hop Question Answering205
Experiments on the HotpotQA benchmark demonstrate that the proposed model achieves new state of the art in both the Distractor and Fullwiki settings.
- Tactical Rewind: Self-Correction via Backtracking in Vision-And-Language Navigation188
The Frontier Aware Search with backTracking (FAST) Navigator is presented, a general framework for action decoding, that achieves state-of-the-art results on the 2018 Room-to-Room (R2R) Vision-and-Language navigation challenge.
- 154
- UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling154
UniTAB’s unified multi-task network and the task-agnostic output sequence design make the model parameter efficient and generalizable to new tasks, and could provide a more comprehensive and interpretable image description.
- Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization154
- Distilling Knowledge Learned in BERT for Text Generation147
A novel approach, Conditional Masked Language Modeling (C-MLM), is presented to enable the finetuning of BERT on target generation tasks, which significantly outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization.
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models144
VALUE (Vision-And-Language Understanding Evaluation), a set of meticulously designed probing tasks generalizable to standard pre-trained V+L models, aiming to decipher the inner workings of multimodal pre-training.
- 141
- Multi-Fact Correction in Abstractive Text Summarization135
Span-Fact is a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection to significantly boost the factual consistency of system- generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.
- Topic-Guided Variational Auto-Encoder for Text Generation135
Experimental results show that the TGVAE outperforms its competitors on both unconditional and conditional text generation, which can also generate semantically-meaningful sentences with various topics.
- Injecting Semantic Concepts into End-to-End Image Captioning134
This paper proposes a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features, and introduces a novel Concept Token Network (CTN) to predict the semantic concepts and then incorporate them into the end-to-end captioning.
- 133
- Multimodal Autoregressive Pre-training of Large Vision Encoders131
AIMV2 is presented, a family of generalist vision encoders characterized by a straightforward pre-training process, scalability, and remarkable performance across a range of downstream tasks by pairing the vision encoder with a multimodal decoder that autoregressively generates raw image patches and text tokens.
- 127
- Wasserstein Contrastive Representation Distillation121
Wasserstein Contrastive Representation Distillation (WCoRD) is proposed, which leverages both primal and dual forms of Wasserstein distance for KD, yielding a contrastive learning objective that maximizes the lower bound of mutual information between the teacher and the student networks.
- Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models116
Ferret-v2 is unveiled, a significant upgrade to Ferret, with three key designs, including a flexible approach that effortlessly handles higher image resolution, improving the model's ability to process and understand images in greater detail.
- Sequential Attention GAN for Interactive Image Editing114
A novel Sequential Attention Generative Adversarial Network (SeqAttnGAN) is proposed, which applies a neural state tracker to encode the previous image and the textual description in each turn of the sequence, and uses a GAN framework to generate a modified version of the image that is consistent with the preceding images and coherent with the description.
- Sequential Attention GAN for Interactive Image Editing via Dialogue.114
A novel Sequential Attention Generative Adversarial Network (SeqAttnGAN) is proposed, which applies a neural state tracker to encode the previous image and the textual description in each turn of the sequence, and uses a GAN framework to generate a modified version of the image that is consistent with the preceding images and coherent with the description.
- EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets113
By slimming the self-attention and fully-connected sub-layers inside a transformer, this work is the first to identify structured winning tickets in the early stage of BERT training.
- Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models113
Surprisingly, it is found that during dataset collection, non-expert annotators can easily attack SOTA VQA models successfully, revealing the fragility of these models while demonstrating the effectiveness of the adversarial dataset.
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog111
This paper presents a new model for visual dialog, Recurrent Dual Attention Network (ReDAN), using multi-step reasoning to answer a series of questions about an image, and demonstrates that ReDAN can locate context-relevant visual and textual clues via iterative refinement, which can lead to the correct answer step-by-step.
- Learning Deep Sigmoid Belief Networks with Data Augmentation111
Deep directed generative models are developed by stacking sigmoid belief networks, with sparsity-encouraging priors placed on the model parameters, and an online version of VB is developed to address large-scale datasets.
- NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis105
NUWA-Infinity is presented, a generative model for infinite visual synthesis, which is defined as the task of generating arbitrarily-sized high-resolution images or long-duration videos and has superior visual synthesis capabilities in terms of resolution and variable-size generation.
- LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling104
Surprisingly, experimental results show that this unified VidL framework LAVENDER achieves competitive performance on 14 VidL benchmarks, covering video question answering, text-to-video retrieval and video captioning.
- POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training103
POINTER (PrOgressive INsertion-based TransformER), a simple yet novel insertion-based approach for hard-constrained text generation, which achieves state-of-the-art performance on constrained text generation.
- Learning Generic Sentence Representations Using Convolutional Neural Networks101
Experimental results on several benchmark datasets, and across a broad range of applications, demonstrate the superiority of the proposed model over competing methods.
- Scalable Deep Poisson Factor Analysis for Topic Modeling100
A new framework for topic modeling is developed, based on deep graphical models, where interactions between topics are inferred through deep latent binary hierarchies, and Scalable inference algorithms are derived by applying Bayesian conditional density filtering algorithm.
- Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation99
Empirical results from both automatic and human evaluations demonstrate that the proposed hierarchically structured reinforced training achieves significantly better performance compared to a strong flat deep reinforcement learning baseline.
- 99
- An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling95
This work systematically examines the potential of MVM in the context of VidL learning, and conducts comprehensive experiments and provides insights into the factors leading to effective MVM training, resulting in an enhanced model VIOLETv2.
- 87
- Contrastive Distillation on Intermediate Representations for Language Model Compression87
CoDIR is proposed, a principled knowledge distillation framework where the student is trained to distill knowledge through intermediate layers of the teacher via a contrastive objective, and achieves superb performance on the GLUE benchmark, outperforming state-of-the-art compression methods.
- TIGEr: Text-to-Image Grounding for Image Caption Evaluation86
The empirical tests show that TIGEr has a higher consistency with human judgments than alternative existing metrics, and comprehensively assess the metric’s effectiveness in caption evaluation by measuring the correlation between human judgments and metric scores.
- 86
- Apple Intelligence Foundation Language Models: Tech Report 202585
Two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services are introduced, including a 3B-parameter on-device model optimized for Apple silicon through architectural innovations and a scalable server model built on a novel Parallel-Track Mixture-of-Experts PT-MoE transformer.
- Meta Module Network for Compositional Visual Reasoning85
This work proposes Meta Module Network (MMN), a more powerful NMN architecture centered on a novel meta module, which can take in function recipes and morph into diverse instance modules dynamically, inheriting the strong explainability and compositionality of NMN.
- FreeLB: Enhanced Adversarial Training for Language Understanding82
A novel adversarial training algorithm - FreeLB - is proposed, that promotes higher robustness and invariance in the embedding space, by adding adversarial perturbations to word embeddings and minimizing the resultant adversarial risk inside different regions around input samples.
- VeCLIP: Improving CLIP Training via Visual-Enriched Captions81
This study introduces a scalable pipeline for noisy caption rewriting with significant advantages in image-text alignment and overall model performance, and showcases the adaptation of this method for training CLIP on large-scale web-crawled datasets, termed VeCLIP.
- 81
- Violin: A Large-Scale Dataset for Video-and-Language Inference81
A new large-scale dataset, named Violin (VIdeO-and-Language INference), is introduced for this task, which consists of 95,322 video-hypothesis pairs from 15,887 video clips, spanning over 582 hours of video.
- Compressing LLMs: The Truth is Rarely Pure and Never Simple79
Knowledge-Intensive Compressed LLM BenchmarK (LLM-KICK), a collection of carefully curated tasks to redefine the evaluation protocol for compressed LLMs, which have significant alignment with their dense counterparts and perplexity fail to capture subtle change in their true capabilities.
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning77
A new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning is presented, and careful data curation and training strategies can yield strong performance even at small scales.
- What Makes A Good Story? Designing Composite Rewards for Visual Storytelling76
This paper proposes three assessment criteria: relevance, coherence and expressiveness, which are observed through empirical analysis could constitute a “high-quality” story to the human eye and proposes a reinforcement learning framework, ReCo-RL, with reward functions designed to capture the essence of these quality criteria.
- MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs72
This work introduces MIA-Bench, a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to strictly adhere to complex instructions, and explores supervised fine-tuning to enhance the models' ability to strictly follow instructions without compromising performance on other tasks.
- 72
- Adversarial Domain Adaptation for Machine Reading Comprehension67
An Adversarial Domain Adaptation framework for Machine Reading Comprehension (MRC), where pseudo questions are first generated for unlabeled passages in the target domain, and then a domain classifier is incorporated into an MRC model to predict which domain a given passage-question pair comes from.
- 66
- Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editing64
This work introduces Pico-Banana-400K, a comprehensive 400K-image dataset for instruction-based image editing, and establishes a robust foundation for training and benchmarking the next generation of text-guided image editing models.
- Playing Lottery Tickets with Vision and Language64
This work performs the first empirical study to assess whether trainable subnetworks also exist in pre-trained VL models, and finds that LTH remains relevant when using other training methods (e.g., adversarial training).
- Domain Adaptive Text Style Transfer62
This paper proposes simple yet effective domain adaptive text style transfer models, enabling domain-adaptive information exchange and evaluates the effectiveness of the proposed models on two style transfer tasks over multiple target domains where only limited non-parallel data is available.
- FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding61
FILTER is proposed, an enhanced fusion method that takes cross-lingual data as input for XLM finetuning and proposes an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
- 59
- Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms58
This paper introduces Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV, and shows strong cross-platform transfer capabilities.
- 58
- 58
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons55
GEA is a single unified model capable of grounding itself across these varied domains through a multi-embodiment action tokenizer and achieves strong generalization performance to unseen tasks across diverse benchmarks compared to other generalist models and benchmark-specific approaches.
- 55
- 54
- 54
- 49
- 49
- Contrastive Localized Language-Image Pre-Training47
A pre-training method called Contrastive Localized Language-Image Pre-training (CLOC) is proposed by complementing CLIP with region-text contrastive loss and modules, which enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks.
- DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search46
DeepMMSearch-R1 is presented, the first multimodal LLM capable of performing on-demand, multi-turn web searches and dynamically crafting queries for both image and text search tools and introduces DeepMMSearchVQA, a novel multimodal VQA dataset created through an automated pipeline intermixed with real-world information from web search tools.
- Efficient Robust Training via Backward Smoothing46
This work develops a new understanding towards Fast Adversarial Training, by viewing random initialization as performing randomized smoothing for better optimization of the inner maximization problem, and proposes a new initialization strategy, backward smoothing, to further improve the stability and model robustness over single-step robust training methods.
- BachGAN: High-Resolution Image Synthesis From Salient Object Layout44
The proposed Background Hallucination Generative Adversarial Network (BachGAN), which leverages a background retrieval module to first select a set of segmentation maps from a large candidate pool, then encodes these candidate layouts via a background fusion module to hallucinate a suitable background for the given objects.
- Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling42
This paper leverages recent advances in stochastic gradient Markov Chain Monte Carlo to learn weight uncertainty in RNNs and yields a principled Bayesian learning algorithm, adding gradient noise during training and model averaging when testing.
- Non-Contrastive Learning Meets Language-Image Pre-Training39
This work empirically observe that the non-contrastive objective benefits representation learning while sufficiently underperforming under zero-shot recognition, and introduces xCLIP, a multi-tasking framework combining CLIP and n CLIP, and shows that nCLIP aids CLIP in enhancing feature semantics.
- 39
- 38
- APo-VAE: Text Generation in Hyperbolic Space36
Extensive experiments in language modeling, unaligned style transfer, and dialog-response generation demonstrate the effectiveness of the proposed APo-VAE model over VAEs in Euclidean latent space, thanks to its superb capabilities in capturing latent language hierarchies in hyperbolic space.
- Contrastively Smoothed Class Alignment for Unsupervised Domain Adaptation36
A novel Contrastively Smoothed Class Alignment (CoSCA) model is proposed, that explicitly incorporates both intra- and inter-class domain discrepancy to better align ambiguous target samples with the source domain.
- Contextual Text Style Transfer36
A Context-Aware Style Transfer (CAST) model is proposed, which uses two separate encoders for each input sentence and its surrounding context and a classifier is further trained to ensure contextual consistency of the generated sentence.
- Deep Poisson factor modeling36
A new deep architecture for topic modeling, based on Poisson Factor Analysis (PFA) modules, that derives efficient inference via MCMC and stochastic variational methods, that scale with the number of non-zeros in the data and binary units, yielding significant efficiency, relative to models based on logistic links.
- Exploring Discrete Diffusion Models for Image Captioning35
A diffusion-based captioning model, dubbed the name DDCap, to allow more decoding flexibility, and proposes several key techniques including best-first inference, concentrated attention mask, text length prediction, and image-free training.
- Cluster-Former: Clustering-based Sparse Transformer for Question Answering35
Cluster-Former is proposed, a novel clustering-based sparse Transformer to perform attention across chunked sequences that allows information integration beyond local windows, which is especially beneficial for question answering (QA) tasks that rely on long-range dependencies.
- UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action32
This work establishes a scalable paradigm bridging primitive GUI interactions and high-level tool intelligence, enabling more resilient and adaptable computer use agents for diverse environments and complex real-world tasks.
- Distilling the Knowledge of BERT for Text Generation31
This paper proposes a new task, Conditional Masked Language Modeling (C-MLM), to enable fine-tuning of BERT on target text-generation dataset, and significantly outperforms strong baselines of Transformer on multiple text generation tasks, including machine translation (MT) and text summarization.
- POINTER: Constrained Text Generation via Insertion-based Generative Pre-training30
POINTER, a simple yet novel insertion-based approach for hard-constrained text generation, operates by progressively inserting new tokens between existing tokens in a parallel manner, and achieves state-of-the-art performance on constrained text generation.
- Adaptive Feature Abstraction for Translating Video to Text29
A novel attention mechanism is developed, that adaptively and sequentially focuses on different layers of CNN features (levels of feature "abstraction"), as well as local spatiotemporal regions of the feature maps at each layer.
- MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA27
MM-Ego is built, an egocentric multimodal LLM that shows powerful performance on egocentric video understanding and includes a specialized multimodal architecture featuring a novel"Memory Pointer Prompting"mechanism that enables the model to more effectively comprehend extended video content.
- 27
- 26
- Discourse-Aware Neural Extractive Model for Text Summarization.25
A graph-based discourse-aware neural summarization model that outperforms state-of-the-art methods by a significant margin and provides a fine-grained granularity for extractive selection, which helps reduce redundancy in extracted summaries.
- 23
- UniVG: A Generalist Diffusion Model for Unified Image Generation and Editing22
UniVG is introduced, a generalist diffusion model capable of supporting a diverse range of image generation tasks with a single set of weights and can even outperform some task-specific models on their respective benchmarks, marking a significant step towards a unified image generation model.
- 22
- Understanding Alignment in Multimodal LLMs: A Comprehensive Study21
A novel way of creating multimodal preference data called Bias-Driven Hallucination Sampling (BDHS) is introduced that needs neither additional annotation nor external models, and can achieve competitive performance to previously published alignment work for multimodal models across a range of benchmarks.
- 21
- 21
- MagGAN: High-Resolution Face Attribute Editing with Mask-Guided Generative Adversarial Network20
This work presents Mask-guided Generative Adversarial Network (MagGAN) for high-resolution face attribute editing, in which semantic facial masks from a pre-trained face parser are used to guide the fine-grained image editing process.
- Cross-Thought for Sentence Encoder Pre-training20
This paper proposes Cross-Thought, a novel approach to pre-training sequence encoder, which is instrumental in building reusable sequence embeddings for large-scale NLP tasks such as question answering and textual entailment tasks.
- Unsupervised Learning of Sentence Representations using Convolutional Neural Networks20
Experimental results on several benchmark datasets, and across a broad range of applications, demonstrate the superiority of the proposed model over competing methods.
- 19
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer17
Manzano is a simple and scalable unified framework that substantially reduces tension by coupling a hybrid image tokenizer with a well-curated training recipe, and achieves state-of-the-art results among unified models, and is competitive with specialist models, particularly on text-rich evaluation.
- 17
- Improving Adversarial Text Generation by Modeling the Distant Future17
This work proposes a novel guider network to focus on the generative process over a longer horizon, which can assist next-word prediction and provide intermediate rewards for generator optimization.
- 17
- Character-level deep conflation for business data analytics17
A character-level deep conflation model that encodes the input text strings from character level into finite dimension feature vectors, which are then used to compute the cosine similarity between the text strings.
- 16
- 16
- Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation15
LayoutBench is proposed, a diagnostic benchmark for layout-guided image generation that examines four categories of spatial control skills: number, position, size, and shape, and IterInpaint is proposed, a new baseline that generates foreground and background regions step-by-step via inpainting, demonstrating stronger generalizability than existing models on OOD layouts in LayoutBench.
- Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models15
This work proposes a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models, and reveals that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone.
- Graph-Driven Generative Models for Heterogeneous Multi-Task Learning14
A novel graph-driven generative model, that unifies multiple heterogeneous learning tasks into the same framework, with a focus on healthcare applications (tasks), that successfully leverages information across different tasks, boosting performance in all tasks and outperforming existing state-of-the-art approaches.
- Adaptive Feature Abstraction for Translating Video to Language14
These experiments quantitatively demonstrate the effectiveness of the proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantics.
- 13
- 12
- 11
- 11
- 10
- MOFI: Learning Image Representations from Noisy Entity Annotated Images9
Experiments show that supervised pre-training with large-scale fine-grained entity labels is highly effective for image retrieval tasks, and multi-task training further improves the performance.
- 9
- 9
- 8
- Unsupervised Learning with Truncated Gaussian Graphical Models8
This paper introduces a novel variant of GGMs, which relaxes the Gaussian restriction and yet admits efficient inference, and imposes a bipartite structure on the GGM and governs the hidden variables by truncated normal distributions.
- 7
- 7
- 7
- Modified DCTNet for audio signals classification7
Experimental results show that the A-DCTNet and Recurrent Neural Networks (RNN) achieve state-of-the-art performance in bird song classification rate, and improve artist identification accuracy in music data.
- 6
- 6
- PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection5
PRISM-Bench is introduced, a benchmark of puzzle-based visual challenges designed to evaluate not only whether models can solve problems, but how their reasoning unfolds, and underscores the need for diagnostic evaluation protocols in the development of trustworthy MLLMs.
- SO-Bench: A Structural Output Evaluation of Multimodal LLMs5
This work conducts a comprehensive study of visual structural output capabilities for MLLMs with a carefully designed SO-Bench benchmark, and conducts training experiments to largely improve the model's structured output capability.
- InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Models5
A visual dialogue dataset, named InfoVisDial, which provides rich informative answers in each round even with external knowledge related to the visual content, and proposes a strong baseline by adapting the GIT model for the visual dialogue task and fine-tune the model on InfoVisDial.
- 5
- 5
- 5
- Adaptive Learning Rates with Maximum Variation Averaging4
An adaptive learning rate rule is proposed in which the running mean squared gradient is replaced by a weighted mean, with weights chosen to maximize the estimated variance of each coordinate, which gives a worst-case estimate for the local gradient variance.
- MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients3
An adaptive learning rate principle is proposed, in which the running mean of squared gradient in Adam is replaced by a weighted mean, with weights chosen to maximize the estimated variance of each coordinate, which results in a faster adaptation to the local gradient variance, which leads to more desirable empirical convergence behaviors than Adam.
- 3
- 3
- Inference of gene networks associated with the host response to infectious disease2
Experimental results on two realworld microarray-based gene expression datasets show that the proposed framework achieves comparatively superior classification performance, with model interpretation delivered via pathway association analysis.
- Taming Outlier Tokens in Diffusion Transformers1
Dual-Stage Registers (DSR) are introduced, a register-based intervention for both components: trained registers when available, recursive test-time registers otherwise, and diffusion registers for the denoiser that consistently reduce outlier artifacts and improve generation quality.
- 1
- 1
- –
- Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling–
The Length Value Model (LenVM), a token-level framework that estimates the remaining generation length at every decoding step, is introduced, establishing generation length as an effective scalable token-level value signal for control, prediction, and future value-based training.
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-14. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.