Jaehong Yoon lists you as their postdoctoral advisor. Claim this profile to confirm it and keep the rest of your record right.
Claim this profileAcademic lineage
View as a treeStudents and postdocs1
Works411 from public data
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers3,068
The LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework, a large-scale Transformer model that consists of three encoders, achieves the state-of-the-art results on two visual question answering datasets and shows the generalizability of the pre-trained cross-modality model.
- End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures1,320
A novel end-to-end neural model to extract entities and relations between them and compares favorably to the state-of-the-art CNN based model (in F1-score) on nominal relation classification (SemEval-2010 Task 8).
- Adversarial NLI: A New Benchmark for Natural Language Understanding1,300
This work introduces a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure, and shows that non-expert annotators are successful at finding their weaknesses.
- MAttNet: Modular Attention Network for Referring Expression Comprehension1,003
This work proposes to decompose expressions into three modular components related to subject appearance, location, and relationship to other objects, which allows for flexibly adapt to expressions containing different types of information in an end-to-end framework.
- Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling836
Experiments on text-to-video retrieval and video question answering on six datasets demonstrate that CLIPBERT outperforms (or is on par with) existing methods that exploit full-length videos, suggesting that end- to-end learning with just a few sparsely sampled clips is often more accurate than using densely extracted offline features from full- length videos, proving the proverbial less-is-more principle.
- TVQA: Localized, Compositional Video Question Answering821
This paper presents TVQA, a large-scale video QA dataset based on 6 popular TV shows, and provides analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVZA task.
- Dynabench: Rethinking Benchmarking in NLP654
It is argued that Dynabench addresses a critical need in the community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios.
- Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting618
An accurate and fast summarization model that first selects salient sentences and then rewrites them abstractively to generate a concise overall summary is proposed, which achieves the new state-of-the-art on all metrics on the CNN/Daily Mail dataset, as well as significantly higher abstractiveness scores.
- 544
- VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks521
The results demonstrate that training the adapter with the weight-sharing technique can match the performance of fine-tuning the entire model, and enhance the efficiency and performance of adapters by sharing their weights to attain knowledge across tasks.
- Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout422
This paper presents a generalizable navigational agent, trained in two stages via mixed imitation and reinforcement learning, outperforming the state-of-art approaches by a large margin on the private unseen test set of the Room-to-Room task, and achieving the top rank on the leaderboard.
- TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval388
The proposed XML model uses a late fusion design with a novel Convolutional Start-End detector (ConvSE), surpassing baselines by a large margin and with better efficiency, providing a strong starting point for future work.
- Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?386
Human subject tests are carried out that are the first of their kind to isolate the effect of algorithmic explanations on a key aspect of model interpretability, simulatability, while avoiding important confounding experimental factors.
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models381
This work proposes a simple yet powerful algorithm, LVLM Hallucination Revisor (LURE), to post-hoc rectify object hallucination in LVLMs by reconstructing less hallucinatory descriptions and consistently ranks at the top in both GPT and human evaluations.
- Rethinking machine unlearning for large language models364
An effective assessment framework for LLM unlearning is outlined and its applications in copyright and privacy safeguards and sociotechnical harm reduction are explored and revisiting methodologies and overlooked principles for future improvements are revisited.
- Combining Fact Extraction and Verification with Neural Semantic Matching Networks361
This paper presents a connected system consisting of three homogeneous neural semantic matching models that conduct document retrieval, sentence selection, and claim verification jointly for fact extraction and verification.
- DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models323
It is demonstrated that recent text-to-image generation models learn specific biases about gender and skin tone from web image-text pairs, which will help guide future progress in improving text-to-image generation models on visual reasoning skills and learning socially unbiased representations.
- VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos319
VideoTree is proposed, a training-free framework which builds a query-adaptive and hierarchical video representation for LLM reasoning over long-form videos that outperforms existing training-free approaches on EgoSchema and NExT-QA with less inference time.
- Tailoring Continuous Word Representations for Dependency Parsing313
It is found that all embeddings yield significant parsing gains, including some recent ones that can be trained in a fraction of the time of others, suggesting their complementarity.
- A Joint Speaker-Listener-Reinforcer Model for Referring Expressions295
A unified framework for the tasks of referring expression comprehension and generation is proposed, composed of three modules: speaker, listener, and reinforcer, which achieves state-of-the-art results for both comprehension andgeneration on three referring expression datasets.
- What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment295
An end-to-end, domain-independent neural encoder-aligner-decoder model for selective generation, i.e., the joint task of content selection and surface realization, achieves the best selection and generation results reported to-date on the benchmark WeatherGov dataset, despite using no specialized features or linguistic resources.
- TVQA+: Spatio-Temporal Grounding for Video Question Answering293
By performing this joint task, the proposed Spatio-Temporal Answerer with Grounded Evidence with STAGE model is able to produce insightful and interpretable spatio-temporal attention visualizations.
- From Paraphrase Database to Compositional Paraphrase Model and Back291
This work proposes models to leverage the phrase pairs from the Paraphrase Database to build parametric paraphrase models that score paraphrase pairs more accurately than the PPDB’s internal scores while simultaneously improving its coverage.
- HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification282
It is shown that the performance of an existing state-of-the-art semantic-matching model degrades significantly on this dataset as the number of reasoning hops increases, hence demonstrating the necessity of many-hop reasoning to achieve strong results.
- An Empirical Survey of Data Augmentation for Limited Data Learning in NLP264
An empirical survey of recent progress on data augmentation for NLP in the limited labeled data setting is provided, summarizing the landscape of methods and carrying out experiments on 11 datasets covering topics/news classification, inference tasks, paraphrasing tasks, and single-sentence tasks.
- Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences247
This work introduces a multi-level aligner that empowers the alignment-based encoder-decoder model with long short-term memory recurrent neural networks (LSTM-RNN) to translate natural language instructions to action sequences based upon a representation of the observable world state.
- 225
- Multi-Reward Reinforced Summarization with Saliency and Entailment214
This work addresses three important aspects of a good summary via a reinforcement learning approach with two novel reward functions: ROUGESal and Entail, on top of a coverage-based baseline, and shows superior performance improvement when these rewards are combined with traditional metric (ROUGE) based rewards.
- MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning210
Memory-Augmented Recurrent Transformer (MART) uses a memory module to augment the transformer architecture to generate a highly summarized memory state from the video segments and the sentence history so as to help better prediction of the next sentence (w.r.t. coreference and repetition aspects).
- Unifying Vision, Text, and Layout for Universal Document Processing209
To the best of the knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization.
- What Can We Learn from Collective Human Opinions on Natural Language Inference Data?207
This work collects ChaosNLI, a dataset with a total of 464,500 annotations to study Collective HumAn OpinionS in oft-used NLI evaluation sets, and argues for a detailed examination of human agreement in future data collection efforts, and evaluating model outputs against the distribution over collective human opinions.
- Charagram: Embedding Words and Sentences via Character n-grams197
It is demonstrated that Charagram embeddings outperform more complex architectures based on character-level recurrent and convolutional neural networks, achieving new state-of-the-art performance on several similarity tasks.
- Commonsense for Generative Multi-Hop Question Answering Tasks193
This work focuses on a more challenging multi-hop generative task (NarrativeQA), which requires the model to reason, gather, and synthesize disjoint pieces of information within the context to generate an answer.
- Polite Dialogue Generation Without Parallel Data186
Human evaluation validates that while the Fusion and the retrieval-based models achieve politeness with poorer context-relevance, the LFT and Polite-RL models can produce significantly more polite responses without sacrificing dialogue quality.
- Scaling Data Generation in Vision-and-Language Navigation184
This work proposes an effective paradigm for generating large-scale data for learning, which applies 1200+ photo-realistic environments from HM3D and Gibson datasets and synthesizes 4.9 million instruction-trajectory pairs using fully-accessible resources on the web.
- 179
- Robustness Gym: Unifying the NLP Evaluation Landscape171
Robustness Gym (RG), a simple and extensible evaluation toolkit that unifies 4 standard evaluation paradigms: subpopulations, transformations, evaluation sets, and adversarial attacks, is proposed.
- Evaluating the Factual Consistency of Large Language Models Through News Summarization162
A new benchmark called FIB(Factual Inconsistency Benchmark) that focuses on the task of summarization, and finds that existing LLMs generally assign a higher score to factually consistent summaries than tofactually inconsistent summaries.
- Improving and Simplifying Pattern Exploiting Training158
This paper focuses on few-shot learning without any unlabeled data and introduces ADAPET, which modifies PET’s objective to provide denser supervision during fine-tuning and outperforms PET on SuperGLUE without any task-specific unlabeling data.
- FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging157
This work presents FastIF, a set of simple modifications to influence functions that significantly improves their run-time, and uses k-Nearest Neighbors to narrow the search space down to a subset of good candidate data points, identify the configurations that best balance the speed-quality trade-off in estimating the inverse Hessian-vector product, and introduces a fast parallel variant.
- Vision Transformers are Parameter-Efficient Audio-Visual Learners148
This work proposes a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio- visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT, eliminating the quadratic cost of standard cross-attention.
- 147
- Revealing the Importance of Semantic Retrieval for Machine Reading at Scale147
This work proposes a simple yet effective pipeline system with special consideration on hierarchical semantic retrieval at both paragraph and sentence level, and their potential effects on the downstream task, and illustrates that intermediate semantic retrieval modules are vital for shaping upstream data distribution and providing better data for downstream modeling.
- SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation145
This work proposes SAFREE, a novel, training-free approach for safe T2I and T2V, that does not alter the model's weights and incorporates a novel self-validating filtering mechanism that dynamically adjusts the denoising steps when applying the filtered embeddings.
- Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering145
This paper proposes two semantics-enhanced rewards obtained from downstream question paraphrasing and question answering tasks to regularize the QG model to generate semantically valid questions, and proposes a QA-based evaluation method which measures the model’s ability to mimic human annotators in generating QA training data.
- Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation143
This work improves important aspects of abstractive summarization via multi-task learning with the auxiliary tasks of question generation and entailment generation with statistically significant improvements over the state-of-the-art on both the CNN/DailyMail and Gigaword datasets, as well as on the DUC-2002 transfer setup.
- Who did What: A Large-Scale Person-Centered Cloze Dataset142
A new "Who-did-What" dataset of over 200,000 fill-in-the-gap (cloze) multiple choice reading comprehension problems constructed from the LDC English Gigaword newswire corpus is constructed and proposed as a challenge task for the community.
- Multi-Target Embodied Question Answering137
This work presents a generalization of EQA -- Multi-Target EQA (MT-EQA), and proposes a modular architecture composed of a program generator, a controller, a navigator, and a VQA module that can outperform previous methods and strong baselines by a significant margin.
- Deep Multilingual Correlation for Improved Word Embeddings137
Deep non-linear transformations of word embeddings of the two languages are learned, using the recently proposed deep canonical correlation analysis, to improve their quality and consistency on multiple word and bigram similarity tasks.
- Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision135
A technique named "vokenization" is developed that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images (which the authors call "vokens").
- Envedit: Environment Editing for Vision-and-Language Navigation134
This work proposes Envedit, a data augmentation method that cre-ates new environments by editing existing environments, which are used to train a more generalizable agent and ensemble the VLN agents augmented on different edited environments and show that these edit methods are complementary.
- Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?130
A leakage-adjusted simulatability (LAS) metric is introduced for evaluating NL explanations, which measures how well explanations help an observer predict a model’s output, while controlling for how explanations can directly leak the output.
- Shortcut-Stacked Sentence Encoders for Multi-Domain Inference130
This work presents a simple sequential sentence encoder based on stacked bidirectional LSTM-RNNs with shortcut connections and fine-tuning of word embeddings that achieves the new state-of-the-art encoding result on the original SNLI dataset.
- Multi-Source Domain Adaptation for Text Classification via DistanceNet-Bandits129
A study of various distance-based measures in the context of NLP tasks, that characterize the dissimilarity between domains based on sample estimates, and develops a DistanceNet model which uses these distance measures as an additional loss function to be minimized jointly with the task's loss function, so as to achieve better unsupervised domain adaptation.
- The Role of Context Types and Dimensionality in Learning Word Embeddings126
The first extensive evaluation of how using different types of context to learn skip-gram word embeddings affects performance on a wide range of intrinsic and extrinsic NLP tasks suggests that while intrinsic tasks tend to exhibit a clear preference to particular types of contexts and higher dimensionality, more careful tuning is required for finding the optimal settings for most of the extrinsics tasks.
- Robust Machine Comprehension Models via Adversarial Training125
A novel alternative adversary-generation algorithm is proposed, AddSentDiverse, that significantly increases the variance within the adversarial training data by providing effective examples that punish the model for making certain superficial assumptions, and can make a state-of-the-art model significantly more robust.
- Detecting Linguistic Characteristics of Alzheimer’s Dementia by Interpreting Neural Models121
This work uses NLP techniques to classify and analyze the linguistic characteristics ofAD patients using the DementiaBank dataset, and shows that first derivative saliency can not only rediscover previous language patterns of AD patients, but also shed light on the limitations of neural models.
- Multi-Task Video Captioning with Video and Entailment Generation121
This work improves video captioning by sharing knowledge with two related directed-generation tasks: a temporally-directed unsupervised video prediction task to learn richer context-aware video encoder representations, and a logically-directed language entailment generation task to learning better video-entailing caption decoder representations.
- Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA118
This paper shows that in the multi-hop HotpotQA dataset, the examples often contain reasoning shortcuts through which models can directly locate the answer by word-matching the question with a sentence in the context, and shows that the 2-hop model trained on the regular data is more robust to the adversaries than the baseline.
- Reinforced Video Captioning with Entailment Rewards118
This work proposes a novel entailment-enhanced reward (CIDEnt) that corrects phrase-matching based metrics (such as CIDEr) to only allow for logically-implied partial matches and avoid contradictions, achieving further significant improvements over the C IDEr-reward model.
- Spoken language interaction with robots: Recommendations for future research113
This report identifies key scientific and engineering advances needed to enable robots to communicate in new environments, for new tasks, and with diverse user populations, without extensive re-engineering or the collection of massive training data.
- Hierarchical Video-Moment Retrieval and Step-Captioning111
The HIREST (HIerarchical REtrieval and STep-captioning) dataset is presented and a new benchmark that covers hierarchical information retrieval and visual/textual stepwise summarization from an instructional video corpus is proposed.
- Machine Comprehension with Syntax, Frames, and Semantics103
This work demonstrates significant improvement on the MCTest question answering task (Richardson et al., 2013) by augmenting baseline features with features based on syntax, frame semantics, coreference, and word embeddings, and combining them in a max-margin learning framework.
- 103
- VindLU: A Recipe for Effective Video-and-Language Pretraining102
A thorough empirical study demystifying the most important factors in the VidL model design and develops a step-by-step recipe, dubbed VindLU, for effective VidL pretraining, which achieves comparable or better than state-of-the-art results on several VidL tasks without relying on external CLIP pretraining.
- When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data98
This paper provides a formal framework for characterizing approaches to learning from explanation data, and proposes a synthetic task for studying how models learn from explanationData, and gives graphical models for the available modeling approaches.
- I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling96
The DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues are introduced and it is shown that the best contradiction detection model correlates well with human judgments and is used in both automatically evaluating and improving the consistency of state-of-the-art generative chatbots.
- Unified Coarse-to-Fine Alignment for Video-Text Retrieval95
This work proposes a Unified Coarse-to-fine Alignment model, dubbed UCoFiA, which outperforms previous state-of-the-art CLIP-based methods on multiple video-text retrieval benchmarks, and applies the Sinkhorn-Knopp algorithm to normalize the similarities of each level before summing them, alleviating over- and under-representation issues at different levels.
- Expressing Visual Relationships via Language93
This work introduces a new language-guided image editing dataset that contains a large number of real image pairs with corresponding editing instructions and proposes a new relational speaker model based on an encoder-decoder architecture with static relational attention and sequential multi-head attention and extended with dynamic relational attention.
- Adversarial Over-Sensitivity and Over-Stability Strategies for Dialogue Models92
Two categories of model-agnostic adversarial strategies are presented that reveal the weaknesses of several generative, task-oriented dialogue models: Should-Not-Change strategies that evaluate over-sensitivity to small and semantics-preserving edits, as well as Should- change strategies that test if a model is over-stable against subtle yet semantics-changing modifications.
- What is More Likely to Happen Next? Video-and-Language Future Event Prediction89
This work collects a new dataset, named Video-and-Language Event Prediction (VLEP), with 28,726 future event prediction examples (along with their rationales) from 10,234 diverse TV Show and YouTube Lifestyle Vlog video clips, and presents a strong baseline incorporating information from video, dialogue, and commonsense knowledge.
- The Power of Negative Thinking: Exploiting Label Disagreement in the Min-cut Classification Framework89
This work reports on work in progress that examines several novel heuristics that allow for the addition of label-disagreement information in a way that improves classification accuracy while preserving the efficiency guarantees of the minimum-cut framework.
- Coherent Dialogue with Attention-Based Language Models88
This work model coherent conversation continuation via RNN-based dialogue models equipped with a dynamic attention mechanism that dynamically increases the scope of attention on the history as the conversation continues, and shows that a vanilla RNN with dynamic attention outperforms more complex memory models by allowing for flexible, long-distance memory.
- Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters85
This paper presents an efficient graph-enhanced approach to multi-document summarization (MDS) with an encoder-decoder Transformer model that leads to significant improvements on the Multi-News dataset, overall leading to an average 1.8 ROUGE score improvement over previous work.
- PRover: Proof Generation for Interpretable Reasoning over Rules84
This work proposes PROVER, an interpretable transformer-based model that jointly answers binary questions over rule-bases and generates the corresponding proofs, and learns to predict nodes and edges corresponding to proof graphs in an efficient constrained training paradigm.
- Analyzing Compositionality-Sensitivity of NLI Models83
This work proposes a compositionality-sensitivity testing setup that analyzes models on natural examples from existing datasets that cannot be solved via lexical features alone, hence revealing the models' actual compositionality awareness.
- PaperRobot: Incremental Draft Generation of Scientific Ideas80
A PaperRobot who performs as an automatic research assistant by conducting deep understanding of a large collection of human-written papers in a target domain and constructing comprehensive background knowledge graphs is presented.
- Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization77
This paper first explores the use of constituency parse trees using a Transformer-based recurrent architecture for encoding structured input, and shows that off-the-shelf dense-captioning models trained on Visual Genome can improve the spatial structure of images from a different target domain without needing fine-tuning.
- InfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection77
This work contributes a novel benchmark for fake news detection at the knowledge element level, as well as a solution for this task which incorporates cross-media consistency checking to detect the fine-grained knowledge elements making news articles misinformative.
- 76
- ManyModalQA: Modality Disambiguation and QA over Diverse Inputs75
A new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities: text, images, and tables, is presented, with the expectation that existing datasets and approaches will be transferred for most of the training.
- Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning74
This work presents an interpretable, controller-based Self-Assembling Neural Modular Network for multi-hop reasoning, where four novel modules (Find, Relocate, Compare, NoOp) are designed to perform unique types of language reasoning.
- 73
- Structured Learning for Taxonomy Induction with Belief Propagation72
This model incorporates heterogeneous relational evidence about both hypernymy and siblinghood, captured by semantic features based on patterns and statistics from Web n-grams and Wikipedia abstracts, and uses loopy belief propagation along with a directed spanning tree algorithm for the core hypernym factor.
- The Amazon Nova Family of Models: Technical Report and Model Card71
A new generation of state-of-the-art foundation models that deliver frontier intelligence and industry-leading price performance are presented, and benchmarking results for core capabilities, agentic performance, long context, functional adaptation, runtime performance, and human evaluation are reported.
- Improving Generation and Evaluation of Visual Stories via Semantic Consistency70
A number of improvements to prior modeling approaches are presented, including the addition of a dual learning framework that utilizes video captioning to reinforce the semantic alignment between the story and generated images, a copy-transform mechanism for sequentially-consistent story visualization, and MART-based transformers to model complex interactions between frames.
- Crowdsourcing Lightweight Pyramids for Manual Summary Evaluation70
This work revisits the Pyramid approach, proposing a lightweight sampling-based version that is crowdsourcable and analyzes the performance of the method in comparison to original expert-based Pyramid evaluations, showing higher correlation relative to the common Responsiveness method.
- Diagnosing the Environment Bias in Vision-and-Language Navigation69
It is observed that neither the language nor the underlying navigational graph, but the low-level visual appearance conveyed by ResNet features directly affects the agent model and contributes to this environment bias in results, so several kinds of semantic representations could be better generalized to unseen testing environments.
- Hierarchically-Attentive RNN for Album Summarization and Storytelling69
This work addresses the problem of end-to-end visual storytelling by using the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets to encode the album photos, select representative photos, and compose the story.
- We are Humor Beings: Understanding and Predicting Visual Humor67
This work analyzes the humor manifested in abstract scenes and design computational models for them, and model two tasks that it is believed demonstrate an understanding of some aspects of visual humor.
- 66
- 64
- 64
- Question Relevance in VQA: Identifying Non-Visual And False-Premise Questions62
These approaches, based on LSTM-RNNs, VQA model uncertainty, and caption-question similarity, are able to outperform strong baselines on both relevance tasks and are shown to be more intelligent, reasonable, and human-like than previous approaches.
- Sort Story: Sorting Jumbled Images and Captions into Stories62
This work proposes the task of sequencing -- given a jumbled set of aligned image-caption pairs that belong to a story, the task is to sort them such that the output sequence forms a coherent story, and presents multiple approaches, via unary and pairwise predictions, and their ensemble-based combinations, achieving strong results.
- On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and Perspective60
A holistic framework for advancing trustworthiness in GenAI is established, paving the way for safer and more responsible integration of GenFMs into critical applications and releasing the toolkit for dynamic evaluation.
- 60
- Interpreting Neural Networks to Improve Politeness Comprehension60
This analysis reveals multiple novel, high-scoring politeness strategies which, when added back as new features, reduce the accuracy gap between the original featurized system and the neural model, thus providing a clear quantitative interpretation of the success of these neural networks.
- 58
- 56
- Continual and Multi-Task Architecture Search55
This work introduces a novel continual architecture search (CAS) approach, so as to continually evolve the model parameters during the sequential training of several tasks, without losing performance on previously learned tasks, thus enabling life-long learning.
- Text processing for text-to-speech systems in Indian languages.55
The efforts in addressing the issues of Font-to-Akshara mapping, pronunciation rules for Aksharas, text normalization in the context of building text- to-speech systems in Indian languages are discussed.
- 54
- DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization54
An improved video-and-language pre-training method that first adds automatically-extracted dense region captions from the video frames as auxiliary text input to alleviate the temporal misalignment issue and incorporates an entropy minimization-based constrained attention loss.
- Parsing Speech: a Neural Approach to Integrating Lexical and Acoustic-Prosodic Information54
A model that integrates transcribed text and acoustic-prosodic features using a convolutional neural network over energy and pitch trajectories coupled with an attention-based recurrent neural network that accepts text and prosodic features is introduced.
- Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning53
Language-Model-based Commonsense Reasoning (LMCR), a new method which enables a robot to listen to a natural language instruction from a human, observe the environment around it, and automatically fill in information missing from the instruction using environmental context and a new commonsense reasoning approach.
- EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents52
It is found that a small RL agent trained with EnvGen can outperform SOTA methods, including a GPT-4 agent, and learns long-horizon tasks significantly faster and how the environments are adapted to help improve RL agents' weaker skills over time is shown.
- EmailSum: Abstractive Email Thread Summarization51
The results reveal the key challenges of current abstractive summarization models in this task, such as understanding the sender’s intent and identifying the roles of sender and receiver and emphasize the importance of human evaluation and the development of better metrics by the community.
- A Sense-Topic Model for Word Sense Induction with Unsupervised Data Enrichment51
A sense-topic model is proposed for WSI, which treats sense and topic as two separate latent variables to be inferred jointly, and achieves significant improvements over the previous state-of-the-art, achieving the best results reported to date on the SemEval-2013 WSI task.
- Coreference Semantics from Web Features49
To address semantic ambiguities in coreference resolution, Web n-gram features that capture a range of world knowledge in a diffuse but robust way are used, exploiting short-distance cues to hypernymy, semantic compatibility, and semantic context, as well as general lexical co-occurrence.
- Good, Great, Excellent: Global Inference of Semantic Intensities48
A primarily unsupervised approach that uses semantics from Web-scale data to rank words by assigning them positions on a continuous scale, which achieves substantial improvements over previous work on both pairwise and rank correlation metrics.
- Explore, Propose, and Assemble: An Interpretable Model for Multi-Hop Reading Comprehension47
This work proposes an interpretable 3-module system called EPAr that approximates the coarse-to-fine-grained comprehension behavior of human readers when facing multiple long documents and achieves significant improvements over the baseline and competitive results compared to the state-of-the-art model.
- SafeCity: Understanding Diverse Forms of Sexual Harassment Personal Stories47
The task of automatically categorizing and analyzing various forms of sexual harassment, based on stories shared on the online forum SafeCity, is presented and analysis using LIME, first-derivative saliency heatmaps, activation clustering, and embedding visualization is presented.
- Data Augmentation for Abstractive Query-Focused Multi-Document Summarization46
This work builds abstractive end-to-end neural network models on the combined datasets that yield new state-of-the-art transfer results on DUC datasets, and introduces new hierarchical encoders that enable a more efficient encoding of the query together with multiple documents.
- Web-Scale Features for Full-Scale Parsing45
This work first presents a method for generating web count features that address the full range of syntactic attachments, and integrates these features into full-scale dependency and constituent parsers.
- Navigational Instruction Generation as Inverse Reinforcement Learning with Neural Machine Translation44
This work proposes a model that enables robots to generate natural language instructions that allow humans to navigate a priori unknown environments using a neural sequence-to-sequence model that learns to generate free-form instructions from natural language corpora.
- The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions42
It is found that the performance of state-of-the-art models on Natural Language Inference and Reading Comprehension analysis/stress sets can be highly unstable, with both theoretical explanations and empirical evidence regarding the source of the instability.
- Automatically Learning Data Augmentation Policies for Dialogue Tasks40
This work adapts AutoAugment to automatically discover effective perturbation policies for natural language processing (NLP) tasks such as dialogue generation, and achieves significant improvements over the previous state-of-the-art, including trained on manually-designed policies.
- 40
- Video Highlight Prediction Using Audience Chat Reactions40
A novel dataset based on League of Legends championships recorded from North American and Taiwanese Twitch.tv channels is presented, and strong results on these using multimodal, character-level CNN-RNN model architectures are demonstrated.
- RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios39
This work introduces RESIN-11, a new schema-guided event extraction&prediction framework that can be applied to a large variety of newsworthy scenarios and builds a demo website based on the dockerized system and schema library.
- 39
- 38
- MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding37
A new QA evaluation benchmark with 1,384 questions over news articles that require cross- media grounding of objects in images onto text, and introduces a novel multimedia data augmentation framework, based on cross-media knowledge extraction and synthetic question-answer generation, to automatically augment data that can provide weak supervision for this task.
- 36
- Towards Improving Abstractive Summarization via Entailment Generation35
The domain mismatch between the entailment (captions) and summarization (news) datasets suggests that the model is learning some domain-agnostic inference skills.
- Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA34
This paper proposes a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions, and evaluates the model on the challenging TVQA dataset, where each of the model components provides significant gains.
- Closed-Book Training to Improve Summarization Encoder Memory34
This paper presents a memory ability test, two saliency metrics, as well as several sanity-check ablations to prove that the encoder of the 2-decoder model does in fact learn stronger memory representations than the baseline encoder.
- 33
- 33
- 31
- 30
- 30
- How can NLP Help Revitalize Endangered Languages? A Case Study and Roadmap for the Cherokee Language29
- 29
- ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models27
Efficient Coarse-to-Fine LayerWise LayerWise Pruning (ECoFLaP), a two-stage coarse-to-fine weight pruning approach for LVLMs that validate the proposed method across various multimodal and unimodal models and datasets, demonstrating significant performance improvements over prevalent pruning techniques in the high-sparsity regime.
- 27
- 26
- 26
- 25
- 25
- 25
- 25
- SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts24
This paper investigates the core difficulties of sharing general knowledge and exploiting task-specific capabilities in learning navigation and proposes a novel State-Adaptive Mixture of Experts (SAME) model that effectively enables an agent to infer decisions based on different-granularity language and dynamic observations.
- 24
- Enhancing Multi-Document Summarization with Cross-Document Graph-based Information Extraction24
This work focuses on improving Multi-Document Summarization (MDS) performance by using cross-document IE output, incorporating two novel components: the use of auxiliary entity and event recognition systems to focus the summary generation model and incorporating an alignment loss between IE nodes and their text spans to reduce inconsistencies between the IE graphs and text representations.
- 24
- 24
- EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidance23
EPiC is introduced, an efficient and precise camera control learning framework that constructs well-aligned training anchor videos without the need for camera pose or point cloud estimation, and generalizes robustly to anchor videos made with point clouds at test time, enabling precise 3D-informed camera control.
- 23
- 23
- 23
- 22
- Enhanced Knowledge Selection for Grounded Dialogues via Document Semantic Graphs22
This work proposes to automatically convert the background knowledge documents into document semantic graphs and then perform knowledge selection over such graphs to preserve sentence-level information through the use of sentence nodes and provide concept connections between sentences.
- 22
- 22
- 20
- 19
- 18
- 18
- 17
- 17
- 17
- 17
- 16
- 16
- 16
- 15
- 15
- 14
- 14
- Predictors of mortality in periprosthetic fractures of the hip: Results from the national PPF study.14
- 14
- 14
- 14
- 13
- 13
- 13
- 13
- 13
- 13
- 13
- 13
- 13
- 12
- 12
- 12
- 12
- 12
- 12
- 12
- 11
- 11
- 11
- Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization10
Video-MSG, a training-free Guidance method for T2V generation based on Multimodal planning and Structured noise initialization, demonstrates its effectiveness in enhancing text alignment with multiple T2V backbones on popular T2V generation benchmarks (T2VCompBench and VBench).
- 10
- 10
- 10
- 10
- 10
- 10
- 10
- 9
- 9
- 9
- 9
- 9
- 9
- 8
- 8
- 8
- 8
- 8
- 8
- 8
- 8
- 8
- 7
- 7
- 7
- 7
- 7
- 7
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- Planning with Sketch-Guided Verification for Physics-Aware Video Generation5
SketchVerify is proposed, a training-free, sketch-verification-based planning framework that improves motion planning quality with more dynamically coherent trajectories prior to full video generation by introducing a test-time sampling and verification loop.
- 5
- 5
- 5
- 5
- 5
- 5
- 5
- 5
- 5
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations4
- 4
- 4
- 4
- 4
- 3
- RSQ: Learning from Important Tokens Leads to Better Quantized LLMs3
RSQ (Rotate, Scale, then Quantize), which applies rotations (orthogonal transformation) to the model to mitigate outliers (those with exceptionally large magnitude), and scales the token feature based on its importance, and quantizes the model using the GPTQ framework with the second-order statistics computed by scaled tokens.
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection3
The effectiveness and efficiency of Adapt-∞ are validated over a sequence of various multimodal instruction tuning datasets with various tasks, including (Knowledge) VQA, multilingual, grounding, reasoning, language-only, and multi-image comprehension tasks.
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 3
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 1
- 1
- 1
- 1
- 1
- 1
- Hierarchy-Aware Multimodal Unlearning for Medical AI1
This work introduces MedForget, a Hierarchy-Aware Multi-modal Unlearning Testbed with explicit retain and forget splits and evaluation sets containing rephrased variants, and introduces a reconstruction attack that progressively adds hierarchical level context to prompts.
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-13. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.