Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Jason M. EisnerSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works431 from public data
- Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks3,263
It is consistently found that multi-phase adaptive pretraining offers large gains in task performance, and it is shown that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable.
- From Tweets to Polls: Linking Text Sentiment to Public Opinion Time Series2,158
This work connects measures of public opinion measured from polls with sentiment measured from text, and finds that temporal smoothing is a critically important issue to support a suc- cessful model.
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models1,890
It is found that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts, and empirically assess several controllable generation methods find that while data- or compute-intensive methods are more effective at steering away from toxicity than simpler solutions, no current method is failsafe against neural toxic degeneration.
- 1,563
- Annotation Artifacts in Natural Language Inference Data1,383
It is shown that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI, and that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes.
- Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments1,122
A tagset is developed, data is annotated, features are developed, and results nearing 90% accuracy are reported on the problem of part-of-speech tagging for English data from the popular micro-blogging service Twitter.
- The Risk of Racial Bias in Hate Speech Detection1,082
This work proposes *dialect* and *race priming* as ways to reduce the racial bias in annotation, showing that when annotators are made explicitly aware of an AAE tweet’s dialect they are significantly less likely to label the tweet as offensive.
- Is Attention Interpretable?895
While attention noisily predicts input components’ overall importance to a model, it is by no means a fail-safe indicator, and there are many ways in which this does not hold, where gradient-based rankings of attention weights better predict their effects than their magnitudes.
- 890
- Linguistic Knowledge and Transferability of Contextual Representations850
It is found that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge.
- Transition-Based Dependency Parsing with Stack Long Short-Term Memory819
This work was sponsored in part by the U. S. Army Research Laboratory and the NSF CAREER grant IIS-1054319 and the European Commission.
- 800
- 767
- Knowledge Enhanced Contextual Word Representations748
After integrating WordNet and a subset of Wikipedia into BERT, the knowledge enhanced BERT (KnowBert) demonstrates improved perplexity, ability to recall facts as measured in a probing task and downstream performance on relationship extraction, entity typing, and word sense disambiguation.
- 740
- 721
- 678
- The Web as a Parallel Corpus677
The use of supervised learning based on structural features of documents to improve classification performance, a new content-based measure of translational equivalence, and adaptation of the system to take advantage of the Internet Archive for mining parallel text from the Web on a large scale are presented.
- Social Bias Frames: Reasoning about Social and Power Implications of Language672
It is found that while state-of-the-art neural models are effective at high-level categorization of whether a given statement projects unwanted social bias, they are not effective at spelling out more detailed explanations in terms of Social Bias Frames.
- 662
- Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics648
The results indicate that a shift in focus from quantity to quality of data could lead to robust models and improved out-of-distribution generalization, and a model-based tool to characterize and diagnose datasets.
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers643
Qasper is presented, a dataset of 5049 questions over 1585 Natural Language Processing papers that is designed to facilitate document-grounded, information-seeking QA, and finds that existing models that do well on other QA tasks do not perform well on answering these questions.
- All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text583
The role untrained human evaluations play in NLG evaluation is examined and three approaches for quickly training evaluators to better identify GPT3-authored text are explored and it is found that while evaluation accuracy improved up to 55%, it did not significantly improve across the three domains.
- Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research577
To facilitate scientific research on language model pretraining, Dolma is curate and released, a three-trillion-token English corpus built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials.
- DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts576
This work highlights the promise of tuning small LMs on text with (un)desirable attributes for efficient decoding-time steering and applies DExperts to language detoxification and sentiment-controlled generation, where it outperform existing controllable generation methods on both automatic and human evaluations.
- 556
- Evaluating Models’ Local Decision Boundaries via Contrast Sets532
A more rigorous annotation paradigm for NLP that helps to close systematic gaps in the test data, and recommends that the dataset authors manually perturb the test instances in small but meaningful ways that (typically) change the gold label, creating contrast sets.
- TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering523
The introduction of TIFA (Text-to-Image Faithfulness evaluation with question Answering), an automatic evaluation metric that measures the faithfulness of a generated image to its text input via visual question answering (VQA), and a comprehensive evaluation of existing text-to-image models using TIFA v1.0.
- RewardBench: Evaluating Reward Models for Language Modeling509
The RewardBench dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries and presents many findings on propensity for refusals, reasoning limitations, and instruction following shortcomings of various reward models towards a better understanding of the RLHF process.
- To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks496
The empirical results across diverse NLP tasks with two state-of-the-art models show that the relative performance of fine-tuning vs. feature extraction depends on the similarity of the pretraining and target tasks.
- How Language Model Hallucinations Can Snowball489
This work hypothesizes that in some cases, when justifying previously generated hallucinations, LMs output false claims that they can separately recognize as incorrect, which leads to hallucination snowballing: an LM over-commits to early mistakes, leading to more mistakes that it otherwise would not make.
- Evaluating Gender Bias in Machine Translation484
An automatic gender bias evaluation method for eight target languages with grammatical gender, based on morphological analysis is devised, which shows that four popular industrial MT systems and two recent state-of-the-art academic MT models are significantly prone to gender-biased translation errors for all tested target languages.
- What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA475
A probabilistic quasi-synchronous grammar, inspired by one proposed for machine translation, and parameterized by mixtures of a robust nonlexical syntax/alignment model with a(n optional) lexical-semantics-driven log-linear model is proposed.
- Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection443
This work disentangle what is annotated as toxic by considering posts with three characteristics: anti-Black language, African American English dialect, and vulgarity, and shows strong associations between annotator identity and beliefs and their ratings of toxicity.
- Better Hypothesis Testing for Statistical Machine Translation: Controlling for Optimizer Instability443
- Contrastive estimation382
A novel approach, contrastive estimation, is described, which outperforms EM, is more robust to degradations of the dictionary, and can largely recover by modeling additional features.
- UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models380
The UnifiedSKG framework is proposed, which unifies 21 SKG tasks into a text-to-text format, aiming to promote systematic SKG research, instead of being exclusive to a single task, domain, or dataset.
- 366
- OLMoE: Open Mixture-of-Experts Language Models352
This work introduces OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE), and presents various experiments on MoE training, analyzes routing in the model showing high specialization, and open-source all aspects of the work: model weights, training data, code, and logs.
- 352
- Measuring the Carbon Intensity of AI in Cloud Instances344
This paper provides a framework for measuring software carbon intensity, and proposes to measure operational carbon emissions by using location-based and time-specific marginal emissions data per energy unit, and provides recommendations for how machine learning practitioners can useSoftware carbon intensity information to reduce environmental impact.
- Selective Annotation Makes Language Models Better Few-Shot Learners343
This work forms an annotation-efficient, two-step framework: selective annotation that chooses a pool of examples to annotate from unlabeled data in advance, followed by prompt retrieval that retrieves task examples from the annotated pool at test time, and proposes an unsupervised, graph-based selective annotation method, voke-k, to select diverse, representative examples to annotation.
- Show Your Work: Improved Reporting of Experimental Results343
It is demonstrated that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best, and a novel technique is presented: expected validation performance of the best-found model as a function of computation budget.
- Predicting risk from financial reports with regression343
This work applies well-known regression techniques to a large corpus of freely available financial reports, constructing regression models of volatility for the period following a report, rivaling past volatility in predicting the target variable.
- A Discriminative Graph-Based Parser for the Abstract Meaning Representation337
The first approach to parse sentences into meaning representation, a semantic formalism for which a grow- ing set of annotated examples is available, is introduced, providing a strong baseline for improvement.
- Frame-Semantic Parsing336
A two-stage statistical model that takes lexical targets in their sentential contexts and predicts frame-semantic structures and results in qualitatively better structures than naïve local predictors, which outperforms the prior state of the art by significant margins.
- Creative Writing with a Machine in the Loop323
Novel natural language models and design choices are suggested that may better support creative writing, as machine suggestions do not necessarily lead to better written artifacts.
- Toward Abstractive Summarization Using Semantic Representations319
This work focuses on the graph-tograph transformation that reduces the source semantic graph into a summary graph, making use of an existing AMR parser and assuming the eventual availability of an AMR-totext generator.
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models307
This work proposes MUSE, a comprehensive machine unlearning evaluation benchmark that enumerates six diverse desirable properties for unlearned models and demonstrates that most algorithms can prevent verbatim memorization and knowledge memorization to varying degrees, but only one algorithm does not lead to severe privacy leakage.
- Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs301
Extensions to a continuousstate dependency parsing method that makes it applicable to morphologically rich languages replace lookup-based word representations with representations constructed from the orthographic representations of the words, also using LSTMs.
- 2 OLMo 2 Furious286
This work describes the modified model architecture and training recipe, focusing on techniques for achieving better training stability and improved per-token efficiency, and introduces a new, specialized data mix called Dolmino Mix 1124, which significantly improves model capabilities across many downstream task benchmarks when introduced via late-stage curriculum training.
- 275
- WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation274
This work introduces a novel approach for dataset creation based on worker and AI collaboration, which brings together the generative strength of language models and the evaluation strength of humans, and demonstrates the promise of leveraging natural language generation techniques and re-imagining the role of humans in the dataset creation process.
- Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2271
The evaluation from multiple perspectives shows that the T\"ULU 2 suite achieves state-of-the-art performance among open models and matches or exceeds the performance of GPT-3.5-turbo-0301 on several benchmarks.
- 263
- The Multilingual Amazon Reviews Corpus249
The use of mean absolute error (MAE) instead of classification accuracy for this task, since MAE accounts for the ordinal nature of the ratings, is proposed.
- Many Languages, One Parser236
This work trains one multilingual model for dependency parsing and uses it to parse sentences in several languages, enabling the parser not only to parse effectively in multiple languages, but also to generalize across languages based on linguistic universals and typological similarities.
- 234
- A Bayesian Mixed Effects Model of Literary Character234
A model that employs multiple effects to account for the influence of extra-linguistic information (such as author) is introduced and it is found that this method leads to improved agreement with the preregistered judgments of a literary scholar, complementing the results of alternative models.
- Paraphrase identification as probabilistic quasi-synchronous recognition231
A generative model is employed that generates a paraphrase of a given sentence, and probabilistic inference is used to reason about whether two sentences share the paraphrase relationship.
- Measuring Association Between Labels and Free-Text Rationales230
It is demonstrated that *pipelines*, models for faithful rationalization on information-extraction style tasks, do not work as well on “reasoning” tasks requiring free-text rationales, and state-of-the-art T5-based joint models exhibit desirable properties for explaining commonsense question-answering and natural language inference.
- Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models225
BTM improves in- and out-of-domain perplexities as compared to GPT-style Transformer LMs, and gains grow with the number of domains, suggesting more aggressive parallelism could be used to efficiently train larger models in future work.
- A Dependency Parser for Tweets221
A new dependency parser for English tweets, TWEEBOPARSER, which builds on several contributions: new syntactic annotations for a corpus of tweets, with conventions informed by the domain; adaptations to a statistical parsing algorithm; and a new approach to exploiting out-of-domain Penn Treebank data.
- Automatic factual question generation from text220
This research supports the idea that natural language processing can help teachers efficiently create instructional content by automating the creation of specific type of assessment item and provides solutions to some of the major challenges in question generation.
- NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics213
NeuroLogic A*esque is proposed, a decoding algorithm that incorporates heuristic estimates of future cost that develops lookahead heuristics that are efficient for large-scale language models, making this method a drop-in replacement for common techniques such as beam search and top-k sampling.
- Event2Mind: Commonsense Inference on Events, Intents, and Reactions212
It is demonstrated how commonsense inference on people’s intents and reactions can help unveil the implicit gender inequality prevalent in modern movie scripts.
- Sparse Overcomplete Word Vector Representations212
This work proposes methods that transform word vectors into sparse (and optionally binary) vectors, which are more similar to the interpretable features typically used in NLP, though they are discovered automatically from raw corpora.
- 207
- The Right Tool for the Job: Matching Model and Instance Complexities206
This work proposes a modification to contextual representation fine-tuning which allows for an early (and fast) “exit” from neural network calculations for simple instances, and late (and accurate) exit for hard instances during inference.
- Concise integer linear programming formulations for dependency parsing199
This formulation of the problem of non-projective dependency parsing as a polynomial-sized integer linear program is able to handle non-local output features in an efficient manner and is compatible with prior knowledge encoded as hard constraints, and can also learn soft constraints from data.
- Neural Cross-Lingual Named Entity Recognition with Minimal Resources197
This work proposes a method that finds translations based on bilingual word embeddings and uses self-attention to improve robustness to word order differences, and demonstrates that these methods achieve state-of-the-art or competitive NER performance on commonly tested languages under a cross-lingual setting, with much lower resource requirements than past approaches.
- 191
- Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning191
This work presents a new crowdsourced dataset containing more than 24K span-selection questions that require resolving coreference among entities in over 4.7K English paragraphs from Wikipedia, and shows that state-of-the-art reading comprehension models perform significantly worse than humans on this benchmark.
- DEMix Layers: Disentangling Domains for Modular Language Modeling188
A new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text, and shows it is possible to add experts to adapt to new domains without forgetting older ones, and remove experts to restrict access to unwanted domains.
- Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts184
This work introduces methods based on sentence mover’s similarity, and finds that sentence-based metrics correlate with human judgments significantly better than ROUGE, both on machine-generated summaries and human-authored essays.
- 179
- Question Generation via Overgenerating Transformations and Ranking179
This framework for question generation composes general-purpose rules to transform declarative sentences into questions, is modular in that existing NLP tools can be leveraged, and includes a statistical component for scoring questions based on features of the input, output, and transformations performed.
- Saturated Transformers are Constant-Depth Threshold Circuits178
Saturated transformers transcend the known limitations of hard-attention transformers, and it is proved saturated transformers with floating-point values can be simulated by constant-depth threshold circuits, giving the class TC0 as an upper bound on the formal languages they recognize.
- Computational Approaches to Morphology and Syntax174
This chapter discusses machine learning of Morphology, which automates the very labor-intensive and therefore time-heavy and expensive process of formalizing Morphological Operations.
- 170
- Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text163
This work proposes a new framework called Scarecrow for scrutinizing machine text via crowd annotation, and quantifies measurable gaps between human authored text and generations from models of several sizes, including fourteen configurations of GPT-3.
- Challenges in Automated Debiasing for Toxic Language Detection163
The findings show that debiasing a model trained on biased toxic language data is not as effective as simply relabeling the data to remove existing biases, and proposes an automatic, dialect-aware data correction method, as a proof-of-concept.
- What's In My Big Data?154
This work proposes What's In My Big Data?
- Deep Multitask Learning for Semantic Dependency Parsing153
A deep neural architecture that parses sentences into three semantic dependency graph formalisms by using efficient, nearly arc-factored inference and a bidirectional-LSTM composed with a multi-layer perceptron is presented.
- What Do Recurrent Neural Network Grammars Learn About Syntax?152
By training grammars without nonterminal labels, it is found that phrasal representations depend minimally on nonterminals, providing support for the endocentricity hypothesis.
- PromptCap: Prompt-Guided Task-Aware Image Captioning150
PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs, achieves state-of-the-art accuracy on knowledge-based VQA tasks and generalizes well to unseen domains.
- 150
- A Neural Framework for Generalized Topic Models.135
A general neural framework is proposed, based on topic models, to enable flexible incorporation of metadata and allow for rapid exploration of alternative models, and achieves strong performance, with a manageable tradeoff between perplexity, coherence, and sparsity.
- Robust Navigation with Language Pretraining and Stochastic Sampling133
This paper adapts large-scale pretrained language models to learn text representations that generalize better to previously unseen instructions and proposes a stochastic sampling scheme to reduce the considerable gap between the expert actions in training and sampled actions in test.
- 131
- Thinking Like a Skeptic: Defeasible Inference in Natural Language131
From Defeasible NLI, both a classification and generation task for defeasible inference are developed, and it is demonstrated that the generation task is much more challenging.
- Shared logistic normal distributions for soft parameter tying in unsupervised grammar induction130
A family of priors over probabilistic grammar weights is presented, called the shared logistic normal distribution, enabling factored covariance between the probabilities of different derivation events in the probabilism grammar, providing a new way to encode prior knowledge about an unknown grammar.
- Time Waits for No One! Analysis and Challenges of Temporal Misalignment127
It is found that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period, which motivates continued research to improve temporal robustness of NLP models.
- Competency Problems: On Finding and Removing Artifacts in Language Data127
This work argues that for complex language understanding tasks, all simple feature correlations are spurious, and formalizes this notion into a class of problems which are called competency problems, and gives a simple statistical test for dataset artifacts that is used to show more subtle biases.
- Narrative framing of consumer sentiment in online restaurant reviews125
It is demonstrated that portraying the self, whether as well–educated, as a victim, or even as addicted to chocolate, is a key function of reviews and suggests the important role of online reviews in exploring social psychological variables.
- The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task121
It is shown how variants of the same writing task can lead to measurable differences in writing style, and a simple linear classifier informed by stylistic features is able to successfully distinguish among the three cases.
- Variational Pretraining for Semi-supervised Text Classification120
VAMPIRE is introduced, a lightweight pretraining framework for effective text classification when data and computing resources are limited and it is found that fine-tuning to in-domain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision.
- Predicting response to political blog posts with topic models120
The models jointly describe the generation of the primary documents as well as the authorship and the contents of the blog community's verbal reactions to each post (comments) and evaluate their model on a novel comment prediction task.
- Neural Discourse Structure for Text Categorization118
This work uses a recursive neural network and a newly proposed attention mechanism to compute a representation of the text that focuses on salient content, from the perspective of both RST and the task.
- Summarization with a joint model for sentence extraction and compression118
This work proposes a one-step approach for document summarization that jointly performs sentence extraction and compression by solving an integer linear program.
- 115
- 115
- Crowdsourcing Annotations for Websites' Privacy Policies115
The results suggest that, if carefully deployed, crowdsourcing can indeed result in the generation of non-trivial annotations and can also help identify elements of ambiguity in policies.
- Nonextensive Information Theoretic Kernels on Measures115
A new family of nonextensive mutual information kernels, which allow weights to be assigned to their arguments, and which includes the Boolean, JS, and linear kernels as particular cases, are defined that generalize the p-spectrum kernel.
- Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets114
This work introduces inoculation by fine-tuning, a new analysis method for studying challenge datasets by exposing models to a small amount of data from the challenge dataset (a metaphorical pathogen) and assessing how well they can adapt.
- 111
- Discriminative Lexical Semantic Segmentation with Gaps: Running the MWE Gamut111
A novel representation, evaluation measure, and supervised models are presented for the task of identifying the multiword expressions (MWEs) in a sentence, resulting in a lexical semantic segmentation, enabling efficient sequence tagging algorithms for feature-rich discriminative models.
- 110
- Probing Across Time: What Does RoBERTa Know and When?110
It is believed that probing-across-time analyses can help researchers understand the complex, intermingled learning that these models undergo and guide us toward more efficient approaches that accomplish necessary learning faster.
- Generation from Abstract Meaning Representation using Tree Transducers110
This paper addresses generating English from the Ab-stract Meaning Representation, consisting of re-entrant graphs whose nodes are concepts and edges are relations, and consists of generating an appropriate spanning tree for the AMR and applying tree-to-string transducers to generate English.
- Translating into Morphologically Rich Languages with Synthetic Phrases110
The approach relies on morphological analysis of the target language, but it is shown that an unsupervised Bayesian model of morphology can successfully be used in place of a supervised analyzer.
- Finetuning Pretrained Transformers into RNNs108
This work proposes a swap-then-finetune procedure, which in an off-the-shelf pretrained transformer, replaces the softmax attention with its linear-complexity recurrent alternative and then finetune, and provides an improved tradeoff between efficiency and accuracy over the standard transformer and other recurrent variants.
- 104
- Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection104
It is argued that privileging any corpus as high quality entails a language ideology, and more care is needed to construct training corpora for language models, with better transparency and justification for the inclusion or exclusion of various texts.
- Syntactic Scaffolds for Semantic Structures104
This work introduces the syntactic scaffold, an approach to incorporating syntactic information into semantic tasks through a multitask objective, and improves over strong baselines on PropBank semantics, frame semantics, and coreference resolution.
- Shortformer: Better Language Modeling using Shorter Inputs103
This work identifies conditions where shorter inputs are not harmful, and achieves perplexity and efficiency improvements through two new methods that decrease input length, and shows how to improve the efficiency of recurrence methods in transformers.
- A Formal Hierarchy of RNN Architectures103
It is hypothesized that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy, and empirical results to support this conjecture are provided.
- Stacking dependency parsers103
Experiments on twelve languages show that stacking transition-based and graph-based parsers improves performance over existing state-of-the-art dependency parsers.
- Tuning Language Models by Proxy102
This work demonstrates the promise of using small tuned LMs to efficiently customize large, potentially proprietary LMs through decoding-time guidance, and introduces proxy-tuning, a lightweight decoding-time algorithm that operates on top of black-box LMs to achieve the same end as direct tuning.
- In-context Pretraining: Language Modeling Beyond Document Boundaries102
In-Context Pretraining offers a simple and scalable approach to significantly enhance LMs' performance, and is seen to see notable improvements in tasks that require more complex contextual reasoning, including in-context learning.
- Distributed Representations of Geographically Situated Language102
In a quantitative evaluation on the task of judging geographically informed semantic similarity between representations learned from 1.1 billion words of geo-located tweets, the joint model outperforms comparable independent models that learn meaning in isolation.
- 100
- Improving Transformer Models by Reordering their Sublayers99
This work proposes a new transformer pattern that adheres to this property, the sandwich transformer, and shows that it improves perplexity on multiple word-level and character-level language modeling benchmarks, at no cost in parameters, memory, or training time.
- 99
- Linguistic Structure Prediction98
This work considers statistical, computational approaches to modeling linguistic structure and approaches to decoding and supervised and unsupervised learning of models that predict discrete structures as outputs are the focus.
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore95
The results suggest that it is possible to build high quality language models while mitigating their legal risk, and which nonparametric approach works best, where the remaining errors lie, and how performance scales with datastore size.
- Comprehensive Annotation of Multiword Expressions in a Social Web Corpus95
This work advocates for a comprehensive annotation approach: proceeding sentence by sentence, the annotators manually group tokens into MWEs according to guidelines that cover a broad range of multiword phenomena.
- Parsing with soft and hard constraints on dependency length95
It is shown that a crude way to use dependency length as a parsing feature can substantially improve parsing speed and accuracy in English and Chinese, and similar improvements by imposing hard bounds on dependency length and modeling the resulting sequence of parse fragments.
- Dynamic Entity Representations in Neural Language Models94
A new type of language model is presented that can explicitly model entities, dynamically update their representations, and contextually generate their mentions and can model an arbitrary number of entities in context while generating each entity mention at an arbitrary length.
- 89
- Annealing structural bias in multilingual weighted grammar induction88
This work shows how a structural locality bias can improve the accuracy of state-of-the-art dependency grammar induction models trained by EM from unannotated examples, and annealing the free parameter that controls this bias achieves further improvements.
- A Corpus and Model Integrating Multiword Expressions and Supersenses87
The online reviews genre is investigated, adding semantic supersense annotations to a 55,000 word English corpus that was previously annotated for multiword expressions, and a sequence tagging model is presented that jointly infers lexical expressions and their supersenses.
- Bilingual parsing with factored estimation: Using English to parse Korean87
This work describes how simple, commonly understood statistical models, such as statistical dependency parsers, probabilistic context-free grammars, and word-to-word translation models can be effectively combined into a unified bilingual parser that jointly searches for the best English parse, Korean parse, andword alignment.
- Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?86
This work formalizes ways in which ungrounded language models appear to be fundamentally limited in their ability to “understand”, and suggests that assertions in code or language do not provide sufficient signal to fully emulate semantic representations.
- Evaluating NLP Models via Contrast Sets86
A new annotation paradigm for NLP is proposed that helps to close systematic gaps in the test data, and it is recommended that after a dataset is constructed, the dataset authors manually perturb the test instances in small but meaningful ways that change the gold label, creating contrast sets.
- Guiding unsupervised grammar induction using contrastive estimation85
It is shown that, using the same features, log-linear dependency grammar models trained using CE can drastically outperform EMtrained generative models on the task of matching human linguistic annotations (the MATCHLINGUIST task).
- Segmental Recurrent Neural Networks for End-to-End Speech Recognition84
Practical training and decoding issues as well as the method to speed up the training in the context of speech recognition are discussed, and the model is self-contained and can be trained end-to-end.
- 83
- Probabilistic models of nonprojective dependency trees83
The Matrix Tree Theorem is exploited to derive an algorithm that efficiently sums the scores of all nonprojective trees in a sentence, permitting the definition of a conditional log-linear model over trees.
- Novel estimation methods for unsupervised discovery of latent structure in natural language text82
The novel estimation methods presented are better suited to adaptation for real engineering tasks than the maximum likelihood baseline, and are shown to achieve significant improvements over maximum likelihood estimation and maximum a posteriori estimation, for a state-of-the-art probabilistic model used in dependency grammar induction.
- Parsing Tweets into Universal Dependencies77
It is shown that it is challenging to deliver consistent annotation due to ambiguity in understanding and explaining tweets and proposed a new method to distill an ensemble of 20 transition-based parsers into a single one that achieves an improvement of 2.2 in LAS over the un-ensembled baseline and outperforms parsers that are state-of-the-art on other treebanks in both accuracy and speed.
- Training with Exploration Improves a Greedy Stack LSTM Parser77
This work adapts the greedy Stack-LSTM dependency parser to support a training-with-exploration procedure using dynamic oracles instead of cross-entropy minimization, which improves parsing accuracies for both English and Chinese.
- Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser77
This work trains the distillation parser using a structured hinge loss objective with a novel cost that incorporates ensemble uncertainty estimates for each possible attachment, thereby avoiding the intractable cross-entropy computations required by applying standard distillation objectives to problems with structured outputs.
- Logistic Normal Priors for Unsupervised Probabilistic Grammar Induction77
A new Bayesian model for probabilistic grammars is explored, exploiting the logistic normal distribution as a prior over the grammar parameters, and it is shown that the model achieves superior results over previous models that use different priors.
- Analyzing Framing through the Casts of Characters in the News75
An unsupervised model for the discovery and clustering of latent “personas” (characterizations of entities) is presented and automated model selection is introduced as a fair and robust form of feature evaluation.
- Polyglot Contextual Representations Improve Crosslingual Transfer74
Rosita is introduced, a method to produce multilingual contextual word representations by training a single language model on text from multiple languages, which provides further evidence for the benefits of polyglot learning, in which representations are shared across multiple languages.
- 74
- Learning Joint Semantic Parsers from Disjoint Data72
A new approach to learning a semantic parser from multiple datasets, even when the target semantic formalisms are drastically different and the underlying corpora do not overlap is presented, by treating annotations for unobserved formalisms as latent structured variables.
- Frame-Semantic Role Labeling with Heterogeneous Annotations72
This work augments an existing model with features derived from Frame net and PropBank and with partially annotated exemplars from FrameNet with the aim of identifying and labeling the semantic arguments of a predicate that evokes a FrameNet frame.
- Transparent Human Evaluation for Image Captioning71
THumB, a rubric-based human evaluation protocol for image captioning models, is established and results reveal that CLIPScore, a recent metric that uses image features, better correlates with human judgments than conventional text-only metrics because it is more sensitive to recall.
- 71
- Rich source-side context for statistical machine translation71
This work explores the augmentation of statistical machine translation models with features of the context of each phrase to be translated, including the use of context in example-based machine translation and the incorporation of word sense disambiguation into a translation model.
- Scaling Expert Language Models with Unsupervised Domain Discovery69
This work introduces a simple but effective method to asynchronously train large, sparse language models on arbitrary text corpora, which clusters a corpus into sets of related documents, trains a separate expert language model on each cluster, and combines them in a sparse ensemble for inference.
- 68
- 68
- 68
- 66
- 66
- Context-based morphological disambiguation with random fields66
A novel source-channel model is applied to the problem of morphological disambiguation (segmentation into morphemes, lemmatization, and POS tagging) for concatenative, templatic, and inflectional languages.
- 64
- Explaining Relationships Between Scientific Documents63
This paper establishes a dataset of 622K examples from 154K documents, and pretrain a large language model to serve as the foundation for autoregressive approaches to the task of explaining relationships between two scientific documents using natural language text.
- Greedy, Joint Syntactic-Semantic Parsing with Stack LSTMs63
This work presents a transition-based parser that jointly produces syntactic and semantic dependencies and obtains the best published parsing performance among models that jointly learn syntax and semantics.
- Compiling Comp Ling: Weighted Dynamic Programming and the Dyna Language62
A declarative specification language, Dyna is proposed; general agenda-based algorithms for computing weights and gradients are given; and it is shown that a first implementation of a Dyna-to-C++ compiler produces code that is efficient enough for real NLP research, though still several times slower than hand-crafted code.
- Sounding Board: A User-Centric and Content-Driven Social Chatbot61
The system architecture consists of several components including spoken language processing, dialogue management, language generation, and content management, with emphasis on user-centric and content-driven design.
- Compiling Comp Ling: practical weighted dynamic programming and the Dyna language61
A declarative specification language, Dyna is proposed; general agenda-based algorithms for computing weights and gradients are given; and it is shown that a first implementation of a Dyna-to-C++ compiler produces code that is efficient enough for real NLP research, though still several times slower than hand-crafted code.
- 60
- Linguistic Structured Sparsity in Text Categorization59
Three linguistically motivated structured regularizers based on parse trees, topics, and hierarchical word clusters for text categorization are introduced, showing that these regularizers consistently improve classification accuracies compared to standard regularizers that penalize features in isolation.
- Preference grammars: softening syntactic constraints to improve statistical machine translation59
A novel probabilistic synchoronous context-free grammar formalism for statistical machine translation, in which syntactic nonterminal labels are represented as "soft" preferences rather than as "hard" matching constraints, which allows the importance of syntactic preferences to be learned alongside other features and for particular labeling procedures.
- Joint Morphological and Syntactic Disambiguation58
Several efficient, probabilisticallyinterpretable ways to apply joint inference to morphological and syntactic disambiguation using lattice parsing are described and shown to compare favorably to pipeline parsing methods across a variety of component models.
- Annealing techniques for unsupervised statistical language learning58
Deterministic annealing is described as an appealing alternative to the Expectation-Maximization algorithm, and a variant, skewed DA, is described, which can incorporate a good initializer when it is available, and show significant improvements over EM on a grammar induction task.
- 56
- Extractive Summarization by Maximizing Semantic Volume56
This work embeds each sentence in a semantic space and construct a summary by choosing a subset of sentences whose convex hull maximizes volume in that space, and provides a greedy algorithm based on the GramSchmidt process to efficiently perform volume maximization.
- 55
- CMU at SemEval-2016 Task 8: Graph-based AMR Parsing with Infinite Ramp Loss55
Improvements to the JAMR parser are presented, including improved concept coverage using external resources and features, an improved aligner, and a novel loss function for structured prediction called infinite ramp, which is a generalization of the structured SVM to problems with un-reachable training instances.
- Weighted and Probabilistic Context-Free Grammars Are Equally Expressive55
Any parsing or labeling accuracy improvement from conditional estimation of WCFGs or conditional random fields (CRFs) over joint estimation of PCFGs or hidden Markov models (HMMs) is due to the estimation procedure rather than the change in model class, becausePCFGs and HMMs are exactly as expressive as W CFGs and chain-structured CRFs, respectively.
- Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation52
The findings suggest that the latency disadvantage for autoregressive translation has been overestimated due to a suboptimal choice of layer allocation, and a new speed-quality baseline for future research toward fast, accurate translation is provided.
- 52
- Generating Scientific Definitions with Controllable Complexity51
A novel reranking approach is introduced and it is found in human evaluations that it offers superior fluency while also controlling complexity, compared to several controllable generation baselines.
- 51
- 50
- 50
- The Importance of Calibration for Estimating Proportions from Annotations48
This paper identifies and differentiate between two relevant data generating scenarios (intrinsic vs. extrinsic labels), introduces a simple but novel method which emphasizes the importance of calibration, and analyzes and experimentally validate the appropriateness of various methods for each of the two scenarios.
- Story Cloze Task: UW NLP System48
University of Washington NLP’s submission for the Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem 2017) shared task—the Story Cloze Task is described, which is a linear classifier with a variety of features, including both the scores of a neural language model and style features.
- Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand47
This work proposes a generalization of leaderboards, bidimensional leaderboards (Billboards), that simultaneously tracks progress in language generation models and metrics for their evaluation and demonstrates that a linear ensemble of a few diverse metrics sometimes substantially outperforms existing metrics in isolation.
- Rethinking the Evaluation of Harness Evolution for Agents46
The evaluation of automatic harness evolution procedures is revisited and the need for matched-budget baselines and held-out evaluation to distinguish genuine harness improvements from benchmark-specific search and overfitting is highlighted.
- 46
- 46
- Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent46
The tendency for transformer parameters to grow in magnitude during training is studied, and its implications for the emergent representations within self attention layers are studied, suggesting saturation is a new characterization of an inductive bias implicit in GD of particular interest for NLP.
- 46
- Polyglot Semantic Role Labeling46
Analysis of the polyglot models’ performance provides a new understanding of the similarities and differences between languages in the shared task, and results in improvement in parsing performance on several languages over a monolingual baseline.
- Analyzing Privacy Policies at Scale46
The results from these efforts show the effectiveness of using automated and semi-automated methods for extracting from privacy policies the data practice details that are salient to Internet users’ interests.
- 46
- Recollection versus Imagination: Exploring Human Memory and Cognition via Neural Language Models45
A measure of narrative flow is introduced and used to examine the narratives for imagined and recalled events and reveals the effect of narrativization of memories in stories (e.g., stories about frequently recalled memories flow more linearly; Bartlett, 1932).
- Hierarchical Character-Word Models for Language Identification45
This work introduces a hierarchical model that learns character and contextualized word-level representations for language identification that performs well against strong base- lines, and can also reveal code-switching.
- Computational Approaches to Morphology and Syntax Brian Roark and Richard Sproat (Oregon Health and Science University and University of Illinois at Urbana-Champaign) Oxford: Oxford University Press (Oxford surveys in syntax and morphology, edited by Robert D. Van Valin Jr, volume 4), 2007, xx+316 pp; hardbound, ISBN 978-0-19-927477-2, $110.00, £60.00; paperbound, ISBN 978-0-19-927478-9, $45.00, £24.9945
The polyamide molding resin produced by this process provides shaped articles having excellent mechanical properties as well as flame retardant properties without blooming.
- A Mixture of h - 1 Heads is Better than h Heads44
This work proposes the mixture of attentive experts model (MAE), a model trained using a block coordinate descent algorithm that alternates between updating the responsibilities of the experts and their parameters and learns to activate different heads on different inputs.
- Learning Topics and Positions from Debatepedia44
A generative model positing latent topics and cross-cutting positions that gives special treatment to person mentions and opinion words is introduced that’s usefulness in attaching opinionated documents to arguments and its consistency with human judgments about positions is evaluated.
- SEMAFOR 1.0: A Probabilistic Frame-Semantic Parser44
An implemented parser is described that transforms an English sentence into a frame-semantic representation and is demonstrated to significantly outperform previously published results and is released for public use.
- DataDecide: How to Predict Best Pretraining Data with Small Experiments43
This work conducts controlled pretraining experiments across 25 corpora with differing sources, deduplication, and filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random seeds and finds that the ranking of models at a single, small size is a strong baseline for predicting best models at the larger target scale (1B).
- What Can Natural Language Processing Do for Peer Review?43
The goal of this work is to provide a foundation for the future efforts in NLP for peer-reviewing assistance, and to set the agenda for research in machine-assisted scientific quality control in the age of AI, within the NLP community and beyond.
- Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank43
Using dependency parsing of four diverse low-resource language varieties as a case study, it is shown that these methods significantly improve performance over baselines, especially in the lowest-resource cases, and the importance of the relationship between such models’ pretraining data and target language varieties.
- Backpropagating through Structured Argmax using a SPIGOT43
It is shown that training with SPIGOT leads to a larger improvement on the downstream task than a modularly-trained pipeline, the straight-through estimator, and structured attention, reaching a new state of the art on semantic dependency parsing.
- World Vaping Day: Contextualizing Vaping Culture in Online Social Media Using a Mixed Methods Approach43
Research about electronic nicotine delivery systems, using Twitter data from “World Vaping Day,” utilized a convergent parallel mixed methods framework, integrating thematic prevalence estimates with phenomenological contextualization.
- ABC: Attention with Bounded-memory Control42
This work shows that disparate approaches can be subsumed into one abstraction, attention with bounded-memory control (ABC), and it outperforms previous efficient attention models; compared to the strong transformer baselines, it significantly improves the inference time and space efficiency with no or negligible accuracy loss.
- Deep Weighted Averaging Classifiers42
A simple way to modify any conventional deep architecture to automatically provide more transparent explanations for classification decisions, as well as an intuitive notion of the credibility of each prediction is proposed, drawing on ideas from nonparametric kernel regression.
- 42
- A Neural Model for Language Identification in Code-Switched Tweets42
A hierarchical neural model is applied to this task, learning character and contextualized word-level representations to make word- level language predictions, which performs well on both the 2014 and 2016 versions of the shared task.
- Polyhedral outer approximations with application to natural language parsing42
This paper establishes risk bounds for max-margin learning with LP relaxed inference and addresses the second issue by proposing a new paradigm that attempts to penalize "time-consuming" hypotheses.
- Plug and Play Autoencoders for Conditional Text Generation41
Evaluations on style transfer tasks both with and without sequence-to-sequence supervision show that the proposed plug and play Emb2Emb method performs better than or comparable to strong baselines while being up to four times faster.
- Rational Recurrences41
It is shown that several recent neural models use rational recurrences, and one such model is presented, which performs better than two recent baselines on language modeling and text classification and demonstrates that transferring intuitions from classical models like WFSAs can be an effective approach to designing and understanding neural models.
- 40
- 40
- Open Extraction of Fine-Grained Political Statements40
This work presents several models that exploit the structure that exists between people and the assertions they make to learn latent positions of people and propositions at the same time, and evaluates them on a novel dataset of propositions judged on a political spectrum.
- Dyna: a declarative language for implementing dynamic programs39
The first version of a new declarative programming language, designed especially for rapid development of new statistical NLP systems, is presented, which intends for it to generalize and encapsulate best practices, and serve as a testbed for new practices.
- Cairo: An Alignment Visualization Tool39
Cairo is a free, open-source, portable, user-friendly, GUI-driven program written in Java that provides a visual representation of word correspondences between bilingual pairs of sentences, as well as relevant translation model parameters.
- 38
- 38
- 38
- Contextual word representations37
Advances in how programs treat natural language words have a big impact in AI, and this research highlights the need to understand these words in more detail.
- 37
- 36
- Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models36
This work presents Choose Your Own Adventure, a collaborative writing setup for pairwise model evaluation, where two models generate suggestions to people as they write a short story; writers are asked to choose one of the two suggestions, and they observe which model’s suggestions they prefer.
- Writing Strategies for Science Communication: Data and Computational Analysis36
This work compiles a set of writing strategies drawn from a wide range of prescriptive sources and develops an annotation scheme allowing humans to recognize them, and collects and annotates a subset of this corpus.
- Dynamic Language Models for Streaming Text36
A probabilistic language model that captures temporal dynamics and conditions on arbitrary non-linguistic context features that serve as important indicators of language changes that are otherwise difficult to capture using text data by itself is presented.
- Know Your Audience: The benefits and pitfalls of generating plain language summaries beyond the "general" audience35
While simpler text led to the best reading experience for readers with little to no familiarity in a topic, high familiarity readers tended to ignore certain details in overly plain summaries (e.g., study limitations).
- Topics to Avoid: Demoting Latent Confounds in Text Classification35
This work proposes a method that represents the latent topical confounds and a model which “unlearns” confounding features by predicting both the label of the input text and the confound; but it shows that this model generalizes better and learns features that are indicative of the writing style rather than the content.
- Modeling User Arguments, Interactions, and Attributes for Stance Prediction in Online Debate Forums35
This work proposes an integrated model to leverage side information, a regression-based latent factor model which jointly models user arguments, interactions, and attributes and demonstrates in experiments that the method has promising results on both micro-level and macro-level stance prediction.
- Promoting Graph Awareness in Linearized Graph-to-Text Generation34
This work uses graph-denoising objectives implemented in a multi-task text-to-text framework and finds that these denoising scaffolds lead to substantial improvements in downstream generation in low-resource settings.
- Transforming Dependencies into Phrase Structures34
This work presents a new algorithm for transforming dependency parse trees into phrase-structure parse trees that is faster than traditional phrasestructure parsing and achieves near the state of the art on both benchmarks.
- Specializing Multilingual Language Models: An Empirical Study33
These evaluations on part-of-speech tagging, universal dependency parsing, and named entity recognition in nine diverse low-resource languages uphold the viability of these approaches while raising new questions around how to optimally adapt multilingual models to low- resource settings.
- 33
- 33
- Predicting Merger Targets and Acquirers from Text33
This study explores the use of a U.S. SEC’s SEC lings to predict whether the rm will be an acquirer or a target of an acquisition within a year of the ling and shows that word and phrase features can be incorporated into predictive models.
- 33
- Vine parsing and minimum risk reranking for speed and precision33
This work provides a detailed error analysis of parses produced for sentences in German and Arabic and describes sources of cross-lingual error and ways to ameliorate them.
- Fluid Language Model Benchmarking32
Fluid Benchmarking is introduced, a new evaluation approach that advances language model benchmarking across multiple dimensions based on the insight that the relative value of benchmark items depends on an LM's capability level, suggesting that evaluation should adapt to each LM.
- GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation32
This work considers design choices for the annotation interface used to elicit human judgments and their impact on reproducibility, and develops an automated mechanism for maintaining annotator quality via a probabilistic model that detects and excludes noisy annotators.
- Sentence Bottleneck Autoencoders from Transformer Language Models32
The construction of a sentence-level autoencoder from a pretrained, frozen transformer language model that achieves better quality than previous methods that extract representations from pretrained transformers on text similarity tasks, style transfer, and single-sentence classification tasks in the GLUE benchmark, while using fewer parameters than large pretrained models.
- 32
- 31
- Unsupervised Bitext Mining and Translation via Self-Trained Contextual Embeddings31
It is demonstrated that unsupervised bitext mining is an effective way of augmenting MT datasets and complements existing techniques like initializing with pre-trained contextual embeddings.
- 31
- PointArena: Probing Multimodal Grounding Through Language-Guided Pointing30
This work conducts extensive evaluations of both state-of-the-art open-source and proprietary multimodal models and finds that Molmo-72B consistently outperforms other models, though proprietary models increasingly demonstrate comparable performance.
- 30
- 30
- 29
- 29
- 29
- 28
- 28
- 28
- Feature-rich translation by quasi-synchronous lattice parsing28
A novel decoder based on lattice parsing with quasi-synchronous grammar that does not require source and target trees to be isomorphic is presented, using generic approximate dynamic programming techniques to handle "non-local" features.
- Relative keyboard input system28
A "relative keyboard," where keystrokes are treated as inputs in a continuous space relative to each other, instead of a discrete, unambiguous sequence, so that a user with the ability to touch-type may type anywhere on the sensing surface without the need for a visual keyboard.
- On Linear Representations and Pretraining Data Frequency in Language Models27
The strength of linear representations in LMs contains signal about the models' pretraining corpora that may provide new avenues for controlling and improving model behavior: particularly, manipulating the models' training data to meet specific frequency thresholds.
- End-to-End Neural Segmental Models for Speech Recognition27
This work reviews neural segmental models, which can be viewed as consisting of a neural network-based acoustic encoder and a finite-state transducer decoder and explores training approaches, including multistage versus end-to-end training and multitask training that combines segmental and frame-level losses.
- Morphosyntactic probing of multilingual BERT models26
An extensive dataset for multilingual probing of morphological information in language models is introduced and it is found that pre-trained Transformer models (mBERT and XLM-RoBERTa) learn features that attain strong performance across these tasks.
- 26
- 26
- Measuring Online Debaters’ Persuasive Skill from Text over Time25
It is found that an interaction feature combining uncertainty cues (hedging) with terms strongly associated with either side of a particular debate (fightin’ words) is more predictive than either feature on its own, indicating the importance of fine- grained linguistic features.
- 25
- From words to corpora: recognizing translation25
This paper presents a technique for discovering translationally equivalent texts comprised of the application of a matching algorithm at two different levels of analysis and a well-founded similarity score that is adaptable to varying levels of multilingual resource availability.
- Explain like I am a Scientist: The Linguistic Barriers of Entry to r/science24
It is shown that r/science uses a specialized language that is distinct from other subreddits, and it is suggested that the specialized language used in r/ science has a gatekeeping effect, preventing participation by people whose language does not align with that used inr/science.
- Bridging CNNs, RNNs, and Weighted Finite-State Machines24
SoPa combines neural representation learning with weighted finite-state automata (WFSAs) to learn a soft version of traditional surface patterns, and it is shown that SoPa is an extension of a one-layer CNN, and that such CNNs are equivalent to a restricted version of SoPa, and accordingly, to arestricted form of WFSA.
- Multitask Learning with CTC and Segmental CRF for Speech Recognition24
It is found that this multitask objective improves recognition accuracy when decoding with either the SCRF or CTC models, and it is shown that CTC can also be used to pretrain the RNN encoder, which improves the convergence rate when learning the joint model.
- 24
- Low-Resource Parsing with Crosslingual Contextualized Representations23
The non-contextual part of the learned language models are examined to demonstrate that polyglot language models better encode crosslingual lexical correspondence compared to aligned monolingual language models, providing further evidence thatpolyglot training is an effective approach toCrosslingual transfer.
- 23
- "You are no Jack Kennedy": On Media Selection of Highlights from Presidential Debates22
A three- decade dataset of presidential debate transcripts and post-debate coverage is built and it is found that crowdworkers can only achieve an accuracy of 60% in this task, indicating that media choices are not entirely obvious.
- Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers21
This work extensively experiment with transformers trained on five synthetic, controlled datasets and establishes a correlation between whether transformers generalize hierarchically on a dataset and if the simplest explanation of that dataset is provided by a hierarchical grammar compared to regular grammars exhibiting linear generalization.
- 21
- Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?20
A novel attack based on a previously overlooked source of information: byte-pair encoding (BPE) tokenizers, used by the vast majority of modern language models, is introduced, showing that the ordered list of merge rules learned by a BPE tokenizer naturally reveals information about the token frequencies in its training data.
- 20
- Citation Text Generation.20
This paper establishes the task of citation text generation with a standard evaluation corpus and develops several strong baseline models for this task, and provides extensive automatic and human evaluations to illustrate the successes and shortcomings of current text generation techniques.
- PaLM: A Hybrid Parser and Language Model20
PaLM is presented, a hybrid parser and neural language model that adds an attention layer over text spans in the left context, and empirically shows that it outperforms strong baselines on language modeling performance.
- 20
- 20
- 20
- Eval3D: Interpretable and Fine-Grained Evaluation for 3D Generation19
Eval3D is introduced, a fine-grained, interpretable evaluation tool that can faithfully evaluate the quality of generated 3D assets based on various distinct yet complementary criteria and comprehensively evaluate existing 3D generation models using Eval3D and highlight the limitations and challenges of current models.
- 19
- Multilevel Text Alignment with Cross-Document Attention19
This work proposes a new learning approach that equips previously established hierarchical attention encoders for representing documents with a cross-document attention component, enabling structural comparisons across different levels (document-to-document and sentence- to-document).
- 18
- 18
- Transparency Helps Reveal When Language Models Learn Meaning17
The systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models successfully learn to emulate semantic relations between expressions, but this ability degrades when the language otherwise unmodified is changed to be context-dependent.
- 17
- 17
- LSTMs Exploit Linguistic Attributes of Data17
It is shown that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input, and it is hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss.
- 17
- Greedy Transition-Based Dependency Parsing with Stack LSTMs17
A greedy transition-based parser that learns to represent parser states using recurrent neural networks using a new control structure for sequential neural networks—the stack long short-term memory unit (LSTM).
- 16
- Friends with Motives: Using Text to Infer Influence on SCOTUS16
A probabilistic model of the influence of language on the behavior of the U.S. Supreme Court, specifically influence of amicus briefs on Court decisions and opinions, leads to improved predictions of justices’ votes and perplexity of opinion language.
- 16
- Friendships, Rivalries, and Trysts: Characterizing Relations between Ideas in Texts15
This work proposes the first framework to systematically characterize the relations between ideas based on their occurrence in a corpus of documents, independent of how these ideas are represented, and reveals a number of different ways in which ideas can cooperate and compete.
- One Parser, Many Languages.15
It is found that multilingual training outperforms standard supervised training on a single language, and that generalization to unseen languages is competitive with existing model-transfer approaches.
- CMU: Arc-Factored, Discriminative Semantic Dependency Parsing15
This work presents an arc-factored statistical model for semantic dependency parsing, as defined by the SemEval 2014 Shared Task 8 on Broad-Coverage Semantic Dependency Parsing, which placed second in the competition.
- Learning Structured Classifiers with Dual Coordinate Ascent15
A new aggressive online algorithm is introduced that optimizes any loss in this family of convex loss functions, properly including CRFs, structured SVMs, and the structured perceptron, without the need to specify any learning rate parameter.
- 14
- 14
- Discovering Phonesthemes with Sparse Regularization14
A simple method for extracting non-arbitrary form-meaning representations from a collection of semantic vectors and applies this model to the problem of automatically discovering phonesthemes, which are submorphemic sound clusters that appear in words with similar meaning.
- Phrase Dependency Machine Translation with Quasi-Synchronous Tree-to-Tree Features14
A tree-to-tree machine translation system inspired by quasi-synchronous grammar is presented that combines phrases and dependency syntax, integrating the advantages of phrase-based and syntax-based translation.
- Dynamic Programming Algorithms as Products of Weighted Logic Programs14
The PRODUCT transformation is described, which can merge two weighted logic programs into a new one and optimizes a product of proof scores from the original programs, constituting a scoring function known in machine learning as a "product of experts."
- RNN Architecture Learning with Sparse Regularization13
This work applies group lasso to rational RNNs (Peng et al., 2018), a family of models that is closely connected to weighted finite-state automata (WFSAs) and shows that sparsifying such models makes them easier to visualize, and presents models that rely exclusively on as few as three WFSAs after pruning more than 90% of the weights.
- 13
- Framing Effects13
It is found that slogan frames impact recruitment significantly; changing the slogan frame from a 'supporting science' frame to a 'comparing oneself to others' frame lead to a 9% increase in recruitment for some studies.
- Overview of the 2014 NLP Unshared Task in PoliInformatics13
In this activity, participants were offered a corpus of text relevant to the 2007‐8 financial crisis and an open-ended prompt and their responses took the form of a short paper and an optional demonstration, to which a panel of judges will respond with the goal of identifying efforts with the greatest potential for future interdisciplinary collaboration.
- Empirical Risk Minimization for Probabilistic Grammars: Sample Complexity and Hardness of Learning13
This work presents a framework for empirical risk minimization of probabilistic grammars using the log-loss, and derives sample complexity bounds in this framework that apply both to the supervised setting and the unsupervised setting.
- 13
- 12
- 12
- 12
- 12
- 12
- 12
- A Taxonomy of Ambiguity Types for NLP11
This work proposes a taxonomy of ambiguity types as seen in English to facilitate NLP analysis and can help make meaningful splits in language ambiguity data, allowing for more fine-grained assessments of both datasets and model performance.
- 11
- 11
- 11
- 11
- 11
- 11
- 11
- 10
- Language Models: A Guide for the Perplexed10
The perspective of researchers and educators can add some clarity to the public's understanding of the technologies beyond what's currently available, which tends to be either extremely technical or promotional material generated about products by their purveyors.
- 10
- 10
- UW-CSE at SemEval-2016 Task 10: Detecting Multiword Expressions and Supersenses using Double-Chained Conditional Random Fields10
This entry to SemEval 2016 Task 10: Detecting Minimal Semantic Units and their Meanings uses a discriminative sequence model similar to Schneider and Smith (2015) and factors the labels into multiword expression and supersense labels.
- A Utility Model of Authors in the Scientific Community10
A probabilistic model of some of the important aspects of the writing of a scientific paper is introduced: that authors have individual preferences, that writing a paper requires trading off among the preferences of authors as well as extrinsic rewards in the form of community response to their papers.
- Variational inference for grammar induction with prior knowledge10
It is shown how a looser restriction on the approximate posterior, requiring it to be a mixture, can help inject prior knowledge to exploit soft constraints during the variational E-step.
- Detection of Translational Equivalence10
Experimental results which support the hypothesis that translational equivalence is empirically observable and suggest profitable directions for improving performance on this recognition task are shown.
- Sampling from Your Language Model One Byte at a Time9
An inference-time method to convert any autoregressive LM with a BPE tokenizer into a character-level or byte-level LM and is also able to unify the vocabularies of language models with different tokenizers, allowing one to ensemble LMs with different tokenizers at inference time or transfer the post-training from one model to another using proxy-tuning.
- Expected Validation Performance and Estimation of a Random Variable’s Maximum9
This paper analyzes three statistical estimators for expected validation performance, a tool used for reporting performance as a function of computational budget, and finds that the two biased estimators lead to the fewest incorrect conclusions, which hints at the importance of minimizing variance and MSE.
- 9
- 9
- 8
- Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation8
Pentathlon is a benchmark for holistic and realistic evaluation of model efficiency, which focuses on inference, which accounts for a majority of the compute in a model's lifecycle, and is designed to mirror real-world applications scenarios.
- Grounded Compositional Outputs for Adaptive Language Modeling8
This work proposes a fully compositional output embedding layer for language models, which is further grounded in information from a structured lexicon (WordNet), namely semantically related words and free-text definitions, and is the first word-level language model with a size that does not depend on the training vocabulary.
- 7
- Risks and NLP Design: A Case Study on Procedural Document QA7
This case study shows that an existing language model, applied in "zero-shot"mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web.
- 7
- Parameter Norm Growth During Training of Transformers.7
The tendency of transformer parameters to grow in magnitude during training is studied to find that in certain contexts, GD increases the parameter $L_2$ norm up to a threshold that itself increases with training-set accuracy, which means increasing training accuracy over time enables the norm to increase.
- Weakly-Supervised Bayesian Learning of a CCG Supertagger7
This work presents a Bayesian formulation for weakly-supervised learning of a Combinatory Categorial Grammar (CCG) supertagger with an HMM and shows that it yields substantial improvements over previous work that used similar biases to initialize an EM-based learner.
- 7
- Competitive grammar writing7
A well-tested introductory activity in which teams of mixed background compete to write probabilistic context-free grammars of English, which brings together symbolic, Probabilistic, algorithmic, and experimental issues in a way that is accessible to novices and enjoyable.
- 6
- Exploring the Effect of Author and Reader Identity in Online Story Writing: the STORIESINTHEWILD Corpus.6
Compared to younger readers, readers age 45 and older consider stories significantly less creative and less entertaining, and suggest that reader and writer demographics, as well as writing setup, should be accounted for in story writing evaluations.
- 6
- Simplified Dependency Annotations with GFL-Web6
GFL-Web, a web-based interface for syntactic dependency annotation with the lightweight FUDG/GFL formalism, is presented, showing that even novices were, with a bit of training, able to rapidly annotate the syntax of English Twitter messages.
- Structured Sparsity in Natural Language Processing: Models, Algorithms and Applications6
This tutorial will cover recent advances in sparse modeling with diverse applications in natural language processing (NLP) by focusing on methods which embed sparse model selection into the parameter estimation problem.
- 5
- Summarization-Based Document IDs for Generative Retrieval with Language Models5
This work finds that abstractive, content-based IDs (ACID) and an ID based on the first 30 tokens are very effective in direct comparisons with previous approaches to ID creation and observes that extractive IDs outperformed abstractive IDs on Wikipedia articles in NQ but not the snippets in MSMARCO, which suggests that document characteristics affect generative retrieval performance.
- 5
- 5
- Semi-Supervised Learning of Sequence Models with Method of Moments5
A fast and scalable method for semi-supervised learning of sequence models, based on anchor words and moment matching, that can handle hidden Markov models with feature-based log-linear emissions and can learn from very few annotated sentences.
- 4
- 4
- Modeling Considerations for Quantitative Social Science Research Using Social Media Data4
A particularly interesting aspect of this meeting was discussion of the different disciplines’ requirements and approaches to modeling and the different considerations that are used to assess model fit.
- 4
- 4
- A Supertag-Context Model for Weakly-Supervised CCG Parser Learning4
A novel parsing model with the capacity to capture the associative adjacent-category relationships intrinsic to CCG by parameterizing the relationships between each constituent label and the preterminal categories directly to its left and right, biasing the model toward constituent categories that can combine with their contexts.
- 4
- Statements for Factual Question Generation4
Experimental results indicate that the method presented is more suitable for factual question generation applications than an alternative text compression algorithm.
- Leveraging structural relations for fluent compressions at multiple compression rates4
This work proposes and successfully evaluates a more comprehensive, generalizable feature set that takes syntactic and structural relationships into account in order to sustain variable compression rates while making compressed sentences more coherent, grammatical and readable.
- 3
- 3
- 3
- 3
- Linguistic structure prediction with the sparseptron3
This research presents a meta-modelling architecture that automates the very labor-intensive and therefore time-heavy and therefore expensive and expensive process of manually modeling language representations.
- Empirical Risk Minimization with Approximations of Probabilistic Grammars3
This work presents a framework for empirical risk minimization of the parameters of a fixed probabilistic grammar using the log-loss, and derives sample complexity bounds in this framework that apply both to the supervised setting and the un-supervised setting.
- How Performance Pressure Influences AI-Assisted Decision Making2
The effects of performance pressure is shown on AI advice reliance when laypeople complete a common AI-assisted task and thus have inherently low performance pressure by leveraging people’s loss aversion towards potential monetary gains when completing a task.
- 2
- 2
- 2
- 2
- Multilingual and Interlingual Semantic Representations for Natural Language Processing: A Brief Introduction2
The Computational Linguistics special issue on Multilingual and Interlingual Semantic Representations for Natural Language Processing is introduced, with a brief summary of the work in the issue, which includes developments on lexical and sentential semantic representations, from symbolic and neural perspectives.
- 2
- 2
- Random Search for Hyperparameters using Determinantal Point Processes.2
An approach that transforms hyperparameter search spaces for efficient use with a k-DPP is described that shows significant benefits over uniform random search in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel.
- 2
- 2
- 2
- 2
- Named entity recognition from Arabic Wikipedia2
This work investigates the Arabic NER problem from the Arabic Wikipedia text, and employs statistical sequence labeling methods for solving the NER task and evaluates the proposed framework in four different text domains ofArabic Wikipedia.
- 2
- 2
- Computational modeling and preclinical validation support targeting somatic instability for Huntington’s disease treatment1
Predictive modeling is used to guide therapeutic targeting of disease modifiers in Huntington’s disease and support the translational potential of AAV-DB-3.miMSH3 as a disease-modifying therapy applicable for HD patients with a broad range of inherited repeat lengths.
- 1
- 1
- 1
- 1
- CPS-TaskForge: Generating Collaborative Problem Solving Environments for Diverse Communication Tasks1
- 1
- 1
- 1
- Dyna: A Language for Weighted Dynamic Programming1
The first version of a new declarative programming language, designed especially for rapid development of new statistical NLP systems, is presented, which aims to generalize and encapsulate best practices, and serve as a testbed for new practices.
- –
- –
- –
- –
- –
- –
- –
- Third-Party Language Model Performance Prediction from Instruction–
This work proposes a third party performance prediction framework, where a separate model is trained to predict the metric resulting from evaluating an instruction-following system on a task while assuming access only to its inputs and outputs at inference time.
- LEXPLAIN: Improving Model Explanations via Lexicon Supervision–
This work proposes a novel framework for guiding model explanations by supervising them explicitly by using task-related lexicons to directly supervise model explanations, which consistently improves the model’s explanations without sacrificing performance on the task, as well as demonstrating on sentiment analysis and toxicity detection.
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- Natural Language Processing: Algorithms and Applications, Old and New–
The probabilistic view is sometimes useful (the authors will see this later), and the choice for y [L] is then: argmax λ ( max λ′ wφlocal(x, L, λ ′, �L) + V [L− 1, �l′] )
- Supplementary appendix to "Learning to Extract International Relations from Political Context" (ACL 2013)–
This work evaluated a singleQ/N where nwk and nvk are averaged counts from multiple samples—using this corresponds to a factored, mean-field-like approximation to the posterior—but it also was very similar to using a single sample.
- –
- –
- –
- –
- –
- Instructions for ACL-08 Proceedings–
This document contains the instructions for preparing a camera-ready manuscript for the proceedings of ACL-08, and is therefore an example of what your manuscript should look like.
- Rich entity recognition in English text–
The task can be performed using machine learning techniques to train a system that recognizes entities with performance comparable to a human annotator, using a large annotated training data corpus.
- –
- DARPA CS Study Panel 2007–
Because my research deals with information -- in particular, automated processing of language data like text and speech -- I saw the greatest connection with my research in the final session when the authors interact with the intelligence community.
- Instructions for NAACL HLT 2009 Proceedings–
This document contains the instructions for preparing a camera-ready manuscript for the proceedings of NAACL HLT 2009, and is therefore an example of what your manuscript should look like.
- Text Based Portfolio Choice–
This work proposes to investigate the usefulness of textual company-level materials in producing better estimates for firms' return distributions, and aims to combine textual and quantitative data to best predict the covariance structure of firms' returns.
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-14. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.