Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Lide WuSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works401 from public data
- How to Fine-Tune BERT for Text Classification?1,861
A general solution for BERT fine-tuning is provided and new state-of-the-art results on eight widely-studied text classification datasets are obtained.
- Pre-trained models for natural language processing: A survey1,752
This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.
- A survey of transformers1,591
This survey provides a comprehensive review of various Transformer variants and proposes a new taxonomy of X-formers from three perspectives: architectural modification, pre-training, and applications.
- Pre-trained models: Past, present and future1,134
A deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, is taken to reveal the crucial position of PTMs in the AI development spectrum.
- 1,008
- BERT-ATTACK: Adversarial Attack Against BERT Using BERT936
This paper proposes a high-quality and effective method to generate adversarial samples using pre-trained masked language models exemplified by BERT against its fine-tuned models and other deep neural models for downstream tasks and successfully misleads the target models to predict incorrectly.
- Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence748
This paper constructs an auxiliary sentence from the aspect and converts ABSA to a sentence-pair classification task, such as question answering (QA) and natural language inference (NLI), and fine-tune the pre-trained model from BERT.
- Adversarial Multi-task Learning for Text Classification666
This paper proposes an adversarial multi-task learning framework, alleviating the shared and private latent feature spaces from interfering with each other, and shows that the shared knowledge learned can be regarded as off-the-shelf knowledge and easily transferred to new tasks.
- The rise and potential of large language model based agents: a survey656
A comprehensive survey on LLM-based agents is conducted, covering their construction frameworks, application scenarios, and the exploration of societies built upon LLM-based agents.
- QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization534
This work defines a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and introduces QMSum, a new benchmark for this task.
- Extractive Summarization as Text Matching529
This paper forms the extractive summarization task as a semantic text matching problem, in which a source document and candidate summaries will be matched in a semantic space to create a semantic matching framework.
- FLAT: Chinese NER Using Flat-Lattice Transformer459
Experiments show FLAT outperforms other lexicon-based models in performance and efficiency and can fully leverage the lattice information and has an excellent parallel ability.
- InternLM2 Technical Report440
InternLM2 is introduced, an open-source LLM that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks, long-context modeling, and open-ended subjective evaluations through innovative pre-training and optimization techniques.
- A Unified Generative Framework for Aspect-based Sentiment Analysis382
Every subtask target is redefined as a sequence mixed by pointer indexes and sentiment class indexes, which converts all ABSA subtasks into a unified generative formulation and exploits the pre-training sequence-to-sequence model BART to solve all ABsa subtasks in an end- to-end framework.
- A Unified Generative Framework for Various NER Subtasks377
This work proposes to formulate the NER subtasks as an entity span sequence generation task, which can be solved by a unified sequence-to-sequence (Seq2Seq) framework, and can leverage the pre-trained Seq1Seq model to solve all three kinds of N ER subtasks without the special design of the tagging schema or ways to enumerate spans.
- LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models318
This work comprehensively analyzed multiple state-of-the-art VLA models and revealed consistent brittleness beneath apparent competence, challenging the assumption that high benchmark scores equate to true competency and highlighting the need for evaluation practices that assess reliability under realistic variation.
- Heterogeneous Graph Neural Networks for Extractive Document Summarization306
This paper presents a heterogeneous graph-based neural network for extractive summarization (HETERSUMGRAPH), which contains semantic nodes of different granularity levels apart from sentences that act as the intermediary between sentences and enrich the cross-sentence relations.
- Star-Transformer296
Star-Transformer replaces the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are connected through a shared relay node, and complexity is reduced from quadratic to linear, while preserving the capacity to capture both local composition and long-range dependency.
- Long Short-Term Memory Neural Networks for Chinese Word Segmentation293
A novel neural network model for Chinese word segmentation is proposed, which adopts the long short-term memory (LSTM) neural network to keep the previous important information in memory cell and avoids the limit of window size of local context.
- Secrets of RLHF in Large Language Models Part I: PPO290
The PPO-max, an advanced version of PPO algorithm, is explored, to efficiently improve the training stability of the policy model, and a comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT is performed.
- GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge289
This paper construct context-gloss pairs and propose three BERT based models for WSD and fine-tune the pre-trained BERT model to achieve new state-of-the-art results on WSD task.
- Convolutional neural tensor network architecture for community-based question answering279
This paper proposes a convolutional neural tensor network architecture to encode the sentences in semantic space and model their interactions with a tensor layer, which outperforms the other methods on two matching tasks.
- 242
- VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks231
VLABench is an open-source benchmark for evaluating universal LCM task learning and provides high-quality training data collected via an automated framework incorporating heuristic skills and prior information to support the downstream finetuning.
- CoLAKE: Contextualized Language and Knowledge Embedding228
The Contextualized Language and Knowledge Embedding (CoLAKE) is proposed, which jointly learns contextualized representation for both language and knowledge with the extended MLM objective, and achieves surprisingly high performance on a synthetic task called word-knowledge graph completion, which shows the superiority of simultaneously contextualizing language andknowledge representation.
- Evaluating the Performance of Large Language Models on GAOKAO Benchmark225
GAOKAO-Bench is introduced, an intuitive benchmark that employs questions from the Chinese GAOKAO examination as test samples, including both subjective and objective questions that contribute a robust evaluation benchmark for future large language models and offers valuable insights into the advantages and limitations of such models.
- Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation222
The Style Transformer is proposed, which makes no assumption about the latent representation of source sentence and equips the power of attention mechanism in Transformer to achieve better style transfer and better content preservation.
- Reinforced Mnemonic Reader for Machine Reading Comprehension216
The Reinforced Mnemonic Reader for machine reading comprehension tasks, which enhances previous attentive readers in two aspects: a reattention mechanism is proposed to refine current attentions by directly accessing to past attentions that are temporally memorized in a multi-round alignment architecture.
- Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning203
A stronger weight-poisoning attack method is proposed that introduces a layerwise weight poisoning strategy to plant deeper backdoors and introduces a combinatorial trigger that cannot be easily detected.
- Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa197
This paper compares the induced trees from PTMs and the dependency parsing trees on several popular models for the ABSA task, showing that the induced tree from fine-tuned RoBERTa (FT-RoBERTa) outperforms the parser-provided tree and reveals that the FT-RoberTa Induced Tree is more sentiment-word-oriented and could benefit theABSA task.
- 185
- Cached Long Short-Term Memory Neural Networks for Document-Level Sentiment Classification183
A Cached Long Short-Term Memory neural networks (CLSTM) to capture the overall semantic information in long texts and introduces a cache mechanism, which divides memory into several groups with different forgetting rates and thus enables the network to keep sentiment information better within a recurrent unit.
- CPT: a pre-trained unbalanced transformer for both Chinese language understanding and generation182
A novel Chinese pre-trained unbalanced transformer (CPT) designed to utilize the shared knowledge between natural language understanding (NLU) and natural language generation (NLG) to boost the performance and save the computational and storage cost.
- Multi-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents180
A multi-timescale long short-termmemory (MT-LSTM) neural network to model long texts that can model very long documents as well as short sentences is proposed.
- Secrets of RLHF in Large Language Models Part II: Reward Modeling179
This report introduces contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses, thereby improving model generalization and employs meta-learning to enable the reward model to maintain the ability to differentiate subtle differences in out-of-distribution samples.
- Adversarial Multi-Criteria Learning for Chinese Word Segmentation178
This paper proposes adversarial multi-criteria learning for CWS by integrating shared knowledge from multiple heterogeneous segmentation criteria, and shows that the performance of each corpus obtains a significant improvement, compared to single-criterion learning.
- Improving Image Captioning with Better Use of Caption170
A novel image captioning architecture is presented to better explore semantics available in captions and leverage that to enhance both image representation and caption generation, resulting in the state-of-the-art performance under a wide range of evaluation metrics.
- EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education168
This initiative aims to promote research and applications of LLMs for intelligent education by learning domain-specific knowledge by pre-training on the educational corpus and stimulating various skills with tool use by fine- tuning on designed system prompts and instructions.
- Searching for Effective Neural Extractive Summarization: What Works and What’s Next165
This paper seeks to better understand how neural extractive summarization systems could benefit from different types of model architectures, transferable knowledge and learning schemas, and finds an effective way to improve the current framework.
- Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders160
A suite of 256 SAEs, trained on each layer and sublayer of the Llama-3.1-8B-Base model, with 32K and 128K features are introduced and modifications to a state-of-the-art SAE variant, Top-K SAEs, are evaluated across multiple dimensions.
- Learning Sparse Sharing Architectures for Multiple Tasks159
This paper proposes a novel parameter sharing mechanism, named Sparse Sharing, which achieves consistent improvement while requiring fewer parameters, and shows that both hard sharing and hierarchical sharing can be formulated as particular instances of the sparse sharing framework.
- 155
- SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models150
SpeechTokenizer is proposed, a unified speech tokenizer for speech large language models that adopts the Encoder-Decoder architecture with residual vector quantization (RVQ) and construct a Unified Speech Language Model (USLM) leveraging SpeechTokenizer.
- Scaling Laws of RoPE-based Extrapolation147
This work proposes a unified framework from the periodic perspective, to describe the relationship between the extrapolation performance and base value as well as tuning context length, and achieves extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.
- Finding Support Examples for In-Context Learning144
A novel metric is proposed, InfoScore, to evaluate the example's in-context informativeness based on the language model's feedback, and further proposed a progressive filtering process to filter out uninformative examples.
- A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook142
This article introduces seminal foundation models proposed or adaptable for reasoning, highlighting the latest advancements in various reasoning tasks, methods, and benchmarks, and delves into the potential future directions behind the emergence of reasoning abilities within foundation models.
- InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning137
This paper introduces and open-source their math reasoning LLMs InternLM-Math which is continue pre-trained from InternLM2 and unify chain-of-thought reasoning, reward modeling, formal reasoning, data augmentation, and code interpreter in a unified seq2seq format and supervise the authors' model to be a versatile math reasoner, verifier, prover, and augmenter.
- Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information137
It is the first time to verify that multiple low-resource language pairs can be utilized to improve rich resource MT, and mRASP is even able to improve the translation quality on exotic languages that never occur in the pre-training corpus.
- Knowledge Graph Representation with Jointly Structural and Textual Encoding136
This paper introduces three neural models to encode the valuable information from text description of entity, among which an attentive model can select related information as needed, and proposes a novel deep architecture to utilize both structural and textual information of entities.
- Contrast and Generation Make BART a Good Dialogue Emotion Recognizer135
This work adopts supervised contrastive learning to make different emotions mutually exclusive to identify similar emotions better and utilizes an auxiliary response generation task to enhance the model's ability of handling context information, thereby forcing the model to recognize emotions with similar semantics in diverse contexts.
- 129
- Gated Recursive Neural Network for Chinese Word Segmentation121
A gated recursive neural network (GRNN) for Chinese word segmentation is proposed, which contains reset and update gates to incorporate the complicated combinations of the context characters.
- YuE: Scaling Open Foundation Models for Long-Form Music Generation114
This work introduces YuE, a family of open foundation models based on the LLaMA2 architecture that scales to trillions of tokens and generates up to five minutes of music while maintaining lyrical alignment, coherent musical structure, and engaging vocal melodies with appropriate accompaniment.
- AgentGym: Evolving Large Language Model-based Agents across Diverse Environments113
This paper proposes AgentGym, a new framework featuring a variety of environments and tasks for broad, real-time, uni-format, and concurrent agent exploration, and proposes a novel method, AgentEvol, to investigate the potential of agent self-evolution beyond previously seen data across tasks and environments.
- Toward Diverse Text Generation with Inverse Reinforcement Learning110
In inverse reinforcement learning (IRL), the IRL framework learns a reward function on training data, and then an optimal policy to maximum the expected total reward, which can produce more dense reward signals.
- Learning context-sensitive word embeddings with neural tensor skip-gram model106
A general architecture to learn the word and topic embeddings efficiently is presented, which is an extension to the Skip-Gram model and can model the interaction between words and topics simultaneously.
- Meta Multi-Task Learning for Sequence Modeling93
A shared meta-network is used to capture the meta-knowledge of semantic composition and generate the parameters of the task-specific semantic composition models in a new sharing scheme of composition function across multiple tasks.
- Paradigm Shift in Natural Language Processing92
This paper highlights several paradigms that have the potential to solve different NLP tasks, including the sequence labeling paradigm, and the classification paradigm, to build a single model to handle diverse tasks.
- KNN-Contrastive Learning for Out-of-Domain Intent Classification89
The density-based novelty detection algorithm is so well-grounded in the essence of the method that it is reasonable to use it as the OOD detection algorithm without making any requirements for the feature distribution, and a simple yet effective method, named KNN-contrastive learning, is proposed.
- Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration87
This paper introduces Corex, a suite of novel general-purpose strategies that transform LLMs into autonomous agents pioneering multi-model collaborations for complex task-solving, and demonstrates the cost-effectiveness of the method, facilitating collaboration among different LLMs and promoting annotation efficiency.
- Implicit Discourse Relation Detection via a Deep Architecture with Gated Relevance Network84
The use of word embeddings to replace the original words is proposed to overcome the data sparsity problem, and a gated relevance network is adopted to capture the semantic interaction between word pairs.
- Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation83
This paper improves the fine-tuning of BERT with two effective mechanisms: self-ensemble and self-distillation, and achieves a new state-of-the-art result on the SNLI dataset.
- BBTv2: Towards a Gradient-Free Future with Large Language Models83
BBTv2 is presented, an improved version of Black-Box Tuning, to drive PTMs for few-shot learning and a divide-and-conquer gradient-free algorithm to optimize the prompts at different layers alternately is proposed.
- FudanNLP: A Toolkit for Chinese Natural Language Processing82
FudanNLP is an open source toolkit for Chinese natural language processing (NLP), which uses statistics-based and rule-based methods to deal with Chinese NLP tasks, such as word segmentation, part-ofspeech tagging, named entity recognition, dependency parsing, time phrase recognition, anaphora resolution and so on.
- 80
- 80
- 77
- Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety76
This survey provides a systematic review of current safety research on large models, covering Vision Foundation Models, Large Language Models, Vision-Language Pre-training models, Vision-Language Models, Diffusion Models, and large-model-powered Agents, and presents a comprehensive taxonomy of safety threats to these models.
- SpeechAlign: Aligning Speech Generation to Human Preferences74
SpeechAlign is introduced, an iterative self-improvement strategy that aligns speech language models to human preferences that can bridge the distribution gap and facilitate continuous self-improvement of the speech language model.
- Training-Free Long-Context Scaling of Large Language Models74
Dual Chunk Attention (DCA) is proposed, which enables Llama2 70B to support context windows of more than 100k tokens without continual training, and achieves performance on practical long-context tasks that is comparable to or even better than that of finetuned models.
- A Survey of Reasoning with Foundation Models73
This paper introduces seminal foundation models proposed or adaptable for reasoning, highlighting the latest advancements in various reasoning tasks, methods, and benchmarks and discusses the relevance of multimodal learning, autonomous agents, and super alignment in the context of reasoning.
- 73
- Multi-Scale Self-Attention for Text Classification72
A Multi-Scale Transformer which uses multi-scale multi-head self-attention to capture features from different scales and a strategy to control the scale distribution for each layer is designed.
- VideoRoPE: What Makes for Good Video Rotary Position Embedding?71
VideoRoPE is introduced, with a 3D structure designed to preserve spatio-temporal relationships, which consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination.
- Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction70
Sparse-dLLM is proposed, the first training-free framework integrating dynamic cache eviction with sparse attention via delayed bidirectional sparse caching, and achieves up to 10 times higher throughput than vanilla dLLMs, with comparable performance and similar peak memory costs, outperforming previous methods in efficiency and effectiveness.
- Reinforced Mnemonic Reader for Machine Comprehension69
The Reinforced Mnemonic Reader for machine comprehension (MC) task, which aims to answer a query about a given context document, is introduced and several novel mechanisms that address critical problems in MC that are not adequately solved by previous works are proposed.
- Deep Fusion LSTMs for Text Semantic Matching69
This paper proposes a model of deep fusion LSTMs (DF-LSTMs) to model the strong interaction of text pair in a recursive matching way, and uses exter-nal memory to increase the capacity of LSTm, thereby possibly capturing more complicated matching patterns.
- MoT: Memory-of-Thought Enables ChatGPT to Self-Improve67
Experimental results show that MoT can help ChatGPT significantly improve its abilities in arithmetic reasoning, commonsense reasoning, factual reasoning, and natural language inference and can lead to consistent improvements across various CoT methods and LLMs.
- Efficient training of large language models on distributed infrastructures: a survey65
This survey explores recent advancements in training systems for LLMs, including innovations in training infrastructure with AI accelerators, networking, storage, and scheduling, as well as optimizations for computation, communication, and memory in distributed LLM training.
- 64
- SIM-CoT: Supervised Implicit Chain-of-Thought61
SIM-CoT significantly improves both in-domain accuracy and out-of-domain stability of implicit CoT methods, boosting Coconut by +8.2\% on GPT-2 and CODI by +3.0\% on LLaMA-3.1 8B.
- Towards Efficient NLP: A Standard Evaluation and A Strong Baseline61
The proposed ELUE has a strong Pareto Frontier and makes a better evaluation for efficient NLP models, and a strong baseline, ElasticBERT, which allows BERT to exit at any layer in both static and dynamic ways is released.
- Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective60
The roadmap to achieving o1 is analyzed from the perspective of reinforcement learning, focusing on four key components: policy initialization, reward design, search, and learning, which underscore how learning and search drive o1's advancement.
- 60
- Origin Tracing and Detecting of LLMs60
This paper introduces a novel algorithm that leverages the contrastive features between LLMs and extracts model-wise features to trace the text origins and proposes an effective method to trace and detect AI-generated contexts.
- Investigating Language Universal and Specific Properties in Word Embeddings60
Three representative models are investigated on a large set of language samples by mapping dense embedding to sparse linguistic property space to reveal the language universal and specific properties encoded in various word representation.
- Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem59
It is shown that utilizing MWP is a reliable and effective approach to assess hallucination and in-context learning and reinforcement learning with human feedback (RLHF) training significantly enhance the model’s ability to avoid hallucination.
- Token-Aware Virtual Adversarial Training in Natural Language Understanding58
This work proposes a Token-Aware Virtual Adversarial Training method, which introduces a token-level accumulated perturbation vocabulary to initialize the perturbations better and uses atoken-level normalization ball to constrain these perturbATIONS pertinently.
- AgentGym: Evaluating and Training Large Language Model-based Agents across Diverse Environments57
A GENT G YM is proposed, a framework featuring 7 real-world scenarios, 14 environments, and 89 tasks for unified, real-time, and concurrent agent interaction that takes the initial step to develop LLM-based agents that can handle diverse tasks via methods like self-improvement or reinforcement learning.
- Deep Multi-Task Learning with Shared Memory for Text Classification57
Experiments on two groups of text classification tasks show that the proposed architectures can improve the performance of a task with the help of other related tasks.
- Can AI Assistants Know What They Don't Know?55
A model-specific"I don't know"(Idk) dataset is constructed for an assistant, which contains its known and unknown questions, based on existing open-domain question answering datasets, and whether it can refuse to answer its unknown questions after alignment.
- MOSS: An Open Conversational Large Language Model54
The development of MOSS is reported, an open-sourced conversational LLM that contains 16 B parameters and can perform a variety of instructions in multi-turn interactions with humans and establishes a complete technical roadmap for large language models from pre-training, supervised fine-tuning to alignment.
- Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision54
This paper delves into a two-player paradigm that separates the roles of reasoning and critique models, where the critique model provides step-level feedback to supervise the reasoning (actor) model during both test-time and train-time, and proposes a critique-in-the-loop self-improvement method.
- 54
- 54
- Enhancing Scientific Papers Summarization with Citation Graph53
This work redefines the task of scientific papers summarization by utilizing their citation graph and proposes a citation graph-based summarization model CGSum which can incorporate the information of both the source paper and its references.
- FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction52
This goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking.
- Flames: Benchmarking Value Alignment of LLMs in Chinese52
A value alignment benchmark named Flames, which encompasses both common harmlessness principles and a unique morality dimension that integrates specific Chinese values such as harmony is proposed, which has far exceeded existing benchmarks.
- 50
- 50
- SpellBERT: A Lightweight Pretrained Model for Chinese Spelling Check50
This work proposes SpellBERT, a pretrained model with graph-based extra features and independent on confusion set that can make a state-of-the-art result on the OCR dataset where most of the errors are not covered by the existing confusion set.
- GenWiki: A Dataset of 1.3 Million Content-Sharing Text and Graphs for Unsupervised Graph-to-Text Generation50
This paper proposes a large-scale, general-domain dataset, GenWiki, which has 1.3M text and graph examples, respectively, and provides this new benchmark dataset for future research on unsupervised text generation from knowledge graphs.
- 50
- Reasoning in Flux: Enhancing Large Language Models Reasoning through Uncertainty-aware Adaptive Guidance48
Uncertainty-aware Adaptive Guidance (UAG) is introduced, a novel approach for guiding LLM reasoning onto an accurate and reliable trajectory that not only enhances the reasoning abilities of LLMs but also consistently outperforms several strong baselines with minimal computational overhead.
- Evaluating Hallucinations in Chinese Large Language Models48
A benchmark named HalluQA (Chinese Hallucination Question-Answering) is established to measure the hallucination phenomenon in Chinese large language models and discusses which types of hallucinations should be prioritized for different types of models.
- Modelling Interaction of Sentence Pair with Coupled-LSTMs48
This paper introduces two coupled ways to model the interdependences of two LSTMs, coupling the local contextualized interactions of two sentences and uses a dynamic pooling to select the most informative features.
- A Re-ranking Model for Dependency Parser with Recursive Convolutional Neural Network48
This work proposes a recursive convolutional neural network (RCNN) architecture to capture syntactic and compositional-semantic representations of phrases and words in a dependency tree to improve the state-of-the-art dependency parsing on both English and Chinese datasets.
- Accelerating BERT Inference for Sequence Labeling via Early-Exit47
The token-level early-exit mechanism brings the gap between training and inference, so an extra self-sampling fine-tuning stage is introduced to alleviate it and can save up to 66% ∼75% inference cost with minimal performance degradation.
- A Closer Look at Data Bias in Neural Extractive Summarization Models47
This paper proposes several properties of datasets, which matter for the generalization of summarization models, and builds the connection between priors residing in datasets and model designs, analyzing how different factors of datasets influence the choices of model structure design and training methods.
- 47
- 46
- 44
- 44
- SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation43
SpeechGPT-Gen is developed, an 8-billion-parameter SLLM efficient in semantic and perceptual information modeling that markedly excels in zero-shot text-to-speech, zero-shot voice conversion, and speech-to-speech dialogue, underscoring CoIG's remarkable proficiency in capturing and modeling speech's semantic and perceptual dimensions.
- 43
- From Hypergraph Energy Functions to Hypergraph Neural Networks42
This paper begins by presenting an expressive family of parameterized, hypergraph-regularized energy functions, and demonstrates how minimizers of these energies effectively serve as node embeddings that, when paired with a parameterized classifier, can be trained end-to-end via a supervised bilevel optimization process.
- Mnemonic Reader for Machine Comprehension.41
The Mnemonic Reader is introduced for machine comprehension tasks, which uses a self-alignment attention to model the long-distance dependency among context words, and obtains query-aware and selfaware contextual representation for each word in the context.
- Analyzing Linguistic Knowledge in Sequential Model of Sentence41
This work proposes to correlate the neuron activation pattern of a LSTM language model with rich language features at sequential, lexical and compositional level and analyzes the influence of modelling strategy on linguistic knowledge encoded implicitly in different sequential models.
- Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder40
The Contrastive EEG-Text Masked Autoencoder is proposed, a novel model that orchestrates compound self-supervised learning across and within EEG and text through a dedicated multi-stream encoder and a framework called E2T-PTR (EEG-to-Text decoding using Pretrained Transferable Representations), which leverages pre-trained modules alongside the EEG stream from CET-MAE and further enables an LLM to decode text from EEG sequences.
- A Graph-based Model for Joint Chinese Word Segmentation and Dependency Parsing38
The proposed graph-based joint model achieves better performance than previous joint models and state-of-the-art results in both Chinese word segmentation and dependency parsing and when BERT is combined, can substantially reduce the performance gap of dependency parsing between joint Models and gold-segmented word-based models.
- 38
- Switch-LSTMs for Multi-Criteria Chinese Word Segmentation38
This paper presents Switch-LSTMs to segment words, which consist of several long short-term memory neural networks (LSTM), and a switcher to automatically switch the routing among these LSTMs, which provides a more flexible solution for multi-criteria CWS, which is also easy to transfer the learned knowledge to new criteria.
- CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs37
This work introduces a visual preference optimization module within the DPO framework, enabling MLLMs to learn from both textual and visual preferences simultaneously, and proposes a hierarchical textual preference optimization module that allows the model to capture preferences at multiple granular levels, including response, segment, and token levels.
- CoNT: Contrastive Neural Text Generation37
Experimental results show that CoNT clearly outperforms the conventional training framework on all the ten benchmarks with a convincing margin, and achieves new state-of-the-art on summarization, code comment generation (without external data) and data-to-text generation.
- 37
- Sentence Modeling with Gated Recursive Neural Network37
A gated recursive neural network (GRNN) to model sentences, which employs a full binary tree (FBT) structure to control the combinations in recursive structure by introducing two kinds of gates.
- Finding Sparse Structures for Domain Specific Neural Machine Translation35
Prune-Tune is a novel domain adaptation method via gradual pruning that alleviates the over-fitting and the degradation problem without model modification and is able to sequentially learn a single network with multiple disjoint domain-specific sub-networks for multiple domains.
- 35
- Contrastive Aligned Joint Learning for Multilingual Summarization34
A large-scale multilingual summarization corpus MLGSum consisting of 1.1 million articles and summaries in 12 different languages is provided and a contrastive learning strategy is used to train the mult bilingual summarization system (CALMS), which achieves improvement over mono-lingual models in all languages.
- A Concise Model for Multi-Criteria Chinese Word Segmentation with Transformer Encoder34
A concise but effective unified model for MCCWS, which is fully-shared for all the criteria and can segment both simplified and traditional Chinese and has an excellent transfer capability is proposed.
- 34
- Incorporating Discriminator in Sentence Generation: a Gibbs Sampling Method34
This paper proposes a novel framework to generate constrained sentences via Gibbs Sampling using recurrent neural networks and their variants to predict following words given previous sequence and target label.
- Application of tongue image characteristics and oral-gut microbiota in predicting pre-diabetes and type 2 diabetes with machine learning33
The integration of tongue diagnosis with microbiome analysis reveals distinct tongue features and microbial markers that significantly improves the diagnostic capability for Pre-DM and T2DM.
- 33
- Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures32
This work investigates two mainstream architectures for language modeling, namely Transformers and Mambas, to explore the extent of their mechanistic similarity and proposes to use Sparse Autoencoders to isolate interpretable features from these models and shows that most features are similar in these two models.
- Syntax-guided text generation via graph neural network32
This paper proposes the word graph model, a graph generation problem exploiting both syntactic and word-ordering relationships that builds a sentence incrementally and maintains syntactic integrity via a syntax-driven, top-down, breadth-first generation process.
- 32
- Dynamic Compositional Neural Networks over Tree Structure32
This paper introduces the dynamic compositional neural networks over tree structure (DC-TreeNN), in which the compositional function is dynamically generated by a meta network, which captures the metaknowledge across the different compositional rules and formulate them.
- Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT31
A circuit discovery framework alternative to activation patching, which digs in a small transformer trained on a synthetic task named Othello and finds a number of human-understandable fine-grained circuits inside of it.
- 31
- Joint Parsing and Generation for Abstractive Summarization31
A novel neural architecture for abstractive summarization that combines a sequential decoder with a tree-based decoder in a synchronized manner to generate a summary sentence and its syntactic parse and describes a novel human evaluation protocol to assess if, and to what extent, a summary remains true to its original meanings.
- Same Representation, Different Attentions: Shareable Sentence Representation Learning from Multiple Tasks31
It is claimed that a good sentence representation should be invariant and can benefit the various subsequent tasks, and a new scheme of information sharing for multi-task learning is proposed.
- A Feature-Enriched Neural Model for Joint Chinese Word Segmentation and Part-of-Speech Tagging31
A feature-enriched neural model for joint Chinese word segmentation and part-of-speech tagging task by using different filters to model the complex compositional features with convolutional and pooling layer, and then utilizing long distance dependency information with recurrent layer.
- Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization30
An adversarial meta-evaluation methodology that allows to diagnose the strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets and search for directions for further improvement by data augmentation is presented.
- 30
- Overview of the NLPCC-ICCPOL 2016 Shared Task: Chinese Word Segmentation for Micro-Blog Texts30
An overview for the shared task at the 5th CCF Conference on Natural Language Processing & Chinese Computing (NLPCC 2016): Chinese word segmentation for micro-blog texts and uses a new psychometric-inspired evaluation metric for ChineseWordSeg-Weibo.
- Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Models29
The SIUO is developed, a cross-modality benchmark encompassing 9 critical safety domains, such as self-harm, illegal activities, and privacy violations, that reveals substantial safety vulnerabilities in both closed- and open-source LVLMs, underscoring the inadequacy of current models to reliably interpret and respond to complex, real-world scenarios.
- COLO: A Contrastive Learning based Re-ranking Framework for One-Stage Summarization29
A Contrastive Learning based re-ranking framework for one-stage summarization called CoLo is proposed, which shows that the summarization model is able to directly generate summaries according to the summary-level score without additional modules and parameters.
- CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search29
The proposed CodeRetriever model, which learns the function-level code semantic representations through large-scale code-text contrastive pre-training, achieves new state-of-the-art with significant improvement over existing code pre-trained models, on eleven domain/language-specific code search tasks with six programming languages in different code granularity.
- BERT for Monolingual and Cross-Lingual Reverse Dictionary29
By using the Multilingual BERT (mBERT), this paper can efficiently conduct the cross-lingual reverse dictionary with one subword embedding, and the alignment between languages is not necessary.
- CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems29
An in-depth analysis of characteristics of different datasets is performed and the performance of different summarization models under a cross-dataset setting is investigated, in which a summarizer trained on one corpus will be evaluated on a range of out-of-domain corpora.
- End-to-End Neural Text Classification for Tibetan29
Three end-to-end neural models for Tibetan text classification are investigated and the experimental results show that the end- to-end models outperform the traditional TibetanText classification methods.
- MouSi: Poly-Visual-Expert Vision-Language Models28
This paper proposes the use of ensemble experts technique to synergizes the capabilities of individual visual encoders, including those skilled in image-text matching, OCR, image segmentation, etc, while bridging the gap between image encoders and pre-trained LLMs.
- Deep learning prediction of glycopeptide tandem mass spectra powers glycoproteomics27
DeepGP, a hybrid deep learning framework based on Transformer and graph neural network, is presented for the prediction of MS/MS spectra and retention time of glycopeptides and it is demonstrated that DeepGP in combination with database searching can significantly increase the detection sensitivity of glycopeptides.
- Efficient Link Prediction via GNN Layers Induced by Negative Sampling27
A novel GNN architecture whereby the forward pass explicitly depends on both positive and negative edges to inform more flexible, yet still cheap node-wise embeddings, which retains the inference speed of node-wise models, while producing competitive accuracy with edge-wise alternatives.
- Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning27
This paper presents Multi-task Pre-trained Modular Prompt (MP2) to boost prompt tuning for few-shot learning and demonstrates that MP2 can achieve surprisingly fast and strong adaptation to downstream tasks by merely learning 8 parameters to combine the pre-trained modular prompts.
- 27
- Hierarchical Text Classification with Latent Concepts27
This paper claims that each class has several latent concepts and its subclasses share information with these different concepts respectively, and proposes a variant Passive-Aggressive algorithm for hierarchical text classification with latent concepts.
- Automatically Identifying Local and Global Circuits with Linear Computation Graphs26
This work introduces its circuit discovery pipeline with Sparse Autoencoders (SAEs) and a variant called Transcoders and a variant called Transcoders, and analyzes three kinds of circuits in GPT-2 Small, revealing new findings underlying existing discoveries.
- 26
- 26
- AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models25
This approach features program-based verification and controllable difficulty levels, enabling more reliable evaluation that better distinguishes models' reasoning abilities, and develops a bilingual benchmark, AutoLogi, to obtain more accurate assessments of models' reasoning capabilities.
- A Probabilistic Framework for Discovering New Intents25
This paper proposes a probabilistic framework for discovering intents where intent assignments are treated as latent variables and adopts the Expectation Maximization framework for optimization, which alleviates the forgetting of prior knowledge transferred from known intents by optimizing the discrimination of labeled data.
- Text Adversarial Purification as Defense against Adversarial Attacks25
This work introduces a novel adversarial purification method that focuses on defending against textual adversarial attacks and test the method on several strong adversarial attack methods including Textfooler and BERT-Attack and experimental results indicate that the purification algorithm can successfully defend against strong word-substitution attacks.
- $$\cal{Y}$$-Tuning: an efficient tuning paradigm for large-scale pre-trained models via label representation learning23
This paper proposes Y -Tuning, an efficient yet effective paradigm to adapt frozen large-scale PTMs to specific downstream tasks and shows that for DeBERTa_XXL with 1.6 billion parameters, Y -Tuning achieves performance more than 96% of full fine-tuning on GLUE Benchmark with only 2% tunable parameters and much fewer training costs.
- Keyphrase Generation with Fine-Grained Evaluation-Guided Reinforcement Learning23
A new fine-grained evaluation metric is proposed to improve the RL framework, which considers different granularities: token-level $F_1$ score, edit distance, duplication, and prediction quantities, and can effectively ease the synonym problem and generate a higher quality prediction.
- IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities22
IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities, aims to facilitate the transfer of textual capabilities of pre-trained LLMs to the speech modality by mitigating the modality gap between text and speech.
- Dynamic and Generalizable Process Reward Modeling21
Dynamic and Generalizable Process Reward Modeling (DG-PRM), which features a reward tree to capture and store fine-grained, multi-dimensional reward criteria, and pioneeringly adopt Pareto dominance estimation to identify discriminative positive and negative pairs.
- DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels21
This research proposes and evaluates the mainstream LLMs, including GPT-4, Claude, and LLaMA, revealing persistent long-context reasoning challenges and demonstrating their evidence-retrieval challenges.
- 21
- UTC-IE: A Unified Token-pair Classification Architecture for Information Extraction21
This work proposes a Unified Token-pair Classification architecture for Information Extraction (UTC-IE), where Plusformer is introduced on top of the token-pair feature matrix, which models axis-aware interaction with plus-shaped self-attention and local interaction with Convolutional Neural Network over token pairs.
- Pre-training with Meta Learning for Chinese Word Segmentation21
A CWS-specific pre-trained model MetaSeg is proposed, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task and can achieve new state-of-the-art performance on twelve widely-used CWS datasets and significantly improve model performance in low-resource settings.
- VCWE: Visual Character-Enhanced Word Embeddings21
A model to learn Chinese word embeddings via three-level composition using a convolutional neural network to extract the intra-character compositionality from the visual shape of a character; a recurrent neural network with self-attention to compose character representation into word embedDings; and the Skip-Gram framework to capture non-compositionality directly from the contextual information.
- Joint Chinese Word Segmentation and POS Tagging on Heterogeneous Annotated Corpora with Multiple Task Learning21
This paper proposes a unified model for Chinese S&T with heterogeneous annotation corpora that can boost the performances of both of the heterogeneous corpora by using the shared information, and achieves significant improvements over the state-of-the-art methods.
- 20
- 20
- 20
- 20
- Learning Multi-Task Communication with Message Passing for Sequence Learning20
This work adopts the idea from message-passing graph neural networks, and proposes a general graph multi-task learning framework in which different tasks can communicate with each other in an effective and interpretable way.
- Hierarchical multi-class text categorization with global margin maximization20
This paper proposes a hierarchical multi-class text categorization method with global margin maximization that not only maximize the margins among leaf categories, but also maximize the margin among their ancestors.
- fastHan: A BERT-based Multi-Task Toolkit for Chinese NLP19
The joint-model is trained and evaluated on 13 corpora of four tasks, yielding near state-of-the-art (SOTA) performance in dependency parsing and NER, achieving SOTA performance in CWS and POS.
- Idiom-Aware Compositional Distributed Semantics19
This paper proposes an idiom-aware distributed semantic model, grounded in the literal-first psycholinguistic hypothesis, which can adaptively learn semantic compositionality of a phrase literally or idiomatically.
- 19
- NEAREST NEIGHBOR DISCRIMINANT ANALYSIS19
A novel nonparametric linear feature extraction method, nearest neighbor discriminant analysis (NNDA), is proposed from the view of the nearest neighbor classification and it is demonstrated that NNDA outperforms the existing variant LDA methods.
- 18
- 18
- 17
- 17
- TextAT: Adversarial Training for Natural Language Understanding with Token-Level Perturbation17
This work introduces a novel training strategy: Text Adversarial Training with token-level perturbation, and proves the effectiveness of such normalization method using large-scale Transformer-based language models.
- Overview of the NLPCC 2017 Shared Task: Chinese News Headline Categorization17
An overview for the shared task at the CCF Conference on Natural Language Processing & Chinese Computing (NLPCC 2017): Chinese News Headline Categorization is given.
- Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded Knowledge16
Hint-before-Solving Prompting (HSP), which guides the model to generate hints for solving the problem and then generate solutions containing intermediate reasoning steps to effectively improve the accuracy of reasoning tasks, is introduced.
- Dual-axial self-attention network for text classification16
The experimental results show that the proposed text classification model can obtain state-of-the-art results and the proposed self-attention outperforms conventional self-Attention models.
- Chinese Word Segmentation via BiLSTM+Semi-CRF with Relay Node16
This paper proposes a simple yet effective framework to help Semi-CRF achieve comparable performance with CRF-based models under similar computation complexity, and proposes a new model named Semi-SRF-Relay, to model arbitrarily long segments within linear time complexity.
- Stepwise nearest neighbor discriminant analysis16
A new nonparametric feature extraction method, stepwise nearest neighbor discriminant analysis (SNNDA), is proposed from the point of view of the nearest neighbor classification, and it is demonstrated that SNNDA outperforms the existing variant LDA methods and the other state-of-art face recognition approaches on three datasets from ATT and FERET face databases.
- R3-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL15
A novel approach is introduced, $R^3$-NL2GQL, integrating both small and large Foundation Models for ranking, rewriting, and refining tasks, which leverages the interpretative strengths of smaller models for initial ranking and rewriting stages, while capitalizing on the superior query generation prowess of larger models for the final transformation of natural language queries into GQL formats.
- Convolutional Interaction Network for Natural Language Inference15
The Convolutional Interaction Network (CIN), a general model to capture the interaction between two sentences, which can be an alternative to the attention mechanism for NLI.
- 15
- Overview of the NLPCC 2015 Shared Task: Chinese Word Segmentation and POS Tagging for Micro-blog Texts15
The dataset and task are introduced, the different approaches of the participating systems are characterized, the test results are reported, and an overview analysis of these results are provided.
- 14
- 14
- 14
- 14
- Convolutional Deep Neural Networks for Document-Based Question Answering14
A convolutional neural network based architecture is presented to learn feature representations of each question-answer pair and compute its match score by taking the interaction and attention between question and answer into consideration.
- Text Classification with Document Embeddings14
This paper proposes a method to learn document embedding with neural network architecture for text classification task, where each document can be represented as a fine-grained representation of different meanings so that the classification can be done more accurately.
- Latent Semantic Tensor Indexing for Community-based Question Answering14
This paper proposes a unified question retrieval model based on latent semantic indexing with tensor analysis, which can capture word associations among different parts of CQA triples simultaneously and can reduce lexical chasm of question retrieval.
- Clinical study of intelligent tongue diagnosis and oral microbiome for classifying TCM syndromes in MASLD13
It is suggested that tongue characteristics are related to microbial metabolism, and different MASLD syndromes possess distinct biomarkers, supporting syndrome classification.
- 13
- Two Birds One Stone: Dynamic Ensemble for OOD Intent Classification13
This paper proposes a two-birds-one-stone method, which allows the model to decide whether to make a decision on OOD classification early during inference and can ensure accuracy and accelerate inference and also proposes a training method based on ensemble methods.
- 13
- Multi-Criteria Chinese Word Segmentation with Transformer13
A concise and effective model for multi-criteria CWS is proposed, which utilizes a shared fully-connected self-attention model to segment the sentence according to a criterion indicator.
- 13
- A New Psychometric-inspired Evaluation Metric for Chinese Word Segmentation13
The performance on a real evaluation shows that the proposed psychometric-inspired evaluation metric for Chinese word segmentation gives more reasonable and distinguishable scores and correlates well with human judgement.
- Thus Spake Long-Context Large Language Model12
This survey illustrates how LLM struggles between the tremendous need for a longer context and its equal need to accept the fact that it is ultimately finite, and gives a global picture of the lifecycle of long-context LLMs from four perspectives: architecture, infrastructure, training, and evaluation.
- 12
- Does Correction Remain A Problem For Large Language Models?12
Light is shed on the significance of correction in the era of large language models and its implications for various NLP applications by conducting two experiments on correction as a standalone and a preparatory task for other NLP tasks.
- 12
- 12
- 12
- 12
- 12
- Unleashing Embodied Task Planning Ability in LLMs via Reinforcement Learning11
Embodied Planner-R1 is introduced, a novel outcome-driven reinforcement learning framework that enables LLMs to develop interactive capabilities through autonomous exploration with minimal supervision, and incorporates three key innovations: Without human annotations, the framework employs pure reinforcement learning with group rollout, incorporating in-environment interaction through parallel exploration.
- Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition11
Low-Rank Sparse Attention (Lorsa) is designed to address the challenge of attention superposition to understand attention-mediated interaction between features in different token positions, and achieves parity with SAE in interpretability while Lorsa exhibits superior circuit discovery properties.
- 11
- Case2Code: Scalable Synthetic Data for Code Generation11
Experimental results show that such induction training benefits not only in distribution Case2Code performance but also enhances various coding abilities of trained LLMs, demonstrating the great potential of learning inductive reasoning.
- Dual Cache for Long Document Neural Coreference Resolution11
A new hybrid cache is proposed that integrates two eviction policies to capture global and local entities separately, and effectively reduces the aggregated cache misses up to half as before, while improving F1 score of coreference by 0.7pt.
- 11
- Two-dimensional nearest neighbor discriminant analysis11
A novel feature extraction method, two-dimensional nearest neighbor discriminant analysis (2DNNDA), is proposed from the view of the nearest neighbor classification, which makes use of the matrix representation of images.
- 11
- ReAttention: Training-Free Infinite Context with Finite Attention Scope10
This work proposes LongCache, a training-free approach that enables LLM to support an infinite context with finite context scope, through full-context cache selection and training-free integration, which effectively frees LLMs from the length extrapolation issue.
- 10
- 10
- Adaptive Semantic Compositionality for Sentence Modelling10
A parameterized compositional switch is introduced, which outputs a scalar to adaptively determine whether the meaning of a phrase should be composed of its two constituents.
- Transition-based Dependency Parsing Using Two Heterogeneous Gated Recursive Neural Networks10
Two heterogeneous gated recursion neural networks are proposed: tree structured gated recursive neural network (Tree-GRNN) and directed acyclic graph structuredgated recursive Neural Network (DAG- GRNN), which integrate to automatically learn the compositions of the dense features for transition-based dependency parsing.
- Detecting Hedge Cues and their Scopes with Average Perceptron10
A hedge detection method with average perceptron, which was used in the closed challenge in CoNLL-2010 Shared Task, and which achieved F-measure with 77.86% in detecting in-domain uncertain sentences, 77.44% in recognizing hedge cues, and 19.27% in identifying the scopes.
- 9
- Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance9
Reinforcement Learning Guidance is introduced, an inference-time method that adapts Classifier-Free Guidance by combining the outputs of the base and RL fine-tuned models via a geometric average, thereby offering unprecedented flexibility in controlling generative alignment.
- UnitCoder: Scalable Iterative Code Synthesis with Unit Test Guidance9
This work presents a scalable approach that leverages model-generated unit tests to guide the synthesis of high-quality code data from pre-training corpora, demonstrating the potential for producing diverse and high-quality post-training data at scale.
- Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models9
A hierarchical reasoning aggregation framework AoR (Aggregation of Reasoning), which selects answers based on the evaluation of reasoning chains and incorporates dynamic sampling, adjusting the number of reasoning chains in accordance with the complexity of the task.
- 9
- 9
- 9
- Automatic Corpus Expansion for Chinese Word Segmentation by Exploiting the Redundancy of Web Information9
This paper proposes a method to automatically expand the training corpus for the out-of-domain texts by exploiting the redundant information on Web and provides a new viewpoint to enhance the performance of CWS by automatically expanding corpus rather than developing complicated algorithms or features.
- 9
- DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting8
DuoDecoding is proposed, a novel approach that strategically deploys the draft and target models on the CPU and GPU respectively, enabling parallel decoding while preserving draft quality, and incorporates a hardware-aware optimal draft budget to minimize idle times and employs dynamic multi-sequence drafting to enhance draft quality.
- Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training8
A novel approach that decomposes any dataset into a linear combination of several meta-domains, a new concept designed to capture the key underlying features of datasets, that helps find the data mixture that enhances downstream task performance with minimal computational overhead is introduced.
- Evolution of Concepts in Language Model Pre-Training8
This work tracks linear interpretable feature evolution across pre-training snapshots using a sparse dictionary learning method called crosscoders and finds that most features begin to form around a specific point, while more complex patterns emerge in later training stages.
- Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learners8
The BoostPFN method can outperform standard PFNs with the same size of training samples in large datasets and achieve a significant acceleration in training times compared to other established baselines in the field, including widely-used Gradient Boosting Decision Trees (GBDTs), deep learning methods and AutoML systems.
- 8
- 8
- 8
- 8
- 8
- A Unified Model for Joint Chinese Word Segmentation and Dependency Parsing.8
Different from previous joint models, the proposed model is a graph-based model and more concise, which results in fewer efforts of feature engineering and achieves the state-of-the-art results in both Chinese word segmentation and dependency parsing.
- 8
- 8
- Boosting image classification scheme8
A boosting image classification scheme with automatic selection of discriminative features automatically is proposed by using an image feature called the orientational color correlogram and the experimental results suggest the proposed method has preferable performances.
- MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval7
MARAG-R1 is proposed, a reinforcement-learned multi-tool RAG framework that enables LLMs to dynamically coordinate multiple retrieval mechanisms for broader and more precise information access.
- INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning7
Experimental results show that the proposed Inst-IT models not only achieve outstanding performance on Inst-IT Bench and other instance understanding benchmarks, but also demonstrate significant improvements across various generic image and video understanding benchmarks, highlighting that the method not only boosts instance-level understanding but also strengthens the overall capabilities of generic image and video comprehension.
- 7
- 7
- An Effective Feature Selection Method for Text Categorization7
This paper proposes an effective feature selection method, which aims at the classification accuracy of KNN, and shows that it is better than the traditional methods, and it is also beneficial to other classifiers, such as Support Vector Machines (SVM).
- 7
- CodecBench: A Comprehensive Benchmark for Acoustic and Semantic Evaluation6
CodecBench is introduced, a comprehensive evaluation dataset to assess audio codec performance from both acoustic and semantic perspectives across four data domains, and aims to identify current limitations, highlight future research directions, and foster advances in the development of audio codec.
- 6
- 6
- 6
- 6
- 6
- Memorize Step by Step: Efficient Long-Context Prefilling with Incremental Memory and Decremental Chunk6
This work introduces Incremental Memory (IM), a method that starts with a small memory size and gradually increases it, optimizing computational efficiency, and proposes Decremental Chunk based on Incremental Memory (IMDC), which reduces chunk size while increasing memory size, ensuring stable and lower GPU memory usage.
- 6
- 6
- 6
- 6
- Mnemonic Reader: Machine Comprehension with Iterative Aligning and Multi-hop Answer Pointing6
Mnemonic Reader for MC tasks is introduced, an end-to-end neural network which aims to tackle the above problem in two aspects: an iterative aligning mechanism which not only captures interactions between the context and the query but also models interactions among the context itself to obtain a fully-aware context representation.
- Joint Segmentation and Tagging with Coupled Sequences Labeling6
A novel joint model for segmentation and tagging, which integrates two dependent Markov chains that could achieve higher performance over traditional models on both English shallow parsing and Chinese word segmentsation and POS tagging tasks.
- Question Answering for Machine Reading with Lexical Chain.6
The system follows the steps of reading comprehension as a language learner and achieves the c@1 measure of 0.28 and 0.26 for the two submissions, respectively on the QA4MRE test dataset.
- 5
- CritiQ: Mining Data Quality Criteria from Human Preferences5
This work introduces CritiQ, a novel data selection method that automatically mines criteria from human preferences for data quality with only ~30 human-annotated pairs and performs efficient data selection, and analyzes how criteria evolve and the effectiveness of majority voting.
- Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache5
FourierAttention is proposed, a training-free framework that exploits the heterogeneous roles of transformer head dimensions: lower dimensions prioritize local context, while upper ones capture long-range dependencies by projecting the long-context-insensitive dimensions onto orthogonal Fourier bases.
- LLMs and Their Applications in Medical Artificial Intelligence5
This work presents a research framework for LLMs in medical AI, and describes the LLM multiplex—important multimodal, multi-model, multicultural, and multi-responsibility considerations for LLMs in medical AI.
- MOSS-MED: A Family of Multimodal Models Serving Medical Image Analysis5
This study introduces MOSS-MED, a suite of models designed for biomedical vision-language understandings, and evaluates the performance of MOSS-MED family on medical visual question answering (VQA) benchmarks.
- Understanding the Role of LLMs in Multimodal Evaluation Benchmarks5
A modified evaluation protocol to disentangle the contributions of the LLM backbone from multimodal integration, and an automatic knowledge identification technique for diagnosing whether LLMs equip the necessary knowledge for corresponding multimodal questions are introduced.
- 5
- SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems5
SpeechAgents is proposed, a multi-modal LLM based multi-agent system designed for simulating human communication that can simulate human communication dialogues with consistent content, authentic rhythm, and rich emotions and demonstrate excellent scalability even with up to 25 agents.
- 5
- 5
- 5
- SDCL: Self-Distillation Contrastive Learning for Chinese Spell Checking5
This work proposes a token-level self-distillation contrastive learning method that employs BERT to encode both the corrupted and corresponding correct sentence, and uses Contrastive learning loss to regularize corrupted tokens’ hidden states to be closer to counterparts in the correct sentence.
- 5
- Part-of-Speech Tagging for Chinese-English Mixed Texts with Dynamic Features5
This paper presents a method using dynamic features to tag part-of-speech (POS) for the English words involved in Chinese-English mixed texts and achieves higher performance than traditional sequence labeling methods.
- Info-margin maximization for feature extraction5
A novel method of linear feature extraction with info-margin maximization (InfoMargin) that aims to achieve a low generalization error by maximizing the information divergence between the distributions of different classes while minimizing the entropy of the distribution in each single class is proposed.
- Learning Semantic Lexicons Using Graph Mutual Reinforcement Based Bootstrapping5
Experimental results show that the GMR-based bootstrapping approach outperforms the existing algorithms both in in- domain data and out-domain data and that the result depends on not only the size of the corpus but also the quality.
- 5
- FiNE: Filtering and Improving Noisy Data Elaborately with Large Language Models4
FiNE is proposed, a method that performs refined filtering and improvement of training data with LLMs that surpasses the open-source chat version of Yi-1.5-9B on HalluQA and the performance gap on AlignBench between the model and the open-source chat version is reduced.
- 4
- Task-Core Memory Management and Consolidation for Long-term Continual Learning4
A novel framework inspired by human memory mechanisms for long-term continual learning (Long-CL) is proposed, which introduces a task-core memory management strategy to efficiently index crucial memories and adaptively update them as learning progresses and develops a long-term memory consolidation mechanism that selectively retains hard and discriminative samples, ensuring robust knowledge retention.
- 4
- FamilyTool: A Multi-hop Personalized Tool Use Benchmark4
FamilyTool is introduced, a novel benchmark grounded in a family-based knowledge graph (KG) that simulates personalized, multi-hop tool use scenarios and proposes KGETool: a simple KG-augmented evaluation pipeline to systematically assess LLM agents' tool use ability in these settings.
- 4
- 4
- 4
- Data-free Weight Compress and Denoise for Large Language Models4
This paper proposes a novel approach termed Data-free Joint Rank-k Approximation for compressing the parameter matrices of large Language Models without necessitating additional involvement of any corpus, while simultaneously preserving orthogonality in conjunction with pruning and quantization methods.
- 4
- What Dense Graph Do You Need for Self-Attention?4
Hypercube Transformer is presented, a sparse Transformer that models token interactions in a hypercube and shows comparable or even better results with vanilla Transformer while yielding O(N\log N)$ complexity with sequence length $N$.
- 4
- Hyper-Gated Recurrent Neural Networks for Chinese Word Segmentation4
This paper proposes the hyper-gated recurrent neural networks for Chinese word segmentation, which enhance the gates to incorporate the historical information of gates and shows that the model outperforms the baseline models as well as the state-of-the-art methods.
- A Long Dependency Aware Deep Architecture for Joint Chinese Word Segmentation and POS Tagging.4
This work proposes a long dependency aware deep architecture for joint S&T task to simulate the feature templates of traditional discrete feature based models, and uses different filters to model the complex compositional features with convolutional and pooling layer, and utilizes long distance dependency information with recurrent layer.
- Discovering logical knowledge for deep question answering4
This paper proposes an approach to discover logical knowledge for deep question answering, which automatically extracts knowledge in an unsupervised, domain-independent manner from background texts and reasons out implicit answers for the questions.
- 3
- 3
- 3
- 3
- Making Large Language Models Better Reasoners with Orchestrated Streaming Experiences3
The architecture that extends an LLM to store all answered reasoning questions and their reasoning steps in a streaming experience pool and orchestrate helpful questions from the pool to assist itself in answering new questions is described.
- 3
- 3
- 3
- 3
- 3
- Text information aggregation with centrality attention3
This work proposes a new way of obtaining aggregation weights, called eigen-centrality self-attention, which builds a fully-connected graph for all the words in a sentence, then compute the eigen -centrality as the attention score of each word.
- fastHan: A BERT-based Joint Many-Task Toolkit for Chinese NLP.3
This work presents fastHan, an open-source toolkit for four basic tasks in Chinese natural language processing: Chinese word segmentation, Part-of-Speech tagging, named entity recognition, and dependency parsing, which is a joint many-task model based on a pruned BERT.
- 3
- Can computers understand words like humans do? Comparable semantic representation in neural and computer systems3
This study provided a finer-grained understanding of the neural dynamics underlying semantic processing and developed an objective biomarker for assessing human-like computation in computational models.
- 3
- 3
- A Fast Accurate Two-stage Training Algorithm for L1-regularized CRFs with Heuristic Line Search Strategy3
Experimental evaluations on Chinese word segmentation and name entity recognition tasks demonstrate that the two-stage training algorithm can produce more accurate and compact model with less training time for L1 regularization.
- Color constancy model based on topology resolve-map in medical image processing3
A Topology Resolve-Map (TRM) model is proposed to restore the gathered colors from the gathered images to standard values, and outperform significantly the professional software.
- 3
- 2
- 2
- 2
- 2
- RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization2
RLoop transforms the standard training process into a virtuous cycle: it first uses RL to explore the solution space from a given policy, then filters the successful trajectories to create an expert dataset, creating a superior starting point for the next iteration.
- 2
- 2
- 2
- Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding2
A hierarchical framework with a coarse-to-fine paradigm, with the bottom level shared to all the tasks, the mid-level divided to different groups, and the top-level assigned to each of the tasks is proposed, which allows the model to learn basic language properties from all tasks, boost performance on relevant tasks, and reduce the negative impact from irrelevant tasks.
- Searching Effective Transformer for Seq2Seq Keyphrase Generation2
Comprehensive experiments on multiple KG benchmarks showed that in KG tasks, uninformative content abounds in documents while salient information is diluted globally, and the vanilla Transformer equipped with a fully-connected self-attention mechanism may overlook the local context, leading to performance degradation.
- 2
- A Simple yet Effective Joint Training Method for Cross-Lingual Universal Dependency Parsing2
This paper jointly train models when two languages are similar according to linguistic typology and then ensemble the models using a simple re-parse algorithm to outperform the baseline method on development (test) set in CoNLL 2018 UD Shared Task.
- 2
- Transition-Based Dependency Parsing with Long Distance Collocations2
This paper proposes a method to improve the performance of transition-based parsing with long distance collocations with an approximate global view of the entire sentence, which is a little bit similar to top-down parsing.
- Recognizing Inference in Texts with Markov Logic Networks2
This article designs specific semantic rules based on the surface, syntactic, and semantic representations of texts, and map these rules to logical representations, and utilizes Markov logic networks framework to make predictions with combining statistical and logical reasoning.
- Adaptive Chinese Word Segmentation with Online Passive-Aggressive Algorithm2
The online passive-aggressive algorithm with domain invariant information for cross-domain Chinese word segmentation is used for CIPS-SIGHAN-2010 bake-off task of Chineseword segmentation.
- Domain Adaptation for Conditional Random Fields2
A novel technique for maximum a posteriori (MAP) adaptation of Conditional Random Fields model and results show that this technique can significantly improve performance on out-of-domain data.
- 2
- 1
- 1
- 1
- Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric1
A novel metric, the Model Utilization Index (MUI), is proposed, which introduces mechanism interpretability techniques to complement traditional performance metrics and deduces a common trend observed in popular LLMs, which is termed the “Utility Law.
- 1
- Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM1
Decoupled Proxy Alignment introduces two key innovations: the use of a proxy LLM during pretraining to decouple the vision-language alignment process from language prior interference, and dynamic loss adjustment based on visual relevance to strengthen optimization signals for visually relevant tokens.
- 1
- Teach2Eval: An Indirect Evaluation Method for LLM by Judging How It Teaches1
Teach2Eval is introduced, an indirect evaluation framework inspired by the Feynman Technique that evaluates a model's multiple abilities to teach weaker student models to perform tasks effectively and captures a broad range of cognitive abilities that are orthogonal to current benchmarks.
- In-Memory Learning: A Declarative Learning Framework for Large Language Models1
A novel learning framework is proposed that adeptly distill insights from past experiences, refining and updating existing notes to enhance their performance in the environment and is implemented through natural language, so it is characterised as In-memory Learning.
- 1
- 1
- 1
- 1
- 1
- Optimizing Non-Autoregressive Transformers with Contrastive Learning1
This paper derives contrastive constraints to stabilize the training process and integrates this resulting objective with the state-of-the-art NAT architecture DA-Transformer, to ease the difficulty of modality learning via sampling from the model distribution instead of the data distribution.
- 1
- 1
- Word-Level Representation From Bytes For Language Modeling1
This work overhauls the existing character-aware method that takes character-level inputs but makes word-level sequence modeling and prediction by introducing a cross-attention network that builds word- level representation directly from bytes, and a sub-word level prediction based on word-levels hidden states to avoid the time and space requirement ofword-level prediction.
- Pre-trained Model for Chinese Word Segmentation with Meta Learning.1
This work proposes a Pre-Trained Model for Chinese Word Segmentation, which can be abbreviated as PTM-CWS, which employs a unified architecture for different segmentation criteria, and is pre-trained on a joint multi-criteria corpus with meta learning algorithm.
- Online Distributed Passive-Aggressive Algorithm for Structured Learning1
Experimental results show that the proposed online distributed Passive-Aggression by averaging parameters for parallel training can accelerate the training process significantly with comparable or even better accuracy.
- 1
- 1
- –
- –
- –
- –
- –
- MCM-DPO : Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation–
Experimental results show that the proposed Multi-faceted Cross-modal Direct Preference Optimization (MCM-DPO) method consistently outperforms both DPO and SFT, establishing a new state of the-art in alt-text generation.
- –
- –
- –
- –
- –
- –
- F-Eval: Assessing Fundamental Abilities with Refined Evaluation Methods–
This work proposes F-Eval, a bilingual evaluation benchmark to evaluate the fundamental abilities of LLMs’ fundamental abilities, including expression, common-sense and logic, and devise new evaluation methods, serving as alternatives to scoring by API models.
- –
- –
- –
- –
- –
- The Open-World Lottery Ticket Hypothesis for OOD Intent Classification–
It is demonstrated that calibrated subnetworks can be un-covered by pruning the (poor-calibrated) over-parameterized model and theoretically bring new insights into why temperature scaling can differentiate In-and Out-of-Domain intent.
- –
- High-density interpolation using in optimizing correlated color temperature calculation–
By comparing with the other classical methods, this work proves that high density interpolation is a programming method on calculating CCT with lesser error and better performance.
- Implicit discourse relation detection using concatenated word embeddings and a gated relevance network–
This work proposes to use a gated relevance network (GRN) that combines a bilinear model and single layer network with a gate to compute the arguments’ relevance scores from the word pairs, to handle the rare word problem.
- –
- –
- Learning to Rank Answers for Definitional Question Answering–
This paper proposes an online learning algorithm, which dynamically construct the supervisor to reduce the adverse effects of the large number of bad answers and noisy data and compares it with two state-of-the-art definitional QA systems and two ranking algorithms.
- Improving Multi-pass Transition-Based Dependency Parsing Using Enhanced Shift Actions–
This paper proposes a method to improve performance of parsing by using enhanced shift actions, which can be further used as features for the next parsing decision.
- Chinese Word Segmentation with Character Abstraction–
This paper focuses on the primary challenges in Chinese word segmentation: low accuracy of out-of-vocabulary word, and group the “similar” characters to generate more abstract representation to resolve this difficult problems.
- –
- –
- FudanNLP at RITE 2011: a Shallow Semantic Approach to Textual Entailment–
This paper presents FDCS’s approach on NTCIR9-RITE Chinese simplified BC & MC subtasks, built on a machine learning architecture with features selected on shallow semantic methods, including named entity recognition, date & time expression extraction, words overlap and negation concept recognition.
- Labelwise Margin Maximization for Sequence Labeling–
An online learning algorithm is proposed that addresses the problem of labelwise margin maximization for sequence labeling by decomposing the sequence margin to per-label margins and maximizing these per- label margins individually, which can result to minimize the Hamming loss of sequence.
- Triplet-Based Chinese Word Sense Induction–
This paper describes the implementation of the system at CLP 2010 of Chinese word sense induction, which extracts the triplets for the target word in each sentence, then uses the intersection of all related words of these triplets to construct feature vectors for the sentence.
- Mining Uncertain Sentences with Multiple Instance Learning–
A method for detecting uncertain sentences with multiple instance learning (MIL) is proposed based on the basic assumption, and two new constraints for estimating the weight vector are derived by defining a probability margin, which is used in an online learning algorithm known as Passive-Aggressive algorithm.
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-12. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.