Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileWorks135 from public data
- Deformable Convolutional Networks7,141
This work introduces two new modules to enhance the transformation modeling capability of CNNs, namely, deformable convolution and deformable RoI pooling, based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from the target tasks, without additional supervision.
- 3,149
- Deformable ConvNets V2: More Deformable, Better Results2,748
This work presents a reformulation of Deformable Convolutional Networks that improves its ability to focus on pertinent image regions, through increased modeling power and stronger training, and guides network training via a proposed feature mimicking scheme that helps the network to learn features that reflect the object focus and classification power of R-CNN features.
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models1,936
InternVL3 incorporates variable visual position encoding (V2PE) to support extended multimodal contexts, employs advanced post-training techniques such as supervised fine-tuning (SFT) and mixed preference optimization (MPO), and adopts test-time scaling strategies alongside an optimized training infrastructure.
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations1,860
A new pre-trainable generic representation for visual-linguistic tasks, called Visual-Linguistic BERT (VL-BERT), which adopts the simple yet powerful Transformer model as the backbone, and extends it to take both visual and linguistic embedded features as input.
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling1,813
This work explores the relationship between model scaling and performance, systematically exploring the performance trends in vision encoders, language models, dataset sizes, and test-time configurations and hopes this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems.
- 1,776
- Planning-oriented Autonomous Driving1,690
This work introduces Unified Autonomous Driving (UniAD), a comprehensive framework up-to-date that incorporates full-stack driving tasks in one network and is exquisitely devised to leverage advantages of each module, and provide complementary feature abstractions for agent interaction from a global perspective.
- Relation Networks for Object Detection1,464
An object relation module is proposed that processes a set of objects simultaneously through interaction between their appearance feature and geometry, thus allowing modeling of their relations, which gives rise to the first fully end-to-end object detector.
- 1,344
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites1,306
InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding, shows competitive performance, achieving state-of-the-art results in 8 of 18 multimodal benchmarks.
- Instance-Aware Semantic Segmentation via Multi-task Network Cascades1,279
This paper presents Multitask Network Cascades for instance-aware semantic segmentation, which consists of three networks, respectively differentiating instances, estimating masks, and categorizing objects, and develops an algorithm for the nontrivial end-to-end training of this causal, cascaded structure.
- InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions1,217
This work presents a new large-scale CNN-based foundation model, termed InternImage, which can obtain the gain from increasing parameters and training data like ViTs, and reduces the strict inductive bias of traditional CNNs and makes it possible to learn stronger and more robust patterns with large- scale parameters from massive datalike ViTs.
- ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation1,154
This paper proposes to use scribbles to annotate images, and develops an algorithm to train convolutional networks for semantic segmentation supervised by scribbles, which shows excellent results on the PASCALCONTEXT dataset thanks to extra inexpensive scribble annotations.
- BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation1,125
This paper proposes a method that achieves competitive accuracy but only requires easily obtained bounding box annotations, and yields state-of-the-art results on PASCAL VOC 2012 and PASCal-CONTEXT.
- Fully Convolutional Instance-Aware Semantic Segmentation1,071
The first fully convolutional end-to-end solution for instance-aware semantic segmentation task, which achieves state-of-the-art performance in both accuracy and efficiency, wins the COCO 2016 segmentation competition by a large margin.
- Vision Transformer Adapter for Dense Predictions938
The ViT-Adapter is proposed, which allows plain ViT to achieve comparable performance to vision-specific transformers and facilitate future research and is verified on multiple dense prediction tasks, including object detection, instance segmentation, and semantic segmentation.
- Flow-Guided Feature Aggregation for Video Object Detection727
This work presents flow-guided feature aggregation, an accurate and end-to-end learning framework for video object detection that improves the per-frame features by aggregation of nearby features along the motion paths, and thus improves the video recognition accuracy.
- 668
- Deep Feature Flow for Video Recognition668
Deep feature flow is presented, a fast and accurate framework for video recognition that runs the expensive convolutional sub-network only on sparse key frames and propagates their deep feature maps to other frames via a flow field and achieves significant speedup as flow computation is relatively fast.
- Exploring Cross-Image Pixel Contrast for Semantic Segmentation596
This work raises a pixel-wise metric learning paradigm for semantic segmentation, by explicitly exploring the structures of labeled pixels, which were rarely explored before, and hopes this work will encourage the community to rethink the current de facto training paradigm in semantic segmentsation.
- An Empirical Study of Spatial Attention Mechanisms in Deep Networks596
An empirical study that ablates various spatial attention elements within a generalized attention formulation, encompassing the dominant Transformer attention as well as the prevalent deformable convolution and dynamic convolution modules, yields significant findings about spatial attention in deep networks, some of which run counter to conventional understanding.
- BEVFormer: Learning Bird’s-Eye-View Representation From LiDAR-Camera via Spatiotemporal Transformers593
This work presents a new framework termed BEVFormer, which learns unified BEV representations from multi-modality data with spatiotemporal transformers to support multiple autonomous driving perception tasks, achieving outstanding results across these tasks.
- BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision490
A novel bird's-eye-view (BEV) detector with perspective supervision, which converges faster and bet-suits modern image backbones and achieves new SoTA results on the large-scale nuScenes dataset.
- EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought459
This work introduces EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi- modal understanding and execution capabilities, and significantly enhances the success rate of the embodied control task by extracting more effective features.
- Convolutional feature masking for joint object and stuff segmentation456
This paper proposes a joint method to handle objects and “stuff” (e.g., grass, sky, water) in the same framework and presents state-of-the-art results on benchmarks of PASCAL VOC and new PASCal-CONTEXT.
- Fast Convergence of DETR with Spatially Modulated Co-Attention414
This work proposes a simple yet effective scheme for improving the DETR framework, namely Spatially Modulated Co-Attention (SMCA) mechanism, which increases DETR’s convergence speed by replacing the original co-attention mechanism in the decoder while keeping other operations in DETR unchanged.
- Instance-Sensitive Fully Convolutional Networks413
This paper develops FCNs that are capable of proposing instance-level segment candidates that do not have any high-dimensional layer related to the mask resolution, but instead exploits image local coherence for estimating instances.
- 411
- Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory368
Ghost in the Minecraft (GITM), a novel framework that integrates Large Language Models (LLMs) with text-based knowledge and memory, aiming to create Generally Capable Agents (GCAs) in Minecraft, shows the potential of LLMs in developing capable agents for handling long-horizon, complex tasks and adapting to uncertainties in open-world environments.
- Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation359
This paper addresses the value of cross-image semantic relations for comprehensive object pattern mining and provides a unified framework that handles well different WSSS settings, i.e., learning WSSD with (1) precise image-level supervision only, (2) extra simple single-label data, and (3) extra noisy web data.
- 330
- DriveMLM: aligning multi-modal large language models with behavioral planning states for autonomous driving319
DriveMLM is introduced, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators and can plug-and-play in existing AD systems such as Autopilot and Apollo for close-loop driving.
- Towards High Performance Video Object Detection285
This work proposes a unified approach based on the principle of multi-frame end-to-end learning of features and cross-frame motion that steadily pushes forward the performance envelope (speed-accuracy tradeoff) towards high performance video object detection.
- Resolution Adaptive Networks for Efficient Inference256
This paper focuses on spatial redundancy of input samples and proposes a novel Resolution Adaptive Network (RANet), inspired by the intuition that low-resolution representations are sufficient for classifying “easy” inputs containing large objects with prototypical features, while only some “hard” samples need spatially detailed information.
- Delving Into the Devils of Bird’s-Eye-View Perception: A Review, Evaluation and Recipe252
A full suite of practical guidebook to improve the performance of BEV perception tasks, including camera, LiDAR and fusion inputs is introduced, and the future research directions in this area are pointed out.
- 211
- 191
- FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting180
This work proposes FuseFormer, a Transformer model designed for video inpainting via fine-grained feature fusion based on novel Soft Split and Soft Composition operations, which surpasses state-of-the-art methods in both quantitative and qualitative evaluations.
- 179
- FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation178
This work proposes Masked Cost Volume Autoencoding (MCVA) to enhance FlowFormer by pretraining the cost-volume encoder with a novel MAE scheme, and introduces a block-sharing masking strategy to prevent masked information leakage, as the cost maps of neighboring source pixels are highly correlated.
- Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures166
Vision-RWKV (VRWKV), a model adapted from the RWKV model used in the NLP field with necessary modifications for vision tasks, surpasses ViT's performance in image classification and has significantly faster speeds and lower memory usage processing high-resolution inputs.
- ConvMAE: Masked Convolution Meets Masked Autoencoders164
The ConvMAE framework demonstrates that multi-scale hybrid convolution-transformer can learn more discriminative representations via the mask auto-encoding scheme, and adopts the masked convolution to prevent information leakage in the convolution blocks.
- Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks163
Results show that the pre-trained model without any tuning can achieve reasonable performance even on novel tasks, and the performance can be improved to a level close to state-of-the-art methods by conducting prompt tuning on 1% of downstream task data.
- 143
- A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook142
This article introduces seminal foundation models proposed or adaptable for reasoning, highlighting the latest advancements in various reasoning tasks, methods, and benchmarks, and delves into the potential future directions behind the emergence of reasoning abilities within foundation models.
- Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance139
This work introduces Mini-InternVL, a series of MLLMs with parameters ranging from 1 billion to 4 billion, which achieves 90% of the performance with only 5% of the parameters, which makes these models more accessible and applicable in various real-world scenarios.
- Influence Selection for Active Learning136
The Influence Selection for Active Learning (ISAL) which selects the unlabeled samples that can provide the most positive influence on model performance is proposed which achieves state-of-the-art performance in different active learning settings for different tasks with different datasets.
- The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World133
The All-Seeing model (ASM), a unified framework for panoptic visual recognition and understanding, is developed with open-ended language prompts and locations, which allows it to generalize to various vision and language tasks with remarkable zero-shot performance.
- VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimation125
VideoFlow concurrently estimates bi-directional optical flows for multiple frames that are available in videos by sufficiently exploiting temporal cues by sufficiently exploiting temporal cues and presents extraordinary performance, ranking 1st on all public benchmarks.
- Siamese Image Modeling for Self-Supervised Vision Representation Learning121
SiameseIM can surpass both ID and MIM on a wide range of downstream tasks, including ImageNet finetuning and linear probing, COCO and LVIS detection, and ADE20k semantic segmentation.
- 121
- Hierarchical Human Parsing With Typed Part-Relation Reasoning115
This work seeks to simultaneously exploit the representational capacity of deep graph networks and the hierarchical human structures by assimilating generic message-passing networks with their edge-typed, convolutional counterparts and lays the foundation for more sophisticated and flexible human relation patterns of reasoning.
- VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition109
A visual-linguistic long-tailed recognition framework that can not only learn visual representation from images but also learn corresponding linguistic representation from noisy class-level text descriptions collected from the Internet, and is close to the prevailing performance training on the full ImageNet.
- Learning Region Features for Object Detection86
This work proposes a general viewpoint that unifies existing region feature extraction methods and a novel method that is end-to-end learnable, which removes most heuristic choices and outperforms its RoI pooling counterparts.
- MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer83
MM-Interleaved is an end-to-end generative model for interleaved image-text data that introduces a multi-scale and multi-image feature synchronizer module, allowing direct access to fine-grained image features in the previous context during the generation process.
- Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks75
This paper proposes Uni-Perceiver v2, which is the first generalist model capable of handling major large-scale vision and vision-language tasks with competitive performance and proposes an effective optimization technique named Task-Balanced Gradient Normalization to ensure stable multi-task learning with an unmixed sampling strategy, helpful for tasks requiring large batch-size training.
- Generative modeling of convolutional neural networks75
Experiments on the challenging ImageNet benchmark show that the proposedGenerative gradient pre-training consistently helps improve the performances of CNNs, and the proposed generative visualization method generates meaningful and varied samples of synthetic images from a large-scale deep CNN.
- Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Framework74
UniGrad is proposed, a simple but effective gradient form for self-supervised learning that does not require a memory bank or a predictor network, but can still achieve state-of-the-art performance and easily adopt other training strategies.
- A Survey of Reasoning with Foundation Models73
This paper introduces seminal foundation models proposed or adaptable for reasoning, highlighting the latest advancements in various reasoning tasks, methods, and benchmarks and discusses the relevance of multimodal learning, autonomous agents, and super alignment in the context of reasoning.
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models71
The Multimodal Multi-image Understanding (MMIU) benchmark is introduced, a comprehensive evaluation suite designed to assess LVLMs across a wide range of multi-image tasks, making it the most extensive benchmark of its kind.
- 70
- RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis66
A tree-structured multimodal code generation framework for generalized robotic behavior synthesis, termed RoboCodeX, which decomposes high-level human instructions into multiple object-centric manipulation units consisting of physical preferences such as affordance and safety constraints, and applies code generation to introduce generalization ability across various robotics platforms.
- OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text60
This paper introduces OmniCorpus, a 10 billion-scale image-text interleaved dataset that has 15 times larger scales while maintaining good data quality and is more flexible, easily degradable from an image-text interleaved format to pure text corpus and image-text pairs.
- Towards All-in-One Pre-Training via Maximizing Multi-Modal Mutual Information58
This paper proposes a general multimodal mutual information formula as a unified optimization target and demonstrates that all mainstream approaches are special cases of the authors' framework, named Maximizing Multi-modal Mutual Information Pre-Training (M3I Pre-training).
- Video Dehazing via a Multi-Range Temporal Alignment Network with Physical Prior58
A memory-based physical prior guidance module to encode the prior-related features into long-range memory and a multi-range scene radiance recovery module to capture space-time dependencies in multiple space- time ranges, which helps to effectively aggregate temporal information from adjacent frames.
- SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding49
To address challenges identified in existing encoder-free unified MLLMs, the token folding mechanism and the vision-expert-based progressive alignment pretraining strategy are introduced, which effectively support high-resolution image understanding while reducing training complexity.
- 46
- 43
- 39
- Docopilot: Improving Multimodal Models for Document-Level Understanding38
This work presents a high-quality document-level dataset, Doc-750K, designed to support in-depth understanding of multimodal documents, and develops a native multimodal model, Docopilot, which can accurately handle document-level dependencies without relying on RAG.
- 33
- PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models31
This work extends each image into a "static" video and introduces a unified token compression strategy called Progressive Visual Token Compression (PVC), where the tokens of each frame are progressively encoded and adaptively compressed to supplement the information not extracted from previous frames.
- 30
- HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving26
This paper proposes a framework to jointly generate the camera images and LiDAR point clouds and employs BEV-to-Camera and Camera-to-BEV transform modules between heterogeneous generative models, and introduces a depth prediction branch in the 2D generative model to disambiguate the un-projecting from image space to BEV space.
- 25
- 23
- 22
- ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Process22
This method represents the first successful development of general representations applicable to both generation and dense recognition tasks and can be used to generate diverse high-fidelity images and also demonstrate excellent transfer performance on recognition tasks.
- 19
- 17
- 16
- 15
- 14
- 14
- Demystify Transformers & Convolutions in Modern Image Deep Networks13
This paper aims to identify the real gains of popular convolution and attention operators through a detailed study, and finds that the key difference among these feature transformation modules, such as attention or convolution, lies in their spatial feature aggregation approach, known as the “spatial token mixer” (STM).
- 10
- 10
- 10
- 9
- 9
- 8
- 7
- 5
- 5
- 5
- 4
- 4
- 4
- 3
- 3
- 3
- 2
- 2
- 2
- 2
- 2
- 1
- 1
- 1
- 1
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-13. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.