Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Liqing ZhangSuggested from co-authorship
- Xiaogang WangSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works206 from public data
- Learning Deep Features for Discriminative Localization10,981
This work revisits the global average pooling layer proposed in [13], and sheds light on how it explicitly enables the convolutional neural network (CNN) to have remarkable localization ability despite being trained on imagelevel labels.
- Places: A 10 Million Image Database for Scene Recognition4,995
The Places Database is described, a repository of 10 million scene photographs, labeled with scene semantic categories, comprising a large and diverse list of the types of environments encountered in the world, using the state-of-the-art Convolutional Neural Networks as baselines, that significantly outperform the previous approaches.
- Scene Parsing through ADE20K Dataset4,155
The ADE20K dataset, spanning diverse annotations of scenes, objects, parts of objects, and in some cases even parts of parts, is introduced and it is shown that the trained scene parsing networks can lead to applications such as image content removal and scene synthesis.
- Learning Deep Features for Scene Recognition using Places Database3,107
A new scene-centric database called Places with over 7 million labeled pictures of scenes is introduced with new methods to compare the density and diversity of image datasets and it is shown that Places is as dense as other scene datasets and has more diversity.
- Unified Perceptual Parsing for Scene Understanding2,690
A multi-task framework called UPerNet and a training strategy are developed to learn from heterogeneous image annotations and it is shown that it is able to effectively segment a wide range of concepts from images.
- Semantic Understanding of Scenes Through the ADE20K Dataset2,505
This work presents a densely annotated dataset ADE20K, which spans diverse annotations of scenes, objects, parts of objects, and in some cases even parts of parts, and shows that the networks trained on this dataset are able to segment a wide variety of scenes and objects.
- Network Dissection: Quantifying Interpretability of Deep Visual Representations1,855
This work uses the proposed Network Dissection method to test the hypothesis that interpretability is an axis-independent property of the representation space, then applies the method to compare the latent representations of various networks when trained to solve different classification problems.
- Object Detectors Emerge in Deep Scene CNNs1,334
This work demonstrates that the same network can perform both scene recognition and object localization in a single forward-pass, without ever having been explicitly taught the notion of objects.
- Interpreting the Latent Space of GANs for Semantic Face Editing1,275
This work proposes a novel framework, called InterFaceGAN, for semantic face editing by interpreting the latent semantics learned by GANs, and finds that the latent code of well-trained generative models actually learns a disentangled representation after linear transformations.
- Temporal Relational Reasoning in Videos1,150
This paper introduces an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales.
- Measuring human perceptions of a large-scale urban region using machine learning798
A deep learning model, which has been trained on millions of human ratings of street-level imagery, was used to predict human perceptions of a street view image and can help to map the distribution of the city-wide human perception for a new urban region.
- InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs722
A framework called InterFaceGAN is proposed to interpret the disentangled face representation learned by the state-of-the-art GAN models and study the properties of the facial semantics encoded in the latent space to suggest that learning to synthesize faces spontaneously brings a disentangling and controllable face representation.
- In-Domain GAN Inversion for Real Image Editing697
An in-domain GAN inversion approach, which not only faithfully reconstructs the input image but also ensures the inverted code to be semantically meaningful for editing, which achieves satisfying real image reconstruction and facilitates various image editing tasks, significantly outperforming start-of-the-arts.
- Closed-Form Factorization of Latent Semantics in GANs664
This work examines the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner and proposes a closedform factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights.
- GAN Inversion: A Survey653
This paper provides a survey of GAN inversion with a focus on its representative algorithms and its applications in image restoration and image manipulation, and discusses the trends and challenges for future research.
- Moments in Time Dataset: One Million Videos for Event Understanding642
The Moments in Time dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis.
- Person Search with Natural Language Description593
An Recurrent Neural Network with Gated Neural Attention mechanism (GNA-RNN) is proposed to establish the state-of-the art performance on person search and a large-scale person description dataset with detailed natural language annotations and person samples from various sources is collected.
- Scene Graph Generation from Objects, Phrases and Region Captions560
This work proposes a novel neural network model, termed as Multi-level Scene Description Network (denoted as MSDN), to solve the three vision tasks jointly in an end-to-end manner and shows the joint learning across three tasks with the proposed method can bring mutual improvements over previous models.
- Scene Graph Generation from Objects, Phrases and Caption Regions.560
This work proposes a novel neural network model, termed as Multi-level Scene Description Network (denoted as MSDN), to solve the three vision tasks jointly in an end-to-end manner and shows the joint learning across three tasks with the proposed method can bring mutual improvements over previous models.
- Understanding the role of individual units in a deep neural network553
This work presents network dissection, an analytic framework to systematically identify the semantics of individual hidden units within image classification and image generation networks, and applies it to understanding adversarial attacks and to semantic image editing.
- Multimodal Motion Prediction with Stacked Transformers537
A novel network architecture based on stacked transformers is designed to model the multimodality at feature level with a set of fixed independent proposals, and achieves the state-of-the-art performance on motion prediction.
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning497
The generalization experiments conducted on both procedurally generated scenarios and real-world scenarios show that increasing the diversity and the size of the training set leads to the improvement of the RL agent's generalizability.
- Places: An Image Database for Deep Scene Understanding495
The Places Database is described, a repository of 10 million scene photographs, labeled with scene semantic categories and attributes, comprising a quasi-exhaustive list of the types of environments encountered in the world.
- Temporal Pyramid Network for Action Recognition456
A generic Temporal Pyramid Network (TPN) at the feature-level is proposed, which can be flexibly integrated into 2D or 3D backbone networks in a plug-and-play manner and shows consistent improvements over other challenging baselines on several action recognition datasets.
- CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers453
The extensive experiments on the V2V perception dataset, OPV2V, demonstrate that CoBEVT achieves state-of-the-art performance for cooperative BEV semantic segmentation and is shown to be generalizable to other tasks, including 1) BEV segmentation with single-agent multi-camera and 2) 3D object detection with multi-agent LiDAR systems.
- 382
- Semantic photo manipulation with a generative image prior372
This paper adapts the image prior learned by GANs to image statistics of an individual image and can accurately reconstruct the input image and synthesize new content, consistent with the appearance of theinput image.
- Interpreting Deep Visual Representations via Network Dissection362
Network Dissection is described, a method that interprets networks by providing meaningful labels to their individual units that reveals that deep representations are more transparent and interpretable than they would be under a random equivalently powerful basis.
- Seeing What a GAN Cannot Generate359
This work visualize mode collapse at both the distribution level and the instance level, and deploys a semantic segmentation network to compare the distribution of segmented objects in the generated images with the target distribution in the training set.
- Image Processing Using Multi-Code GAN Prior347
A novel approach is proposed, called mGANprior, to incorporate the well-trained GANs as effective prior to a variety of image processing tasks, by employing multiple latent codes to generate multiple feature maps at some intermediate layer of the generator and composing them with adaptive channel importance to recover the input image.
- V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception344
V2V4Real introduces three perception tasks, including cooperative 3D object detection, cooperative3D object tracking, and Sim2Real domain adaptation for cooperative perception, and provides comprehensive benchmarks of recent cooperative perception algorithms on three tasks.
- Cross-View Semantic Segmentation for Sensing Surroundings336
A novel visual task called Cross-view Semantic Segmentation as well as a framework named View Parsing Network (VPN) to address it and the experimental results show that the model can effectively make use of the information from different views and multi-modalities to understanding spatial information.
- Interpretable Basis Decomposition for Visual Explanation333
A new framework called Interpretable Basis Decomposition for providing visual explanations for classification networks is proposed, decomposing the neural activations of the input image into semantically interpretable components pre-trained from a large concept corpus.
- 303
- Deep Flow-Guided Video Inpainting285
This work first synthesizes a spatially and temporally coherent optical flow field across video frames using a newly designed Deep Flow Completion network, then uses the synthesized flow fields to guide the propagation of pixels to fill up the missing regions in the video.
- Factorizable Net: An Efficient Subgraph-Based Framework for Scene Graph Generation285
A subgraph-based connection graph is proposed to concisely represent the scene graph during the inference to improve the efficiency of scene graph generation and outperforms the state-of-the-art method in both accuracy and speed.
- TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization274
This paper introduces the token semantic coupled attention map (TS-CAM) to take full advantage of the self-attention mechanism in visual transformer for long-range dependency extraction and achieves state-of-the-art performance.
- 254
- TPNet: Trajectory Proposal Network for Motion Prediction234
This work proposes a novel two-stage motion prediction framework, Trajectory Proposal Network (TPNet), which first generates a candidate set of future trajectories as hypothesis proposals, then makes the final predictions by classifying and refining the proposals which meets the physical constraints.
- 229
- Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis216
This work shows that highly-structured semantic hierarchy emerges in the deep generative representations from the state-of-the-art GANs like StyleGAN and BigGAN, trained for scene synthesis, and quantifies the causality between the layer-wise activations and the semantics occurring in the output image.
- Experiment-free exoskeleton assistance via learning in simulation215
This work shows an experiment-free method to learn a versatile control policy in simulation that may offer a generalizable and scalable strategy for the rapid development and widespread adoption of a variety of assistive robots for both able-bodied and mobility-impaired individuals.
- Visual Question Generation as Dual Task of Visual Question Answering181
This paper proposes an end-to-end unified model, the Invertible Question Answering Network (iQAN), to introduce question generation as a dual task of question answering to improve the VQA performance and shows that the proposed dual training framework can consistently improve model performances of many popular V QA architectures.
- Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation177
Semantic-aware Speaking Portrait NeRF (SSP-NeRF), which creates delicate audio-driven portraits using one unified set of NeRF through two semantic-aware modules, and renders more realistic video portraits compared to previous methods.
- TrafficGen: Learning to Generate Diverse and Realistic Traffic Scenarios172
This work introduces a data-driven method called TrafficGen for traffic scenario generation that learns from the fragmented human driving data collected in the real world and then generates realistic traffic scenarios and shows substantial improvements over baselines for generating traffic scenarios.
- Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation169
This work proposes a novel framework named Hierarchical Audio-to-Gesture (HA2G) for co-speech gesture generation, and develops a contrastive learning strategy based on audio-text alignment for better audio representations.
- Instance Localization for Self-supervised Detection Pretraining169
This paper proposes a new self-supervised pretext task, called instance localization, based on the inherent difference between classification and detection, and shows that integration of bounding boxes into pretraining promotes better task alignment and architecture alignment for transfer learning.
- AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection164
This paper proposes AutoAlign, an automatic feature fusion strategy for 3D object detection that model the mapping relationship between the image and point clouds with a learnable alignment map, and designs a self-supervised cross-modal feature interaction module that can learn feature aggregation with instance-level feature guidance.
- 161
- SegICP: Integrated deep semantic segmentation and pose estimation158
SegICP couples convolutional neural networks and multi-hypothesis point cloud registration to achieve both robust pixel-wise semantic segmentation as well as accurate and real-time 6-DOF pose estimation for relevant objects.
- 154
- Coherent Filtering: Detecting Coherent Motions from Crowd Clutters154
A prior of coherent motion called Coherence Neighbor Invariance is proposed, which characterizes the local spatiotemporal relationships of individuals in coherent motion and a general technique of detecting coherent motion patterns from noisy time-series data called Coherent Filtering is proposed.
- Recognizing City Identity via Attribute Analysis of Geo-tagged Images148
This work proposes to characterize the identity of a city via an attribute analysis of 2 million geo-tagged images from 21 cities over 3 continents, using the scene attributes of these images to build a higher-level set of 7 city attributes, tailored to the form and function of cities.
- Open Vocabulary Scene Parsing146
A joint image pixel and word concept embeddings framework, where word concepts are connected by semantic relations, and validated the open vocabulary prediction ability of this framework on ADE20K dataset which covers a wide variety of scenes and objects.
- 3D-aware Image Synthesis via Learning Structural and Textural Representations145
This work proposes a novel framework, termed as VolumeGAN, for high-fidelity 3D-aware image synthesis, through explicitly learning a structural representation and a textural representation in a Generative Adversarial Network.
- Learning Collective Crowd Behaviors with Dynamic Pedestrian-Agents139
A new mixture model of dynamic pedestrian-Agents (MDA) is proposed to learn the collective behavior patterns of pedestrians in crowded scenes from video sequences and can infer the past behaviors and predict the future behaviors of pedestrians given their partially observed trajectories.
- A Local-to-Global Approach to Multi-Modal Movie Scene Segmentation138
This work builds a large-scale video dataset MovieScenes, which contains 21K annotated scene segments from 150 movies, and proposes a local-to-global scene segmentation framework, which integrates multi-modal information across three levels, i.e. clip, segment, and movie.
- FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition136
Free Control enables convenient training-free control over many different architectures and checkpoints, allows the challenging input conditions on which most of the existing training-free methods fail, and achieves competitive synthesis quality compared to training-based approaches.
- Street-View Image Generation From a Bird's-Eye View Layout131
A conditional generative model that synthesizes a set of realistic and spatially consistent surrounding images that match the BEV layout of a traffic scenario and incorporates a novel cross-view transformation with spatial attention design which learns the relationship between cameras and map views to ensure their consistency.
- Generative Hierarchical Features from Synthesizing Images126
This work considers the pre-trained StyleGAN generator as a learned loss function and utilizes its layer-wise representation to train a novel hierarchical encoder, termed as Generative Hierarchical Feature (GH-Feat), which has strong transferability to both generative and discriminative tasks.
- C-IMAGE: city cognitive mapping through geo-tagged photos103
This paper introduces a project called C-IMAGE, which analyzes the interactions between city and human perception through the massive amount of photos taken in 26 different cities: one based on the metadata and the other based on image content.
- A Unified Framework for Shot Type Classification Based on Subject Centric Lens100
A learning framework Subject Guidance Network (SGNet) for shot type recognition is proposed, which separates the subject and background of a shot into two streams, serving as separate guidance maps for scale and movement type classification respectively.
- Positional Encoding as Spatial Inductive Bias in GANs97
This work shows that SinGAN's impressive capability in learning internal patch distribution, to a large extent, is brought by the implicit positional encoding when using zero padding in the generators, and proposes a new multi-scale training strategy and demonstrates its effectiveness in the state-of-the-art unconditional generator StyleGAN2.
- CAT: Closed-loop Adversarial Training for Safe End-to-End Driving86
A novel resampling technique is developed to turn log-replay real-world driving scenarios into safety-critical ones via probabilistic factorization, where the adversarial traffic generation is modeled as the multiplication of standard motion prediction sub-problems.
- SimIPU: Simple 2D Image and 3D Point Cloud Unsupervised Pre-training for Spatial-Aware Visual Representations84
This is the first study to explore contrastive learning pre-training strategies for outdoor multi-modal datasets, containing paired camera images and LIDAR point clouds, in an unsupervised end-to-end fashion.
- DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis83
The proposed model spatially disentangles the whole scene into object-centric generative radiance fields by learning on only 2D images with the global-local discrimination, and obtains the generation fidelity and editing flexibility of individual objects while being able to efficiently compose objects and the background into a complete scene.
- Every Frame Counts: Joint Learning of Video Segmentation and Optical Flow80
This paper proposes a novel framework for joint video semantic segmentation and optical flow estimation that is able to utilize both labeled and unlabeled frames in the video through joint training, while no additional calculation is required in inference.
- Single Image Intrinsic Decomposition Without a Single Intrinsic Image79
This paper presents a two stream convolutional neural network framework that is capable of learning the decomposition effectively in the absence of any ground truth intrinsic images, and can be easily extended to a (semi-)supervised setup.
- 78
- A Graph-Based Framework to Bridge Movies and Synopses71
A dataset called Movie Synopses Associations (MSA) is constructed, which provides a synopsis for each movie, together with annotated associations between synopsis paragraphs and movie segments, and a framework to perform matching between movie segments and synopsis paragraphs is developed.
- Towards Text-guided 3D Scene Composition70
SceneWiz3D is introduced - a novel approach to synthesize high-fidelity 3D scenes from text by introducing a hybrid 3D representation - explicit for objects and implicit for scenes, and achieves superior quality over previous approaches.
- One-Shot Generative Domain Adaptation70
This work aims to transfer a Generative Adversarial Network pre-trained on one image domain to another domain referred to as few as just one reference image, and proposes to constrain the diversity of the latent space through truncation in the training process, alleviating the optimization difficulty.
- PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks69
This work addresses the problem of how to leverage instructional videos to facilitate the understanding of human decision-making processes, focusing on training a model with the ability to plan a goal-directed procedure from real-world videos with Planning Transformer (PlaTe), which has the advantage of circumventing the compounding prediction errors that occur with single-step models during long model-based rollouts.
- Visualizing and Understanding Generative Adversarial Networks (Extended Abstract).68
This work presents an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level, and identifies a group of interpretable units that are closely related to concepts with a segmentation-based network dissection method.
- Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognition66
This work proposes a more effective pseudo-labeling scheme, called Cross-Model Pseudo-Labeling (CMPL), which introduces a lightweight auxiliary network in addition to the primary backbone, and asks them to predict pseudo-labels for each other, and observes that these two models tend to learn complementary representations from the same video clips.
- TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting66
A lightweight video motion retargeting approach TransMoMo that is capable of transferring motion of a person in a source video realistically to another video of a target person by exploiting invariance properties of three orthogonal factors of variation including motion, structure, and view-angle.
- V2XP-ASG: Generating Adversarial Scenes for Vehicle-to-Everything Perception61
The first open adversarial scene generator V2XP-ASG is proposed that can produce realistic, challenging scenes for modern LiDAR-based multi-agent perception systems and learns to construct an adversarial collaboration graph and simultaneously perturb multiple agents' poses in a adversarial and plausible manner.
- 59
- Discovering place-informative scenes and objects using social media photos55
This work proposes a data-driven framework to explore the place-informative scenes and objects by employing deep convolutional neural network to learn and measure the visual knowledge of place appearance automatically from a massive dataset of photos and imagery.
- Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining54
This work aims to pretrain policy representations for driving tasks by watching hours-long uncurated YouTube videos by training an inverse dynamic model with a small amount of labeled data and using it to predict action labels for all the YouTube video frames.
- Texture Memory-Augmented Deep Patch-Based Image Inpainting53
A new deep inpainting framework where texture generation is guided by a texture memory of patch samples extracted from unmasked regions is proposed, which has a novel design that allows texture memory retrieval to be trained end-to-end with the deep inPainting network.
- Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation51
Vid2Sim is proposed, a novel framework that effectively bridges the sim2real gap through a scalable and cost-efficient real2sim pipeline with 3D scene reconstruction and simulation to enable the reinforcement learning of visual navigation agents in complex urban environments.
- 51
- ConceptLearner: Discovering visual concepts from weakly labeled image collections48
This paper proposes ConceptLearner, which is a scalable approach to discover visual concepts from weakly labeled image collections, which shows promising performance compared to fully supervised and weakly supervised methods.
- V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction47
This paper designs one-step and multi-step communication strategies for spatio-temporal fusion in V2X scenarios and examines their integration with three fusion strategies - early, late, and intermediate - providing comprehensive benchmarks with 11 fusion models (how to fuse).
- Next Steps for Human-Centered Generative AI: A Technical Perspective47
A comprehensive research agenda that lays out future directions of Generative AI spanning three levels: aligning with human values; assimilating human intents; and augmenting human abilities is contributed.
- Exploit Reward Shifting in Value-Based Deep-RL: Optimistic Curiosity-Based Exploration and Conservative Exploitation Via Linear Reward Shaping46
The key insight is brought that a positive reward shifting leads to conservative exploitation, while a negative reward shifting leads to curiosity-driven exploration, which improves offline RL value estimation, and optimistic value estimation improves exploration for online RL.
- A Phase Discrepancy Analysis of Object Motion42
A surprisingly simple algorithm to detect objects in such conditions by showing that the displacement of the foreground and the background can be represented by the phase change of Fourier spectra and the motion of background objects can be extracted by Phase Discrepancy in an efficient and robust way.
- Disentangled Inference for GANs With Latently Invertible Autoencoder39
In LIA, an invertible network and its inverse mapping are symmetrically embedded in the latent space of an autoencoder, and the decoder of LIA is first trained as a standard GAN with the invertable network, and then the encoder is learned from a disentangled autoen coder by detaching the inverted network from LIA.
- 37
- Reasoning About Human-Object Interactions Through Dual Attention Networks37
A Dual Attention Network model which reasons about human-object interactions is proposed, which not only finds when an action is happening and which object is being manipulated, but also identifies which part of the object isbeing interacted with.
- Recurrent Residual Module for Fast Inference in Videos37
This work proposes a framework called Recurrent Residual Module (RRM) to accelerate the CNN inference for video recognition tasks, which has a novel design of using the similarity of the intermediate feature maps of two consecutive frames to largely reduce the redundant computation.
- Improving GAN Equilibrium by Raising Spatial Awareness35
This work aims to alleviate the issue of D dominating the competition in GANs, and proposes to align the spatial awareness of G with the attention map induced from D, effectively lessen the information gap between D and G.
- Real-Time Object Pose Estimation with Pose Interpreter Networks34
This work introduces pose interpreter networks for 6-DoF object pose estimation and shows that when combined with a segmentation model trained on RGB images, this synthetically trained pose interpreter network is able to generalize to real data.
- Joint Optimization for 4D Human-Scene Reconstruction in the Wild33
A novel optimization-based method for 4D human-scene reconstruction in the wild from monocular videos, JOSH, which achieves better results on both global human motion estimation and dense scene reconstruction by joint optimization of scene geometry and human motion.
- 3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting33
3DitScene is proposed, a novel and unified scene editing framework leveraging language-guided disentangled Gaussian Splatting that enables seamless editing from 2D to 3D, allowing precise control over scene composition and individual objects.
- Visual Sound Localization in the Wild by Cross-Modal Interference Erasing32
Quantitative and qualitative evaluations demonstrate that the Interference Eraser framework achieves superior results on sound localization tasks, especially under real world scenarios.
- 31
- 28
- Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoning27
This work introduces Counterfactual VLA (CF-VLA), a self-reflective VLA framework that enables the model to reason about and revise its planned actions before execution, and proposes a rollout-filter-label pipeline that mines high-value scenes from a base (non-counterfactual) VLA's rollouts and labels counterfactual reasoning traces for subsequent training rounds.
- DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation27
It is shown that DeepMiner not only enables better understanding of the nuances of CNN classification decisions, but also possibly discovers new visual knowledge relevant to medical diagnosis.
- 26
- Embodied Scene Understanding for Vision Language Models via MetaVQA23
MetaVQA is presented, a comprehensive benchmark designed to assess and enhance VLMs’ understanding of spatial relationships and scene dynamics through Visual Question Answering (VQA) and closed-loop simulations, and exhibits strong transferability from simulation to real-world observation.
- 23
- Guarded Policy Optimization with Imperfect Online Demonstrations22
An Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared Control (TS2C), which incorporates teacher intervention based on trajectory-based value estimation is instantiated and validates that the proposed TS2C algorithm attains efficient exploration and substantial safety guarantee without being affected by the teacher's own performance.
- 22
- CooPre: Cooperative Pretraining for V2X Cooperative Perception21
This paper presents a self-supervised learning framwork for V2X cooperative perception, which utilizes the vast amount of unlabeled 3D V2X data to enhance the perception performance and develops a V2X bird-eye-view (BEV) guided masking strategy which effectively allows the model to pay attention to 3D features across heterogeneous V2X agents.
- 21
- LIA: Latently Invertible Autoencoder with Adversarial Learning21
The two-stage stochasticity-free training is devised to train LIA via adversarial learning, in the sense that it first train a standard GAN whose generator is the decoder of LIA and then an autoencoder in the adversarial manner by detaching the invertible network from LIA.
- Vista4D: Video Reshooting with 4D Point Clouds20
Vista4D is presented, a robust and flexible video reshooting framework that grounds the input video and target cameras in a 4D point cloud and generalizes to real-world applications such as dynamic scene expansion and 4D scene recomposition.
- Towards Autonomous Micromobility through Scalable Urban Simulation20
A scalable urban simulation solution to advance autonomous micromobility and a high-performance robot learning platform for large-scale training of embodied agents in interactive urban scenes and a suite of essential tasks and benchmarks to gauge various capabilities of the AI agents in achieving autonomous micromobility.
- 20
- Towards Smooth Video Composition19
This work investigates modeling the temporal relations for composing video with arbitrary length, from a few frames to even infinite, using generative adversarial networks (GANs), and develops a novel B-Spline based motion representation to ensure temporal smoothness to achieve infinite-length video generation.
- Accurate prediction of disease-risk factors from volumetric medical scans by a deep vision model pre-trained with 2D scans18
A deep-learning model pre-trained on readily available 2D scans outperforms domain-specific state-of-the-art models in the prediction of disease-risk factors from 3D medical-scan modalities.
- Expert identification of visual primitives used by CNNs during mammogram classification18
This work interprets the internal representations of deep neural networks trained for classification of diseased tissue in 2D mammograms using an expert-in-the-loop inter- pretation method to label the behavior of internal units in convolutional neural networks (CNNs).
- Comparing the Interpretability of Deep Networks via Network Dissection17
This chapter introduces Network Dissection, a general framework to quantify the interpretability of the units inside a deep convolutional neural networks (CNNs), and compares the different vocabularies of interpretable units as concept detectors emerged from the networks trained to solve different supervised learning tasks.
- MetaUrban: An Embodied AI Simulation Platform for Urban Micromobility16
MetaUrban, a compositional simulation platform for Embodied AI research in urban spaces, is presented and experiments demonstrate that the compositional nature of the simulated environments can substantially improve the generalizability and safety of the trained mobile agents.
- Efficient 3D Articulated Human Generation with Layered Surface Volumes16
This work introduces layered surface volumes (LSVs) as a new 3D representation for articulatable humans and trains on unstructured, single-view 2D image datasets, which generates high-quality and viewconsistent 3D articulated digital humans without the need for view-inconsistent 2D upsampling networks.
- Human-AI Shared Control Via Policy Dissection16
The experiments show that human-AI shared control achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes and suggest the promising direction of implementing human- AI shared autonomy through interpreting the learned representation of the autonomous agents.
- X-Fusion: Introducing New Modality to Frozen Large Language Models15
XFusion is proposed, a framework that extends pretrained Large Language Models for multimodal tasks while preserving their language capabilities by employing a dual-tower design with modality-specific weights, keeping the LLM's parameters frozen while integrating vision-specific information for both understanding and generation.
- Urban Scene Diffusion through Semantic Occupancy Map15
This work proposes UrbanDiffusion, a 3D diffusion model that is conditioned on a Bird's-Eye View (BEV) map and generates an urban scene with geometry and semantics in the form of semantic occupancy map and introduces a novel paradigm that learns the data distribution of scene-level structures within a latent space and further enables the expansion of the synthesized scene into an arbitrary scale.
- 15
- In-Domain GAN Inversion for Faithful Reconstruction and Editability14
In-domain GAN inversion is proposed, which consists of a domain-guided encoder and a domain-regularized optimizer, to regularize the inverted code in the native latent space of the pre-trained GAN model, to sufficiently reuse the knowledge learned by GANs for image reconstruction, facilitating a wide range of editing applications without any retraining.
- 14
- 14
- TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction12
This work introduces TurboTrain, a novel and efficient training framework for multi-agent perception and prediction, and demonstrates that it further improves the performance of state-of-the-art multi-agent perception and prediction models.
- Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism12
The Adaptive Intervention Mechanism (AIM), a novel robot-gated IIL algorithm that learns an adaptive criterion for requesting human demonstrations that achieves a 40% improvement in terms of human take-over cost and learning efficiency.
- 12
- Occupancy Learning with Spatiotemporal Memory11
ST-Occ is proposed, a scene-level occupancy representation learning framework that effectively learns the spatiotemporal feature with temporal consistency and significantly enhances the spatiotemporal representation learned for 3D occupancy prediction tasks by exploiting the temporal dependency between multi-frame inputs.
- 11
- 11
- 11
- BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Driving10
This paper demonstrates that OL policies suffer from Observational Domain Shift and Objective Mismatch, and proposes a Test-Time Adaptation (TTA) framework that calibrates observational shift, reduces state-action biases, and enforces temporal consistency.
- 10
- Localization and recognition of human action in 3D using transformers9
This work introduces a new, challenging, and more complex benchmark dataset, BABEL-TAL (BT), for 3D action localization, and proposes a strong baseline model, i.e., Localizing Actions with Transformers (LocATe), that jointly localizes and recognizes actions in a 3D sequence.
- LocATe: End-to-end Localization of Actions in 3D with Transformers9
LocATe is proposed, an end-to-end approach that jointly localizes and recognizes actions in a 3D sequence and outperforms previous approaches on the existing PKU-MMD 3D-TAL benchmark (mAP=93.2%).
- 9
- UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos8
Experiments show that UrbanVerse scenes preserve real-world semantics and layouts, achieving human-evaluated realism comparable to manually crafted scenes, in urban navigation and in urban navigation policies trained in UrbanVerse exhibit scaling power laws and strong generalization.
- AnimaMimic: Imitating 3D Animation from Video Priors8
This work presents AnimaMimic, a framework that animates static 3D meshes using motion priors learned from video diffusion models, and integrates a differentiable simulation module that refines mesh deformation through physically grounded soft-tissue dynamics.
- 8
- Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers7
WorldWeaver (W^2), a streaming multi-agent video diffusion model that augments rollout with cross-agent world state registers: learnable tokens that store shared world information, track individual agent status, and are dynamically updated after each generated chunk is presented.
- Data-Efficient Learning from Human Interventions for Mobile Robots7
An online human-in-the-loop learning method PVP4Real enables efficient real-time policy learning from online human intervention and demonstration, without reward or any pretraining, significantly improving data efficiency and training safety.
- BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation7
A practical and efficient 3D representation that incorporates an equivariant radiance field with the guidance of a bird's-eye view (BEV) map is proposed that allows for large-scale, even infinite-scale, 3D scenes via synthesizing local scenes and then stitching them with smooth consistency.
- Learning to Generate Diverse Pedestrian Movements from Web Videos with Noisy Labels7
A generative model called PedGen that contains a novel context encoder that lifts the 2D scene context to 3D and can incorporate various context factors in generating realistic pedestrian movements in urban scenes and achieves zero-shot generalization in both real-world and simulated environments.
- 7
- MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systems6
This paper introduces MDrive, a closed-loop cooperative driving benchmark comprising 225 scenarios grounded in both NHTSA pre-crash typologies and real-world V2X datasets that establishes a reproducible foundation for evaluating and improving the generalization and robustness of cooperative driving systems.
- Dreamland: Controllable World Creation with Simulator and Generative Models6
This work proposes Dreamland, a hybrid world generation framework combining the granular control of a physics-based simulator and the photorealistic content output of large-scale pretrained generative models, and designs a layered world abstraction that encodes both pixel-level and object-level semantics and geometry as an intermediate representation to bridge the simulator and the generative model.
- Interpreting Generative Adversarial Networks for Interactive Image Generation6
This chapter gives a summary of recent works on interpreting deep generative models and sees how the human-understandable concepts that emerge in the learned representation can be identified and used for interactive image generation and editing.
- 6
- 6
- 6
- Learning Sidewalk Autopilot from Multi-Scale Imitation with Corrective Behavior Expansion5
This work proposes a framework that advances IL through corrective behavior expansion and multi-scale imitation learning and introduces a multi-scale IL architecture that captures both short-horizon interactive behaviors and long-horizon goal-directed intentions via horizon-based trajectory clustering and hierarchical supervision.
- DiSCo: Diffusion Sequence Copilots for Shared Autonomy5
Diffusion Sequence Copilots (DiSCo) is introduced: a method of shared autonomy with diffusion policy that plans action sequences consistent with past user actions that substantially improves task performance in simulated driving and robotic arm tasks.
- BEVCon: Advancing Bird's Eye View Perception With Contrastive Learning5
The dense contrastive learning designed on top of detection losses leads to improved feature representations across both the BEV encoder and the backbone, and highlights the critical role of representation learning in BEV perception.
- WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving5
By incorporating interaction and intention language from WOMD-Reasoning, the authors see significant enhancements in the performance of the state-of-the-art trajectory prediction model, Multipath++, with improvements of 10.14% in MR 6 and 6.90% in minFDE 6, proving the effectiveness of WOMD-Reasoning.
- Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping5
The key insight is brought that a positive reward shifting leads to conservative exploitation, while a negative reward shift leads to curiosity-driven exploration, which improves offline RL value estimation, and optimistic value estimation improves exploration for online RL.
- 5
- Improving Out-of-Distribution Robustness of Classifiers via Generative Interpolation4
This paper develops a simple yet effective method called Generative Interpolation to fuse generative models trained from multiple domains for synthesizing diverse OoD samples and explicitly increases the diversity of training domains and achieves consistent improvements over baselines across datasets and multiple different distribution shifts.
- HiABP: Hierarchical Initialized ABP for Unsupervised Representation Learning4
Hierarchical Initialized Alternating Backpropagation (HiABP) is proposed for efficient Bayesian inference of hierarchical latent representation with controllable image synthesis and can outperform other popular deep generative models in modeling natural images and learning from incomplete data.
- 4
- 4
- 4
- Interpretable representation learning for visual intelligence4
This thesis investigates several novel approaches for opening up the “black box” of neural networks used in visual recognition tasks and understanding their inner working mechanism, and describes an approach that can efficiently explain the output prediction for any given image.
- Scene Gist: A Holistic Generative Model of Natural Image4
This paper proposes a novel generative model for natural image representation and scene classification that is decomposed with learned holistic basis called scene gist components, a global and adaptive image descriptor, generatively including most essential information related to visual perception.
- 3
- Monocular 3D Occupancy Perception for Robots on Sidewalks via Hybrid 2D-3D Learning3
WalkOCC is proposed, a hybrid Ray-marching monocular 3D occupancy perception framework for robots operating on sidewalks that yields stable optimization and improved generalization without requiring costly 3D occupancy annotations.
- Verbalized Representation Learning for Interpretable Few-Shot Generalization3
This work proposes Verbalized Representation Learning (VRL), a novel approach for automatically extracting human-interpretable features for object recognition using few-shot data and achieves a 24% absolute improvement over prior state-of-the-art methods while using 95% less data and a smaller model.
- Unsupervised Discovery of Steerable Factors When Graph Deep Generative Models Are Entangled3
This work proposes GraphCG, a method for the unsupervised discovery of steerable factors in the latent space of pretrained graph DGMs via maximizing the mutual information between semantic-rich directions, where the controlled graph moving along the same direction will share the same steerable factors.
- Spatial Steerability of GANs via Self-Supervision from Discriminator3
This work designs randomly sampled Gaussian heatmaps to be encoded into the intermediate layers of generative models as spatial inductive bias, and incorporates DragGAN into the framework, which facilitates fine-grained manipulation within a reasonable time and supports a coarse-to-fine editing process.
- GH-Feat: Learning Versatile Generative Hierarchical Features From GANs3
This work investigates that a generative feature learned from image synthesis exhibits great potentials in solving a wide range of computer vision tasks, including both generative ones and more importantly discriminative ones, and develops GH-Feat, a hierarchical visual feature with multi-level semantics.
- Learning Driving Decisions by Imitating Drivers' Control Behaviors3
This work proposes a hybrid framework for learning a decision module, which is agnostic to the mechanisms of perception, planning, and control modules, and demonstrates that the simulation agents with a learned decision module can be generalized to various complex driving scenarios where the rule-based approach fails.
- SidewalkBench: Benchmarking Visual Navigation on Urban Sidewalks2
This work proposes SidewalkBench, a comprehensive benchmark designed for visual navigation on urban sidewalks, built upon NVIDIA Isaac Sim, which brings GPU-accelerated simulation of diverse, high-fidelity sidewalk environments, including both procedurally generated and real-world scanned scenes.
- AURA: Multimodal Shared Autonomy for Real-World Urban Navigation2
AURA is a new multi-modal framework that decomposes urban navigation into high-level human instruction and low-level AI control, and effectively follows human instructions, reduces manual operation effort, and improves navigation stability, while enabling online adaptation.
- Slow Brain, Fast Planner: Latency-Resilient VLM-Augmented Urban Navigation2
A VLM-Planner interface is proposed that uses a VLM to select a candidate index from the planner's proposal set and then fuse it with the planner's initial output, which turns a stale VLM selection into real-time planner scoring via geometric similarity with exponential decay.
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 1
- 1
- From Imitation to Alignment: Human-Preference Flow Policies for Long-Horizon Sidewalk Navigation1
FlowPilot is introduced, a mapless navigation policy that achieves robust and efficient long-horizon navigation performance using only a monocular RGB camera and a human-in-the-loop preference learning scheme to tune the policy on a small amount of human intervention data strengthens the model's counterfactual reasoning and social compliance on sidewalks.
- 1
- Group Diffusion: Enhancing Image Generation by Unlocking Cross-Sample Collaboration1
This work proposes Group Diffusion, unlocking the attention mechanism to be shared across images, rather than limited to just the patches within an image, which enables images to be jointly denoised at inference time, learning both intra and inter-image correspondence.
- 1
- Hyper: Hyperparameter Robust Efficient Exploration in Reinforcement Learning1
This paper describes the difficulty and the instability of the optimization when the agent learns with curiosity and proposes a method, hyperparameter robust exploration, which extensively mitigates the problem by effectively regularizing the visitation of the exploration and decoupling the exploitation to ensure stable training.
- 1
- A Hierarchial Model for Visual Perception1
Analysis of the perceptual manifold reveals that scene images which share similar perceptual similarities stay nearby in the manifold space, and the dimensions of the space could describe the spatial layout of scenes, which are like the degree of naturalness, openness supervised trained in this model.
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- Modeling Manifold Ways of Scene Perception–
Under the efficient coding theory, a computational model to explore the intrinsic dimensionality of scene perception by pooling together the activity of local low-level feature detectors across a large regions of the visual fields is proposed.
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-15. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.