Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileWorks486 from public data
- Visual interpretability for deep learning: a survey949
It is believed that high model interpretability may help people break several bottlenecks of deep learning, e.g., learning from a few annotations, learning via human–computer communications at the semantic level, and semantically debugging network representations.
- Interpretable Convolutional Neural Networks855
A method to modify a traditional convolutional neural network into an interpretable CNN, in order to clarify knowledge representations in high conv-layers of the CNN, which can help people understand the logic inside a CNN.
- Cross-View Action Modeling, Learning, and Recognition659
A novel multiview spatio-temporal and-or graph (MST-AOG) representation for cross-view action recognition, which takes advantage of the 3D human skeleton data obtained from Kinect cameras to avoid annotating enormous multi-view video frames, but the recognition does not need 3D information and is based on 2D video input.
- 647
- Learning Human-Object Interactions by Graph Parsing Neural Networks596
This paper addresses the task of detecting and recognizing human-object interactions (HOI) in images and videos with the Graph Parsing Neural Network (GPNN), a framework that incorporates structural knowledge while being differentiable end-to-end.
- Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning571
This work constructs a new largescale benchmark, Geometry3K, consisting of 3,002 geometry problems with dense annotation in formal language, and proposes a novel geometry solving approach with formal language and symbolic reasoning, called Interpretable Geometry Problem Solver (InterGPS).
- Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning476
A novel approach is proposed, PromptPG, which utilizes policy gradient to learn to select in-context examples from a small amount of training data and then constructs the corresponding prompt for the test example, which verifies its effectiveness in selecting in- context examples.
- An Embodied Generalist Agent in 3D World453
This work introduces LEO, an embodied multi-modal generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world and demonstrates its remarkable proficiency across a wide spectrum of tasks, including 3D captioning, question answering, embodied reasoning, navigation and manipulation.
- RAVEN: A Dataset for Relational and Analogical Visual REasoNing421
This work proposes a new dataset, built in the context of Raven's Progressive Matrices (RPM) and aimed at lifting machine intelligence by associating vision with structural, relational, and analogical reasoning in a hierarchical representation and establishes a semantic link between vision and reasoning by providing structure representation.
- 395
- Learning Pose Grammar to Encode Human Body Configuration for 3D Pose Estimation392
This paper proposes a pose grammar to tackle the problem of 3D human pose estimation, which takes 2D pose as input and learns a generalized 2D-3D mapping function and enforces high-level constraints over human poses.
- Diffusion-based Generation, Optimization, and Planning in 3D Scenes358
The SceneDiffuser is introduced, a conditional generative model for 3D scene understanding that jointly formulates the scene-aware generation, physics-based optimization, and goal-oriented planning via a diffusion-based denoising process in a fully differentiable fashion.
- Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds276
Experimental results demonstrate that, compared with supervised learning methods, the learned self-supervised representation facilitates various models to attain comparable or even better performances while capable of generalizing pre-trained models to downstream tasks, including 3D shape classification, 3D object detection, and 3D semantic segmentation.
- What are Textons?271
A three-level generative image model for learning textons from texture images and the geometric, dynamic, and photometric structures of the texton representation are studied by further extending the generative model to account for motion and illumination variations.
- Interpreting CNN Knowledge via an Explanatory Graph255
This paper learns a graphical model, namely an explanatory graph, which reveals the knowledge hierarchy hidden inside a pre-trained CNN in an unsupervised manner, i.e., without a need of annotating object parts.
- Introduction to a Large-Scale General Purpose Ground Truth Database: Methodology, Annotation Tool and Benchmarks246
A large scale general purpose image database with human annotated ground truth consisting of more than 636,748 annotated images and video frames is presented.
- 241
- Human-Centric Indoor Scene Synthesis Using Stochastic Grammar209
A human-centric method to sample and synthesize 3D room layouts and 2D images thereof, to obtain large-scale 2D/3D image data with the perfect per-pixel ground truth is presented.
- 207
- DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare203
A novel end-to-end framework for jointly estimating 3D human pose and body shape from a monocular RGB image and a large-scale synthetic dataset utilizing web-crawled Mocap sequences, 3D scans and animations is constructed.
- On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models195
This study investigates the effects of Markov chain Monte Carlo (MCMC) sampling in unsupervised Maximum Likelihood (ML) learning and identifies a variety of ML learning outcomes that depend solely on the implementation of MCMC sampling.
- CERN: Confidence-Energy Recurrent Network for Group Activity Recognition193
This work is about recognizing human activities occurring in videos at distinct semantic levels, including individual actions, interactions, and group activities, using a two-level hierarchy of Long Short-Term Memory networks, forming a feed-forward deep architecture, which can be trained end-to-end.
- Synthesizing Diverse and Physically Stable Grasps With Arbitrary Hand Structures Using Differentiable Force Closure Estimator188
A fast and differentiable force closure estimator, capable of producing diverse and physically stable grasps with arbitrary hand structures, without any training data is formulated, which can test a force closure within milliseconds.
- 188
- Joint inference of groups, events and human roles in aerial videos187
This paper addresses a new problem of parsing low-resolution aerial videos of large spatial areas, in terms of grouping, recognizing events and 3) assigning roles to people engaged in events, using a spatiotemporal AND-OR graph.
- Learning Active Basis Model for Object Detection and Recognition183
An active basis model, a shared sketch algorithm, and a computational architecture of sum-max maps for representing, learning, and recognizing deformable templates are proposed.
- 175
- ProAgent: Building Proactive Cooperative Agents with Large Language Models172
ProAgent is a novel framework that harnesses large language models (LLMs) to create proactive agents capable of dynamically adapting their behavior to enhance cooperation with teammates, and exhibits a high degree of modularity and interpretability, making it easily integrated into various of coordination scenarios.
- 172
- Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense169
It is argued that the next generation of AI must embrace "dark" humanlike common sense for solving novel tasks, and is demonstrated how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning.
- From image parsing to painterly rendering169
This work presents a semantics-driven approach for stroke-based painterly rendering, based on recent image parsing techniques, which benefits from richer meaningful image semantic information, which leads to better simulation of painting techniques of artists using the high-quality brush dictionary.
- 166
- Learning Descriptor Networks for 3D Shape Synthesis and Analysis157
A 3D shape descriptor network, which is a deep convolutional energy-based model, for modeling volumetric shape patterns and can be used to train a 3D generator network via MCMC teaching.
- 157
- Embedding Gestalt laws in Markov random fields154
This paper provides a quantitative measure for the so-called nonaccidental statistics and, thus, justifies some empirical observations of Gestalt psychology by information theory.
- 153
- 153
- A tale of two explanations: Enhancing human trust by explaining robot behavior152
A psychological experiment to examine what forms of explanations best foster human trust in the robot found that comprehensive and real-time visualizations of the robot’s internal decisions were more effective in promoting human trust than explanations based on summary text descriptions.
- Cooperative Training of Descriptor and Generator Networks152
It is shown that the cooperative algorithm can learn highly realistic generative models, including a deep energy-based model and a non-linear version of factor analysis.
- Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image148
A Holistic Scene Grammar (HSG) is introduced to represent the 3D scene structure, which characterizes a joint distribution over the functional and geometric space of indoor scenes, and significantly outperforms prior methods on 3D layout estimation, 3D object detection, and holistic scene understanding.
- Cost-Sensitive Top-Down/Bottom-Up Inference for Multiscale Activity Recognition148
A three-layered AND-OR graph is used to jointly model group activities, individual actions, and participating objects and optimally schedules the following computational processes: direct application of activity detectors, bottom-up inference based on detecting activity parts, and top-down inference based upon detecting activity context.
- 146
- 145
- 142
- Alternating Back-Propagation for Generator Network140
It is shown that the alternating back-propagation algorithm can learn realistic generator models of natural images, video sequences, and sounds and can also be used to learn from incomplete or indirect training data.
- 137
- Exploring texture ensembles by efficient Markov chain Monte Carlo-Toward a "trichromacy" theory of texture136
The intrinsic link between the scientific definition of texture on Z/sup 2/ and the mathematical models of texture on finite lattices is established and an efficient Markov chain Monte Carte algorithm is proposed for sampling Julesz ensembles.
- Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning133
A spatio-temporal graph neural network is proposed to explicitly represent the diverse gaze interactions in the social scenes and to infer atomic- level gaze communication by message passing and an event network with encoder-decoder structure to predict the event-level gaze communication.
- 133
- Holistic++ Scene Understanding: Single-View 3D Holistic Scene Parsing and Human Pose Estimation With Human-Object Interaction and Physical Commonsense132
A new 3D holistic++ scene understanding problem, which jointly tackles two tasks from a single-view image: (i) holistic scene parsing and reconstruction---3D estimations of object bounding boxes, camera pose, and room layout, and (ii) 3D human pose estimation.
- 128
- 127
- 126
- 126
- Reasoning Visual Dialogs With Structural and Partial Observations125
This paper introduces an Expectation Maximization algorithm to infer both the underlying dialog structures and the missing node values (desired answers) and proposes a differentiable graph neural network (GNN) solution that approximates this process.
- Joint Video and Text Parsing for Understanding Events and Answering Queries121
A multimedia analysis framework to process video and text jointly for understanding events and answering user queries and shows that deep joint parsing facilitates subsequent applications such as generating narrative text descriptions and answering queries in the forms of who, what, when, where, and why.
- 119
- Primal sketch: Integrating structure and texture119
A generative image model, which is called ''primal sketch,'' following Marr's insight and terminology is proposed, which combines two prominent classes of generative models, namely, sparse coding model and Markov random field model for representing geometric structures and stochastic textures, respectively.
- 117
- MindAgent: Emergent Gaming Interaction115
This work proposes a novel infrastructure - MindAgent - to evaluate planning and coordination emergent capabilities for gaming interaction and introduces CUISINEWORLD, a new gaming scenario and related benchmark that dispatch a multi-agent collaboration efficiency and supervise multiple agents playing the game simultaneously.
- In situ bidirectional human-robot value alignment113
This work proposes an explainable artificial intelligence system in which a group of robots predicts users' values by taking in situ feedback into consideration while communicating their decision processes to users through explanations, and integrates a cooperative communication model for inferring human values associated with multiple desirable goals.
- Examining CNN Representations With Respect to Dataset Bias113
This paper proposes a simple yet effective method to diagnose feature representations of the CNN by comparing latent relationships between representations of different attributes inside the CNN to discover the CNN's blind spots and failure modes due to dataset bias.
- 113
- CoCoX: Generating Conceptual and Counterfactual Explanations via Fault-Lines108
It is argued that, due to the conceptual and counterfactual nature of fault-lines, the CoCoX explanations are practical and more natural for both expert and non-expert users to understand the internal workings of complex deep learning models.
- 108
- 108
- Latent Diffusion Energy-Based Model for Interpretable Text Modeling107
A novel symbiosis between the diffusion models and latent space EBMs in a variational learning framework is introduced, coined as the latent diffusion energy-based model, which develops a geometric clustering-based regularization jointly with the information bottleneck to further improve the quality of the learned latent space.
- 105
- 101
- 99
- Image Parsing with Stochastic Scene Grammar98
An efficient MCMC inference algorithm, namely Hierarchical cluster sampling, to search in the large solution space of scene configurations, and achieves richer structures in the parse tree.
- 97
- Predicting Human Activities Using Stochastic Grammar96
This paper uses a stochastic grammar model to capture the compositional structure of events, integrating human actions, objects, and their affordances, and presents a novel method to predict future human activities from partially observed RGB-D videos.
- 96
- 95
- ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes93
The results indicate that current models for language-conditioned manipulations continue to experience significant challenges in novel goal-state generalizations, scene generalizations, and object generalizations, which highlights the need to develop new algorithms that address this gap.
- Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners92
It is hypothesized that the learned semantics of language tokens do the most heavy lifting during the reasoning process of Large Language Models, and a novel perspective is provided on the role of semantics in developing and evaluating language models' reasoning abilities.
- Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification91
This model synthesizes point clouds by matching observed examples in terms of statistical properties defined by the energy function, and can learn a short-run MCMC toward the energy-based model as a flow-like generator for point cloud reconstruction and interpolation.
- Generative PointNet: Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification91
This model synthesizes point clouds by matching observed examples in terms of statistical properties defined by the energy function, and can learn a short-run MCMC toward the energy-based model as a flow-like generator for point cloud reconstruction and interpolation.
- 91
- A massively parallel and scalable multi-GPU material point method90
This work introduces a new particle data structure that promotes coalesced memory access patterns on the GPU and eliminates the need for complex atomic operations on the memory hierarchy when writing particle data to the grid, and proposes a kernel fusion approach using a new Grid-to-Particles- to-Grid (G2P2G) scheme that efficiently reduces GPU kernel launches, improves latency, and significantly reduces the amount of global memory needed to store particle data.
- Cross-View People Tracking by Scene-Centered Spatio-Temporal Parsing90
A Spatio-temporal Attributed Parse Graph (ST-APG) is proposed to integrate semantic attributes with trajectories for cross-view people tracking to achieve promising multi-view tracking results.
- 89
- 89
- 88
- Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution87
A neuro-symbolic Probabilistic Abduction and Execution (PrAE) learner is proposed, central to the process of probabilistic abduction and execution on a probabilism scene representation, akin to the mental manipulation of objects, that improves cross-configuration generalization and is capable of rendering an answer.
- 86
- 83
- 83
- LEMMA: A Multi-view Dataset for L Earning Multi-agent Multi-task Activities82
The LEMMA dataset is introduced to provide a single home to address missing dimensions of daily human activities, including the goal-directed actions, concurrent multi-tasks, and collaborations among multi-agents with meticulously designed settings.
- Learning Generative ConvNets via Multi-grid Modeling and Sampling82
It is shown that this multi-grid method can learn realistic energy-based generative ConvNet models, and it outperforms the original contrastive divergence (CD) and persistent CD.
- Online Object Tracking, Learning and Parsing with And-Or Graphs82
Three key issues in online inference and learning are addressed: (i) maintaining purity of positive and negative examples collected online, (ii) controling model complexity in latent structure learning, and (iii) identifying critical moments to re-learn the structure of AOG based on its intrackability.
- Integrating Context and Occlusion for Car Detection by Hierarchical And-Or Model81
A method of learning reconfigurable hierarchical And-Or models to integrate context and occlusion for car detection using Weak-Label Structural SVM and compares with state-of-the-art variants of deformable part-based models and other methods.
- 79
- ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting77
This work introduces ArtGS, a novel approach that leverages 3D Gaussians as a flexible and efficient representation to address issues of part-mesh reconstruction and part dynamics modeling, particularly for complex multi-part articulated objects.
- 77
- 77
- 77
- 76
- Divergence Triangle for Joint Training of Generator Model, Energy-Based Model, and Inferential Model74
The divergence triangle is a compact and symmetric (anti-symmetric) objective function that seamlessly integrates variational learning, adversarial learning, wake-sleep algorithm, and contrastive divergence in a unified probabilistic formulation that makes the processes of sampling, inference, and energy evaluation readily available.
- 74
- Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning74
A learning strategy that embeds object-part concepts into a pre-trained convolutional neural network in an attempt to explore explicit semantics hidden in CNN units and gradually transform the pre- trained CNN into a semantically interpretable graphical model for hierarchical object understanding.
- Learning Perceptual Causality from Video74
This article provides a framework for the unsupervised learning of this perceptual causal structure from video, and takes action and object status detections as input and uses heuristics suggested by cognitive science research to produce the causal links perceived between them.
- Interpretable CNNs for Object Classification73
This paper proposes a generic method to learn interpretable convolutional filters in a deep Convolutional neural network for object classification, where each interpretable filter encodes features of a specific object part.
- 72
- Portrait painting using active templates71
This paper presents an example-based method to render portrait paintings from photographs, by transferring brush strokes from previously painted portrait templates by artists, with a dictionary of portrait painting templates for different types of faces.
- 70
- 70
- Sisley the abstract painter70
An interactive abstract painting system named Sisley, which automatically generates corresponding abstract painting images, with increased ambiguities of both the scene and individual objects at desired levels, and has similar abstract effects to original abstract paintings by artists.
- 70
- 69
- YouRefIt: Embodied Reference Understanding with Language and Gesture68
YouRefIt is introduced, a new crowd-sourced dataset of embodied reference collected in various physical scenes that allows us to study referring expressions in daily physical scenes to understand referential behavior, human communication, and human-robot interaction.
- 68
- 67
- Learning FRAME Models Using CNN Filters67
This conceptual paper proposes to learn the generative FRAME model using the highly expressive filters pre-learned by the CNN at the convolutional layers, and shows that the learning algorithm can generate realistic and rich object and texture patterns in natural scenes.
- 66
- 66
- Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage65
A multi-modal agent tuning method that automatically generates multi-modal tool-usage data and tunes a vision-language model (VLM) as the controller for powerful tool-usage reasoning is proposed.
- Learning by Fixing: Solving Math Word Problems with Weak Supervision65
This paper proposes a novel learning-by-fixing (LBF) framework, which corrects the misperceptions of the neural network via symbolic reasoning and achieves comparable top-1 and much better top-3/5 answer accuracies than fully-supervised methods.
- 65
- CX-ToM: Counterfactual explanations with theory-of-mind for enhancing human trust in image recognition models64
CX-ToM, short for counterfactual explanations with theory-of-mind, is a new explainable AI (XAI) framework for explaining decisions made by a deep convolutional neural network (CNN) that significantly outperforms the state-of theart XAI models.
- Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions64
This work critically examines RefCOCOg, a standard benchmark for this task, using a human study and shows that 83.7% of test instances do not require reasoning on linguistic structure, i.e., words are enough to identify the target object, the word order doesn’t matter.
- Joint Training of Variational Auto-Encoder and Latent Energy-Based Model63
This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM), and shows that the joint training greatly improves the synthesis quality of the VAE.
- 63
- Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance60
A novel "Sense–Learn–Reason–Predict" framework that is able to generate collision- free trajectory predictions in a synthetic dataset designed for collision avoidance evaluation and remains competitive on the commonly used NGSIM US-101 highway dataset.
- ACRE: Abstract Causal REasoning Beyond Covariation60
This work introduces the Abstract Causal REasoning (ACRE) dataset for systematic evaluation of current vision systems in causal induction and notices that pure neural models tend towards an associative strategy under their chance-level performance, whereas neuro-symbolic combinations struggle in backward-blocking reasoning.
- A virtual reality platform for dynamic human-scene interaction60
A system in which Virtual Reality and human / finger pose tracking is integrated to allow agents to interact with virtual environments in real time and how this system can be used for robot task planning and image semantic segmentation is proposed.
- Stochastic jump-diffusion process for computing medial axes in Markov random fields60
It is argued that the stochastic algorithm for computing medial axes is compatible with existing algorithms for image segmentation, such as region growing, snake, and region competition, and provides a new direction for Computing medial axes from texture images.
- Scene Understanding by Reasoning Stability and Safety59
In order to detect unsafe objects in a static scene, the method further infers hidden and situated causes (disturbances) in the scene, and then introduces intuitive physical mechanics to predict possible effects (e.g., falls) as consequences of the disturbances.
- 57
- Learning Energy-Based Spatial-Temporal Generative ConvNets for Dynamic Patterns56
It is shown that an energy-based spatial-temporal generative ConvNet can be used to model and synthesize dynamic patterns and that the learning algorithm can synthesize realistic dynamic patterns.
- Recognizing Unseen Attribute-Object Pair with Generative Model55
This paper proposes a generative model with the encoder-decoder mechanism that bridges visual and linguistic information in a unified end-to-end network that presents the impressive potential to find an intrinsic attribute-object feature representation.
- 54
- Recognizing Car Fluents from Video53
A spatial-temporal And-Or hierarchical model to represent car fluents is formulated under the latent structural SVM framework and outperforms several highly related baseline methods in terms of car fluent recognition and car part localization.
- Parsing Images into Regions, Curves, and Curve Groups53
An algorithm for parsing natural images into middle level vision representations—regions, curves, and curve groups (parallel curves and trees) is presented for an integrated solution to image segmentation and curve grouping through Bayesian inference.
- M 2 Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes52
M2 Diffuser is introduced, a diffusion-based, scene-conditioned generative model that directly generates coordinated and efficient whole-body motion trajectories for mobile manipulation based on robot-centric 3D scans and outperforms state-of-the-art neural planners and successfully transfers the generated trajectories to a real-world robot.
- 52
- Artistic paper-cut of human portraits51
This paper presents a method to render artistic paper-cut of human portraits that integrates bottom-up and top-down cues to better determine the binary values.
- 51
- 50
- Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference50
Noise initialized non-persistent short run MCMC, such as finite step Langevin dynamics initialized from the prior distribution of the latent variables, are used as an approximate inference engine and show that the proposed method outperforms variational auto-encoder (VAE) in terms of reconstruction error and synthesis quality.
- 50
- Learning Sparse FRAME Models for Natural Image Patterns49
The experiments show that the sparse FRAME models are capable of representing a wide variety of object patterns in natural images and that the learned models are useful for object classification.
- From information scaling of natural images to regimes of statistical models49
This work shows that the entropy rate of the image data changes over the viewing distance (as well as the camera resolution), and proposes a full-zoom primal sketch model that integrates both sparse coding and Markov random fields.
- Sequential Manipulation Planning on Scene Graph48
In a series of simulated and real experiments, robots successfully complete complex sequential object rearrangement tasks that are difficult to specify using conventional planning language like Planning Domain Definition Language (PDDL), demonstrating high potential of planning sequential manipulation tasks on cg+.
- Learning Dynamic Generator Model by Alternating Back-Propagation through Time48
This paper shows that the dynamic generator model for spatial-temporal processes such as dynamic textures and action sequences in video data can be learned by an alternating back-propagation through time algorithm that iteratively samples the noise vectors and updates the parameters in the transition model and the generator model.
- Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning47
This work showcases a prototype with algebraic representation for the abstract spatial-temporal reasoning task of Raven's Progressive Matrices (RPM) and presents the ALgebra-Aware Neuro-Semi-Symbolic (ALANS) learner, a hybrid approach to improve systematic generalization in reasoning.
- Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks47
A novel explainable AI (XAI) framework for achieving human-like communication in human-robot collaborations, where the robot builds a hierarchical mind model of the human user and generates explanations of its own mind as a form of communications based on its online Bayesian inference of the user’s mental state is proposed.
- 47
- 47
- CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents46
CivRealm, an environment inspired by the Civilization game, is introduced, providing interfaces for two typical agent types: tensor-based agents that focus on learning, and language-based agents that emphasize reasoning.
- 45
- 44
- 44
- 44
- Learning social affordance grammar from videos: Transferring human interactions to human-robot interactions43
A general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions is presented, and the grammar is transferred to humanoids to enable a real-time motion inference for human-robot interaction (HRI).
- 43
- 43
- Probabilistic Simulation Predicts Human Performance on Viscous Fluid-Pouring Problem.42
It is found that Computational results from a probabilistic simulation model can account for human sensitivity to hidden attributes and their performance on the water-pouring task, and converging evidence supporting mental simulation in physical reasoning is provided.
- 42
- Intent-Aware Multi-Agent Reinforcement Learning41
This paper proposes an intent-aware multi-agent planning framework as well as a learning algorithm that can easily integrate existing intent prediction and low-level planning algorithms and does not suffer from sparse feedbacks in the action space.
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior40
DP-Recon is proposed, which employs diffusion priors in the form of Score Distillation Sampling (SDS) to optimize the neural representation of each individual object under novel views and introduces a visibility-guided approach to dynamically adjust the per-pixel SDS loss weights.
- Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments40
This paper reconstructs an interactive scene using RGB-D data stream, which captures the semantics and geometry of objects and layouts by a 3D volumetric panoptic mapping module, and object affordance and contextual relations by reasoning over physical common sense among objects, organized by a graph-based scene representation.
- Extraction of an Explanatory Graph to Interpret a CNN40
The explanatory graph transferred features of object parts to the task of part localization, and the method significantly outperformed other approaches.
- Inferring human intent from video by sampling hierarchical plans40
A method which allows robots to infer a human's hierarchical intent from partially observed RGBD videos by imagining how the human will behave in the future by using a Bayesian probabilistic programming framework.
- MetaScenes: Towards Automated Replica Creation for Real-world 3D Scans39
MetaScenes is presented, a large-scale simulatable 3D scene dataset constructed from real-world scans, and SCAN2SIM, a robust multi-modal alignment model, which enables the automated, high-quality replacement of assets, thereby eliminating the reliance on artist-driven designs for scaling 3D scenes.
- Jointly Learning Grounded Task Structures from Language Instruction and Visual Demonstration39
The empirical results on a cloth-folding domain have shown that, although state detection through visual processing is full of uncertainties and error prone, by a tight integration with language the agent is able to learn an effective AoG for task representation.
- Customizing painterly rendering styles using stroke processes39
A solution named stroke processes is presented, which enables intuitive and interactive customization of painting styles by mapping perceptual characteristics to rendering parameters, and is capable of reflecting various styles in a single framework.
- Painterly animation using video semantics and feature correspondence39
An interactive system that stylizes an input video into a painterly animation using a rich set of example-based brush strokes and proposes a series of novel teniques to reduce the scintillation effects.
- Layered graph matching by composite cluster sampling with collaborative and competitive interactions39
- Scene Reconstruction with Functional Objects for Robot Autonomy38
This paper addresses the problem of scene reconstruction from an embodied agent’s perspective by reconstructing a functionally equivalent and interactive scene from RGB-D data streams, where the objects within are segmented by a dedicated 3D volumetric panoptic mapping module and subsequently replaced by part-based articulated CAD models to afford finer-grained robot interactions.
- GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning37
A grammar-based dialogue dataset, G RICE, designed to bring impli-cature into pragmatic reasoning in the context of conversations, and shows an overall performance boost in conversational reasoning.
- 37
- A Competence-Aware Curriculum for Visual Concepts Learning via Question Answering36
This work designs a neural-symbolic concept learner for learning the visual concepts and a multi-dimensional Item Response Theory (mIRT) model for guiding the learning process with an adaptive curriculum.
- Structured Attention for Unsupervised Dialogue Structure Induction36
This work proposes to incorporate structured attention layers into a Variational Recurrent Neural Network model with discrete latent states to learn dialogue structure in an unsupervised fashion and shows that on two-party dialogue datasets, VRNN with structured attention learns semantic structures that are similar to templates used to generate this dialogue corpus.
- SMART: A Situation Model for Algebra Story Problems via Attributed Grammar35
This work introduces the concept of a situation model, which originates from psychology studies to represent the mental states of humans in problem-solving, and proposes SMART, which adopts attributed grammar as the representation of situation models for algebra story problems.
- Deformable Generator Networks: Unsupervised Disentanglement of Appearance and Geometry35
An extensive set of qualitative and quantitative experiments shows that the appearance and geometric information can be well disentangled, and the learned geometric generator can be conveniently transferred to other image datasets that share similar structure regularity to facilitate knowledge transfer tasks.
- 35
- Learning Artistic Lighting Template from Portrait Photographs35
This paper presents a method for learning artistic portrait lighting template from a dataset of artistic and daily portrait photographs, which adopts Haar-like local lighting contrast features and is selected to form a log-linear model using a stepwise feature pursuit algorithm.
- 35
- Efficient Task Planning for Mobile Manipulation: a Virtual Kinematic Chain Perspective34
A Virtual Kinematic Chain perspective, a simple yet effective method, to improve task planning efficacy for mobile manipulation by consolidating the kinematics of the mobile base, the arm, and the object being manipulated collectively as a whole, naturally defines abstract actions and eliminates unnecessary predicates in describing intermediate poses.
- 34
- Unsupervised Structure Learning of Stochastic And-Or Grammars34
This work presents a unified formalization of stochastic And-Or grammars that is agnostic to the type of the data being modeled, and proposes an unsupervised approach to learning the structures as well as the parameters of such Grammars.
- RulE: Knowledge Graph Reasoning with Rule Embedding33
A novel and principled framework called RulE is proposed to effectively leverage logical rules to enhance KG reasoning, and outperforms the majority of existing embedding-based and rule-based approaches.
- Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning33
A dataset, Machine Number Sense (MNS), consisting of visual arithmetic problems automatically generated using a grammar model—And-Or Graph (AOG), called for attention in fusing the classic search-based algorithms with modern neural networks to discover the essential number concepts in future research.
- 33
- Primal Sketch: Integrating Texture and Structure33
The primal sketch model is not only a parsimonious image representation for lossy image coding, but also provides a meaningful mid-level generic representation for other vision tasks.
- Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations32
Ag2Manip (Agent-Agnostic representations for Manipulation), a framework aimed at addressing challenges through two key innovations: an agent-agnostic visual representation derived from human manipulation videos, with the specifics of embodiments obscured to enhance generalizability, demonstrates its effectiveness and generalizability across both simulated and real environments.
- CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization32
This paper proposes a semi-automatic framework for generating disentangled shifts by introducing a controllable visual question-answer generation (VQAG) module that is capable of generating highly-relevant and diverse question- answer pairs with the desired dataset style.
- MetaStyle: Three-Way Trade-off among Speed, Flexibility, and Quality in Neural Style Transfer32
A novel method is proposed, coined MetaStyle, which formulates the neural style transfer as a bilevel optimization problem and combines learning with only a few post-processing update steps to adapt to a fast approximation model with satisfying artistic effects, comparable to the optimization-based methods for an arbitrary style.
- 32
- Image parsing: segmentation, detection, and recognition32
This method makes use of bottom-up proposals combined with top-down generative models using the Data Driven Markov Chain Monte Carlo (DDMCMC) algorithm which is guaranteed to converge to the optimal estimate asymptotically.
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis31
Beacon3D is proposed, a benchmark for 3D-VL grounding and QA tasks, delivering a perspective shift in the evaluation of 3D-VL understanding, and evaluation of state-of-the-art 3D-VL models on Beacon3D reveals that object-centric evaluation elicits true model performance and particularly weak generalization in QA.
- Theory-Based Causal Transfer:Integrating Instance-Level Induction and Abstract-Level Structure Learning31
This paper adopts a Bayesian perspective of causal theory induction and uses these theories to transfer knowledge between environments, and compares performances against a set of predominate model-free reinforcement learning (RL) algorithms.
- 31
- 31
- 30
- 30
- 29
- Human-Robot Interaction in a Shared Augmented Reality Workspace29
A new shared Augmented Reality workspace for Human-Robot Interaction is designed and developed, which establishes a bi-directional communication between human agents and robots and allows a robot to proactively interact with human agents, instead of purely passively executing received commands.
- VRGym29
It is demonstrated that VRGym is able to collect human interactions and fine manipulations, accommodate various robots with a ROS bridge, support experiments for human-robot interaction, and provide toolkits for training the state-of-the-art machine learning algorithms.
- On Learning Non-Convergent Short-Run MCMC Toward Energy-Based Model.29
It is shown that the learned short-run MCMC is capable of generating realistic images, unlike traditional EBM or MCMC, and is also capable of reconstructing observed images and interpolating different images, like generator model or flow model.
- 29
- Learning and parsing video events with goal and intent prediction29
An unsupervised learning algorithm to learn the atomic actions, the temporal relations and the And-Or nodes under the information projection principle in a coherent probabilistic framework and an event parsing algorithm based on the T-AOG which can understand events, infer the goal of agents, and predict their plausible intended actions.
- VLGrammar: Grounded Grammar Induction of Vision and Language28
This work presents VLGrammar, a method that uses compound probabilistic context-free grammars (compound PCFGs) to induce the language grammar and the image grammar simultaneously, and proposes a novel contrastive learning framework to guide the joint learning of both modules.
- Consolidating Kinematic Models to Promote Coordinated Mobile Manipulations28
A Virtual Kinematic Chain is constructed that readily consolidates the kinematics of the mobile base, the arm, and the object to be manipulated in mobile manipulations, and results manifest that VKC-based joint modeling and planning promote task success rates and produce more efficient trajectories.
- Learning Triadic Belief Dynamics in Nonverbal Communication from Videos28
Using a social account to represent, model, learn, and infer agents’ mental states from pure visual inputs provides a better video summary on videos with rich social interactions compared with state-of-the-art keyframe video summary methods.
- Mirroring without Overimitation: Learning Functionally Equivalent Manipulation Actions27
The proposed mirroring approach, inspired by the neuroscience discovery of the mirror neurons, to transfer demonstrated manipulation actions to robots is demonstrated by teaching a real Baxter robot with a complex manipulation task involving haptic feedback—opening medicine bottles.
- Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real World26
Bongard-OpenWorld is introduced, a new benchmark for evaluating real-world few-shot reasoning for machine vision and to which extent the recently introduced Large Language Models (LLMs) and Vision-Language Models (VLMs) can solve the task, by directly probing VLMs, and combining VLMs and LLMs in an interactive reasoning scheme.
- Mining Object Parts from CNNs via Active Question-Answering26
This paper uses active question-answering to semanticize neural patterns in conv-layers of the CNN and mine part concepts to construct an And-Or graph (AOG) to represent a four-layer semantic hierarchy of the part.
- Perceptual Scale-Space and Its Applications26
A perceptual scale-space theory which differs from the traditional image scale- space theory in two aspects, and it is shown that the sketch pyramid is a more parsimonious representation than a multi-resolution Gaussian/Wavelet pyramid.
- 25
- 25
- 25
- Artificial Social Intelligence: A Comparative and Holistic View24
This article begins with a review of ASI from a cognitive science standpoint, including social perception, theory of mind (ToM), and social interaction, and the recently-emerged computational counterpart in the AI community.
- Learning to infer human attention in daily activities24
To infer human attention, a deep neural network model is proposed that fuses both low-level human pose cue and high-level task encoding cue and defines human attention as attentional objects that coincide with the task the human is doing.
- Scene-Centric Joint Parsing of Cross-View Videos24
This paper introduces a joint parsing framework that integrates view-centric proposals into scene-centric parse graphs that represent a coherentscene-centric understanding of cross-view scenes.
- Using Causal Induction in Humans to Learn and Infer Causality from Video24
A Bayesian grammar model for human-perceived causal relationships that is learnable from video and methods for learning the model are described by using co-occurrence to identify potential causal relationships between events and ap- plying the heuristics listed above.
- 24
- 24
- Bayesian Inference for Layer Representation with Mixed Markov Random Field24
This paper presents a Bayesian inference algorithm for image layer representation [26], 2.1D sketch [6], with mixed Markov random field, based on Swendsen-Wang Cut algorithm and satisfactory results are shown in the experiments.
- Generalizing Swendsen–Wang for Image Analysis24
A general cluster algorithm that extends the Swendsen–Wang method to general Bayesian inference on graphs and is at least two orders of magnitude faster (in CPU time) than the single-site Gibbs sampler.
- Asymptotically Admissible Texture Synthesis24
A theoretical framework for designing and analyzing texture sampling algorithms and a novel texture synthesis algorithm which samples from a subset of the Julesz ensemble by pasting texture patches from the sample texture.
- Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns23
This model defines a notion of intrackability by the separation of warped component and residual component in each image frame, and can synthesize realistic dynamic pattern, and disentangling appearance, trackable and intrackable motions in unsupervised setting.
- 23
- Perception of Human Interaction Based on Motion Trajectories: From Aerial Videos to Decontextualized Animations23
A hierarchical model was developed that provides a good fit to human judgments and can also be generalized to the original Heider-Simmel animations, providing a viable tool for studying animacy and social perception.
- Inducing wavelets into random fields via generative boosting23
This paper proposes a learning algorithm for the random field models whose energy functions are in the form of linear combinations of rectified filter responses from subsets of wavelets selected from a given over-complete dictionary and demonstrates that the models can be learned in an unsupervised setting.
- Unsupervised learning of compositional sparse code for natural image representation23
Experiments show that the proposed unsupervised learning method is capable of learning meaningful compositional sparse code, and the learned templates are useful for image classification.
- Hierarchical Space Tiling for Scene Modeling23
An efficient method for learning hierarchical compositional models from a set of scene configurations that is not only fully generative but also yields discriminative scene classification performance which outperforms the state-of-the-art methods.
- An Automatic Portrait System Based on And-Or Graph Representation23
An automatic human portrait system based on the And-Or graph representation that explains the hierarchical structure and variability of portrait and applies it into practice, which can simultaneously generate a set of impressive portraits in different styles.
- Towards Socially Intelligent Agents with Mental State Transition and Human Value22
A hybrid mental state parser that extracts information from both the dialogue and event observations and maintains a graphical representation of the agent’s mind and a transformer-based value model that learns human preferences from the human value dataset, ValueNet.
- Joint Inference of States, Robot Knowledge, and Human (False-)Beliefs22
This paper proposes to adopt a graphical model to unify the representation of object states, robot knowledge, and human (false-)beliefs, and achieves better cross-view accuracy on a challenging small object tracking dataset.
- Learning Knowledge-guided Pose Grammar Machine for 3D Human Pose Estimation.22
A knowledge-guided pose grammar network which efficiently captures pose-aligned features and a hierarchy of Bidirectional RNNs on top of it to explicitly incorporate a set of knowledge and thus enforce high-level constraints over human poses is proposed.
- Evaluating Human Cognition of Containing Relations with Physical Simulation.22
The physical simulation is a good approximation to the human cognition of container and containing relations with physical simulation as well as to human judgments with respect to results of physical simulation under different scenarios.
- 22
- Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuning21
This paper proposes an iterative tool usage exploration method for multimodal agents without any pre-collected data, namely SPORT, via step-wise preference optimization to refine the trajectories of tool usage to eliminate the bottleneck of human annotation.
- 21
- SocAoG: Incremental Graph Parsing for Social Relation Inference in Dialogues21
This work model the social network as an And-or Graph, named SocAoG, for the consistency of relations among a group and leveraging attributes as inference cues, and proposes an incremental structure prediction task.
- Sparse Winograd Convolutional Neural Networks on Small-scale Systolic Arrays21
A framework on FPGA is implemented by combining the sparse Winograd convolution, clusters of small-scale systolic arrays, and a tailored recursive Z-Morton memory layout design, which achieves very high computation resource utilization, 20x~30x energy efficiency, and more than 5x speedup compared with the dense implementation.
- Human Causal Transfer: Challenges for Deep Reinforcement Learning.21
This study developed a novel OpenLock task, in which participants explored a virtual “escape room” environment by moving levers that served as “locks” to open a door, and found that a standard deep reinforcement learning model is unable to capture the causal abstraction presented between trials with the same causal schema and trials with a transfer of causal schema.
- 21
- CO3 for ultra-fast and accurate interactive segmentation21
This paper presents an interactive image segmentation framework which is ultra-fast and accurate, outperforms those established approaches in both accuracy and efficiency by a large margin and achieves state-of-the-art results.
- Learning explicit and implicit visual manifolds by information projection21
An information theoretical learning framework is presented that derives common models for these manifolds through information projection, and a manifold pursuit algorithm is studied that clusters image patches into those atomic subspaces and ranks them according to their information gains.
- 20
- 19
- Consistent Probabilistic Simulation Underlying Human Judgment in Substance Dynamics.19
The results reported herein expand on previous work proposing human use of mental simulation in physical reasoning and demonstrate human proficiency in predicting the dynamics of sand, a substance that is less common in daily life than liquid or rigid objects.
- 19
- 19
- Abstract painting with interactive control of perceptual entropy19
A framework for generating abstract art from photographs by measuring the level of perceptual ambiguity using entropy, as it measures uncertainty levels in information theory, and proposing a painterly rendering method with interactive control of the ambiguity levels.
- 19
- VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video18
VideoArtGS enables practical digital twin creation from monocular video, establishing a new benchmark for video-based articulated object reconstruction and proposing a motion prior guidance pipeline that analyzes 3D tracks, filters noise, and provides reliable initialization of articulation parameters.
- 18
- Representation Learning: A Statistical Perspective18
Recent advances in learning representations from a statistical perspective are reviewed, including unsupervised learning of vector representations and learning of both vector and matrix representations.
- Tracking Occluded Objects and Recovering Incomplete Trajectories by Reasoning About Containment Relations and Human Actions18
The proposed algorithm formulates tracking problem as a network flow representation encoding containment relations and their changes, and demonstrates better performance on tracking occluded objects compared with baseline methods.
- A tale of three probabilistic families: Discriminative, descriptive, and generative models18
Three families of probability models, namely, the discriminative models, the descriptive models, and the generative models are reviewed, which take advantage of the high approximation capacities of deep neural networks.
- 18
- 18
- Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC17
The model has a particularly simple form in the space of the latent variables of the flow-based model, and MCMC sampling of the EBM in the latent space, which is a simple special case of neural transport MCMC, mixes well and traverses modes in the data space.
- Inferring Human Attention by Learning Latent Intentions17
Given an RGB-D video with 3D human skeletons, a joint-state dynamic programming algorithm is utilized to jointly infer the latent intentions, the 3D attention directions, and the attention voxels in scene point clouds.
- 17
- Background modeling by subspace learning on spatio-temporal patches17
This paper presents a novel background model for video surveillance-Spatio-Temporal Patch based Background Modeling (STPBM), which uses spatio-temporal patches, called bricks, to characterize both the appearance and motion information.
- 17
- 17
- On the Emergence of Symmetrical Reality16
The symmetrical reality framework is introduced, which offers a unified representation encompassing various forms of physical-virtual amalgamations that enables researchers to better comprehend how AI agents can collaborate with humans and how distinct technical pathways of physical-virtual integration can be consolidated from a broader perspective.
- MindDial: Enhancing Conversational Agents with Theory-of-Mind for Common Ground Alignment and Negotiation16
MindDial is proposed, a novel conversational framework that can generate situated free-form responses to align and negotiate common ground and shows that models with mind modeling can generate more human-like responses when aligning and negotiating common ground.
- Neural-Symbolic Recursive Machine for Systematic Generalization16
The findings demonstrate that NSR's design, imbued with the inductive biases of equivariance and compositionality, grants it the expressiveness to adeptly handle diverse sequence-to-sequence tasks and achieve unparalleled systematic generalization.
- 16
- 16
- 16
- 16
- 16
- 15
- 14
- Human-level few-shot concept induction through minimax entropy learning14
This work introduces a computational model designed to emulate human inductive reasoning on abstract reasoning tasks, such as those in IQ tests, using a minimax entropy approach, and demonstrates the potential of minimax entropy learning for enabling machines to learn relational concepts efficiently with minimal input.
- Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation14
A generative framework, in which the probability distribution of each domain is represented by a generative cooperative network that consists of an energy-based model and a latent variable model, which is useful for unsupervised image-to-image translation and unpaired image sequence translation.
- Mining Interpretable AOG Representations From Convolutional Networks via Active Question Answering14
A question-answering method that uses active human-computer communications to mine patterns from a pre-trained CNN, in order to incrementally explain more features in conv-layers, and achieves similar or better part-localization performance than fast-RCNN methods.
- Congestion-aware Evacuation Routing using Augmented Reality Devices14
A congestion-aware routing solution for indoor evacuation is presented, which produces real-time individual-customized evacuation routes among multiple destinations while keeping tracks of all evacuees’ locations and generates a more time-efficient evacuation route for each individual that minimizes the overall congestion.
- 14
- Decomposing Human Causal Learning: Bottom-up Associative Learning and Top-down Schema Reasoning.14
This paper adopts a Bayesian framework to model causal theory induction and uses the inferred causal theory to transfer abstract knowledge between similar environments, and trains a simulated agent to discover and transfer useful relational and abstract knowledge.
- 14
- 14
- Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning13
HOP is a novel multi-agent decision-making algorithm that enables few-shot adaptation to unseen policies in mixed-motive environments and improves efficiency by updating beliefs about others' goals both across and within episodes and by using information from the opponent modeling module to guide planning.
- Fast Peer Adaptation with Context-aware Exploration13
A peer identification reward is proposed, which rewards the learning agent based on how well it can identify the behavior pattern of the peer over the historical context, such as the observation over multiple episodes, and induces more active exploration behavior, achieving faster adaptation and better outcomes than existing methods.
- Rearrange Indoor Scenes for Human-Robot Co-Activity13
The rearrangement aims to afford sufficient accessible space for robot activities without compromising everyday human activities, and the quality of the rearranged scenes is qualitatively validated by a human study, indicating the efficacy of the proposed strategy.
- Part-level Scene Reconstruction Affords Robot Interaction13
This work introduces a part-level reconstruction approach that reassembles objects using primitive shapes to precisely replicate the observed physical scenes and simulate robot interactions with both rigid and articulated objects.
- 13
- 13
- 13
- 13
- Learning Active Basis Models by EM-Type Algorithms13
This review article explains that EM algorithm is a natural computational scheme for learning image templates of object categories where the learning is not fully supervised, and iterates between recognition and supervised learning.
- 12
- Mind the Context: The Impact of Contextualization in Neural Module Networks for Grounding Visual Referring Expressions12
This work substantially reduces the number of modules in NMN by up to 75% by parameterizing the module arguments, and proposes a method to contextualize the parameterized model to enhance the module’s capacity in exploiting the visiolinguistic associations.
- 12
- 12
- 12
- Intrackability: Characterizing Video Statistics and Pursuing Video Representations12
This paper proposes to measure the video complexity by the entropy of the posterior probability, called the intrackability, to characterize the video statistics and pursue optimal video representations and develops a simple criterion for model transition and for pursuing the optimal video representation.
- G4Splat: Geometry-Guided Gaussian Splatting with Generative Prior11
This paper proposes to leverage the prevalence of planar structures to derive accurate metric-scale depth maps, providing reliable supervision in both observed and unobserved regions, and incorporates this geometry guidance throughout the generative pipeline to improve visibility mask estimation, guide novel view selection, and enhance multi-view consistency when inpainting with video diffusion models, resulting in accurate and consistent scene completion.
- Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models11
This perspective paper comprehensively review existing evaluations of Large Language Models using both standardized tests and ability-oriented benchmarks to highlight the missing pieces in artificial general intelligence, that is, the unity of knowing and acting.
- A unified psychological space for human perception of physical and social events11
It is demonstrated that a theoretical unification based on physical potential functions and goal-related values can account for the human ability to form an immediate impression of physical and social events and provides an important pathway from perception to higher cognition.
- Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis11
This work proposes an approach to learn neural representation of camera poses and 3D scenes, coupled with neural representations of local camera movements, represented as vectors and the local camera movement is represented as a matrix operating on the vector of the camera pose.
- 11
- 11
- Visual Discourse Parsing11
This work proposes an approach to identify discourse cues from the videos without the need to explicitly identify and annotate the scenes, and presents a novel dataset containing 310 videos and the corresponding discourse cues to evaluate the approach.
- Explainable AI as Collaborative Task Solving11
Compared to the most popularly used attribution based explanations (saliency maps), the new framework X-ToM significantly improves human trust in the underlying vision system and quantitatively evaluates human’s trust of machine behaviors.
- Building a telescope to look into high-dimensional image spaces11
This work presents novel computational experiments that map and visualize the local mode structure of Gibbs densities, and proposes a new MCMC tool called Attraction-Diffusion (AD) that can capture the macroscopic structure of highly non-convex densities by measuring metastability of local modes.
- 11
- 11
- 10
- Synthesizing Dynamic Textures and Sounds by Spatial-Temporal Generative ConvNet.10
This paper studies the problem of modeling and synthesizing dynamic textures using a generative version of the convolution neural network (ConvNet or CNN) that consists of multiple layers of spatial-temporal filters to capture the spatial-Temporal patterns in the dynamic textures and shows that such spatial- Temporal generative ConvNet can synthesize realistic dynamic textures.
- 10
- 10
- 10
- 10
- 9
- 9
- 9
- 9
- VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks9
This work design and implement a virtual reality system, VRKitchen, with integrated functions which enable embodied agents to perform real life cooking tasks involving a wide range of object manipulations and state changes, and allow human teachers to provide demonstrations for training agents.
- 9
- 9
- A Unified Framework for Human-Robot Knowledge Transfer.9
A robot system that not only learns in real-time from human demonstrations, but also transfers this knowledge in human-robot interactions is implemented.
- 9
- 9
- 9
- Mathematical modeling of clutter: descriptive vs. generative models9
This article presents two mathematical paradigms for clutter modeling, and discusses a texton model for clutter and effective Markov chain Monte Carlo methods for stochastic inference and the deep relationship between the two modeling paradigm.
- MusicAOG: An Energy-Based Model for Learning and Sampling a Hierarchical Representation of Symbolic Music8
A novel symbolic representation is introduced that amalgamates both explicit and implicit musical information across diverse traditions and granularities and serves as the foundation for an energy-based model, uniquely tailored to learn musical concepts through a flexible algorithm framework relying on the minimax entropy principle.
- 8
- Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration8
This work proposes an approach to calibrate humans’ estimate of a robot’s reachable workspace through a small number of demonstrations before collaboration, and develops a novel motion planning method, reachability-expressive motion planning (REMP), which jointly optimizes the physical cost and the expressiveness of robot motion to reveal the robot�'s reachability to a human observer.
- 8
- Represent and Infer Human Theory of Mind for Human-Robot Interaction.8
To infer a human’s mental state – intent and belief – from an observed RGBD video for human-robot interaction is proposed, to integrate symbolic reasoning, a well-studied approach within A.I. domains, with the uncertainty native to computer vision strategies.
- Heterogeneous Value Alignment Evaluation for Large Language Models7
This paper proposes a Heterogeneous Value Alignment Evaluation (HVAE) system, designed to assess the success of aligning LLMs with heterogeneous values, and brings the Social Value Orientation framework from social psychology to examine the behavior of these LLMs.
- Hamiltonian and Langevin Monte Carlo7
L Langevin Monte Carlo is a special case of HMC that is widely used in Deep Learning applications and uses Hamilton’s Equations to simulate the time evolution of a physical system with potential energy U.
- Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Multi-Modal Conditional Learning7
The proposed method to learn the two models jointly jointly, where the fast thinking initializer serves to initialize the sampling of the slow thinking solver, and the solver refines the initial output by an iterative algorithm, is demonstrated.
- 7
- 7
- Visuomotor Adaptation and Sensory Recalibration in Reversed Hand Movement Task.7
It was found that participants who preferred distal-smooth movements produced more linear trajectories than those who preferred proximal-awkward movements, which appears consistent with linear motor movement planning guided by increased attention to visual feedback.
- Video Primal Sketch: A Unified Middle-Level Representation for Video7
A middle-level video representation named video primal sketch (VPS), which integrates two regimes of models: sparse coding model using static or moving primitives to explicitly represent moving corners, lines, feature points, etc., and FRAME /MRF model reproducing feature statistics extracted from input video to implicitly represent textured motion.
- Mapping the Energy Landscape of Non-convex Optimization Problems7
The utility of ELMs in analyzing non-convex energy minimization problems with two case studies: clustering with Gaussian mixture models and learning mixtures of Bernoulli templates from images are demonstrated.
- 7
- Video Echoed in Music: Semantic, Temporal, and Rhythmic Alignment for Video-to-Music Generation6
Video Echoed in Music is proposed, a latent music diffusion that generates high-quality soundtracks with semantic, temporal, and rhythmic alignment for input videos, and experimental results demonstrate superiority, particularly in semantic relevance and rhythmic precision.
- On the Complexity of Bayesian Generalization6
Experimental results show that representations with relatively high subjective complexity outperform those with relatively low subjective complexity in the rule- based generalization, while the trend is the opposite in the similarity-based generalization.
- 6
- 6
- 6
- Active Basis for Modeling, Learning and Recognizing Deformable Templates6
An active basis model, a shared sketch algorithm, and a computational architecture of sum-max maps for representing, learning, and recognizing deformable templates are proposed.
- 5
- M${}^{3}$Bench: Benchmarking Whole-Body M otion Generation for M obile M anipulation in 3D Scenes5
The proposed Bench features 30,000 object rearrangement tasks across 119 diverse scenes, providing expert demonstrations generated by the newly developed BenchMaker, an automatic data generation tool that produces whole-body motion trajectories from high-level task instructions using only basic scene and robot information.
- 5
- 5
- Learning from the Tangram to Solve Mini Visual Tasks5
This work presents the Tangram dataset, a game that requires replicating an abstract pattern from seven dissected shapes, and shows that a pre-trained neural model on theTangram helps solve some mini visual tasks based on low-resolution vision.
- VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse5
NDOOR IT is introduced : a built-in toolkit for N VIDIA that providesible pipelines for indoor scene building, scene randomizing, and animation controls and assists researchers in creating real-time training and controlling for avatars and robotics.
- 5
- 5
- Partitioning the Perception of Physical and Social Events Within a Unified Psychological Space.5
A unified psychological space is proposed that reveals the partition between the perceptions of physical events involving inanimate objects and the perception of social events involving human interactions with other agents and a physics engine and a deep reinforcement learning model are adopted to synthesize a rich set of motion patterns.
- Learning Human Utility from Video Demonstrations for Deductive Planning in Robotics5
In these experiments, a robot learns how to fold t-shirts from visual demonstrations, and proposes a plan (by answering why, how, and what) when folding never-beforeseen articles of clothing.
- Critical Features of Joint Actions that Signal Human Interaction.5
The visual perception of joint actions, in which two individuals coordinate their body movements in space and time to achieve a joint goal, is examined to enable rapid detection of meaningful inter-personal interactions in complex scenes.
- Learning And-Or Models to Represent Context and Occlusion for Car Detection and Viewpoint Estimation5
- First Hitting Time Analysis of the Independence Metropolis Sampler5
This paper studies a special case of the Metropolis algorithm, the Independence Metropolis Sampler (IMS), in the finite state space case and shows how some non-independence Metropolis–Hastings algorithms can perform better than the IMS and deriving general lower and upper bounds for the mean first hitting times of a Metropolis-HASTings algorithm.
- 4
- 4
- Learning V1 Simple Cells with Vector Representation of Local Content and Matrix Representation of Local Motion4
A representational model for image pairs such as consecutive video frames that are related by local pixel displacements that can learn Gabor-like filter pairs of quadrature phases and achieve competitive results on optical flow estimation is proposed.
- 4
- 4
- 4
- 4
- A Representational Model of Grid Cells Based on Matrix Lie Algebras.4
By reconstructing the kernel functions for place cells, the model learns hexagon grid patterns that characterize the grid cells, and the learned model is capable of near perfect path integral, and it is also capable of error correction.
- Multimodal Conditional Learning with Fast Thinking Policy-like Model and Slow Thinking Planner-like Model.4
This paper solves the supervised learning of the conditional distribution of a high-dimensional output given an input, where the output and input belong to two different modalities, e.g., the output is an image and the input is a sketch by learning two models that bear similarities to those in reinforcement learning and optimal control.
- Learning Grid-like Units with Vector Representation of Self-Position and Matrix Representation of Self-Motion.4
A model for learning grid-like units for spatial awareness and navigation by minimizing a loss function that combines the loss functions of the two sub-models, which can be used for path integral and path planning.
- 4
- Exploring generative perspective of convolutional neural networks by learning random field models4
This paper proposes to learn Markov random field models called FRAME (Filters, Random field, And Maximum Entropy) models using the highly sophisticated filters pre-learned by the ConvNet on the big ImageNet dataset.
- Learning Generative ConvNet with Continuous Latent Factors by Alternating Back-Propagation.4
This paper shows that the unsupervised learning of a popular top-down generative ConvNet model with latent continuous factors can be accomplished by a learning algorithm that consists of alternatively performing back-propagation on both the latent factors and the parameters.
- Learning FRAME Models Using CNN Filters for Knowledge Visualization4
This paper proposes to learn the generative FRAME (Filters, Random field, And Maximum Entropy) model using the highly expressive filters pre-learned by the CNN at the convolutional layers, and explains how this model corresponds to a CNN unit at a layer above the layer of filters employed by the model.
- 4
- Dynamic Feature Cascade for Multiple Object Tracking with Trackability Analysis4
Based on the analysis of trackability and computational complexity of the features under various occlusion conditions, a dynamic feature method cascade is presented to match the objects in consecutive frames to measure how well given features can be used to find the correspondence in videos with multiple objects.
- Cluster Sampling and Its Applications in Image Analysis4
A general cluster sampling method which extends the SW algorithm to general Bayesian inference on graphs and presents a generalized Gibbs sampler which samples the color of a cluster according to a conditional probability weighted by a product of edge probabilities.
- How Do Heuristics Expedite Markov Chain Search? Hitting-time Analysis of the Independence Metropolis Sampler4
The results show, though in humble cases, that one can indeed reach optimal solutions in very few steps with good proposal probabilities regardless of the size of the original search space, which is different from previous analysis on the Markov chain convergence rate.
- 3
- Mapping the Energy Landscape3
This chapter introduces a novel MCMC method for identifying macroscopic structures in locally noisy energy landscapes and explores and visualizes the model space for clustering, bi-clustering, and grammar learning.
- 3
- 3
- Cluster Sampling Methods3
The Swendsen-Wang algorithm has been initially designed for addressing the critical slowing down in sampling the Ising and Potts models described below at or near the critical temperature where phase transitions occur.
- 3
- 3
- Learning vector representation of local content and matrix representation of local motion, with implications for V13
A representational model for image pair such as consecutive video frames that are related by local pixel displacements that can learn to infer local motions and can learn Gabor-like filter pairs of quadrature phases is proposed.
- Mining deep And-Or object structures via cost-sensitive question-answer-based active annotations3
A cost-sensitive active Question-Answering (QA) framework for learning a nine-layer And-Or graph (AOG) from web images that iteratively selects different storylines of questions to update different nodes in the AOG.
- Learning Multi-grid Generative ConvNets by Minimal Contrastive Divergence.3
It is shown that this method can learn realistic energy-based generative ConvNet models, and it outperforms the original contrastive divergence (CD) and persistent CD.
- Single-Image 3D Scene Parsing Using Geometric Commonsense3
A novel commonsense reasoning framework capable of reconstructing a variety of scene types from a single input image and a data-driven Monte Carlo method to infer the optimal solution with both bottom-to-up and top-down computations is presented.
- 3
- 3
- Statistical Principles in Image Modeling3
This review article describes three statistical principles for modeling image patterns: the sparse coding principle, the minimax entropy principle, and the meaningful alignment principle, which correspond to three regimes of composition patterns of Gabor wavelets.
- Object Category Recognition Using Generative Template Boosting3
This framework integrates the learnable And-Or graph model, a hierarchical structure that combines the reconfigurability of a stochastic context free grammar with the constraints of a Markov random field, and samples object configurations as training templates from this generative model.
- 2
- SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning2
This work proposes a novel gradient-based state representation for multi-agent reinforcement learning, and introduces a data-driven method, where it employs denoising score matching to learn the social gradient fields (SocialGFs) from offline samples.
- 2
- 2
- 2
- 2
- Data Driven Markov Chain Monte Carlo2
Data Driven Markov chain Monte Carlo (DDMCMC) provides a principled way to use low-level information from processes such as edge detection and clustering to guide the MCMC search by making informed jumps in the solution space, achieving significant speed-ups in convergence to the modes of the posterior probability.
- 2
- 2
- Interpretable CNNs2
A generic method to learn interpretable convolutional convolutional filters in a deep convolutional neural network, where each interpretable convolutional encodes features of a specific object part.
- Learning Deep Generative Models with Short Run Inference Dynamics2
This paper proposes to use short run inference dynamics guided by the log-posterior, such as finite-step gradient descent algorithm initialized from the prior distribution of the latent variables, as an approximate sampler of the posterior distribution, where the step size of the gradient descent dynamics is optimized by minimizing the Kullback-Leibler divergence.
- 2
- 2
- 2
- Integration and goal-guided scheduling of bottom-up and top-down computing processes in hierarchical models2
A framework of integrating and scheduling bottom-up and top-down computing processes in a recursively defined hierarchical And-Or graph (AoG) to address the vision-goal-guided trade-off between accuracy performance and computational efficiency in both loss cost-insensitive and cost-sensitive situations is presented.
- Cluster sampling and its applications to segmentation, stereo and motion2
A new stochastic algorithm capable of sampling arbitrary energy functions defined on graph partitions is presented and is applied to 4 important problems in computer vision: image segmentation, perceptual organization, stereo matching and motion segmentation.
- v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound1
This work introduces v-HUB, a novel video humor understanding benchmark, and evaluates a diverse set of multimodal large language models (MLLMs) that can natively process audio, covering both open-source and proprietary domains.
- 1
- 1
- 1
- Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models1
Aligner, a novel Parameter-Efficient Fine-Tuning (PEFT) method for aligning multi-billion-parameter-sized Large Language Models (LLMs), can still perform comparably well to state-of-the-art LLM adaptation methods like LoRA that require millions of parameters.
- 1
- CORE: Common Random Reconstruction for Distributed Optimization with Provable Low Communication Complexity1
This paper proposes a new technique named Common randOm REconstruction (CORE), which can be used to compress the information transmitted between machines in order to reduce communication complexity without other strict conditions.
- Triangular Character Animation Sampling with Motion, Emotion, and Relation1
This paper presents a novel energy-based framework to sample and synthesize animations by associating the characters' body motions, facial expressions, and social relations with a Spatial-Temporal And-Or graph (ST-AOG), a stochastic grammar model, to encode the contextual relationship between motion, emotion, and relation.
- 1
- 1
- 1
- A Causal And-Or Graph Model for Visibility Fluent Reasoning in Human-Object Interactions.1
A Causal And-Or Graph is introduced to represent the causal-effect relations between an object's visibility fluents and its activities, and a probabilistic graph model is developed to jointly reason the visibility fluent change and track humans in videos.
- Multi-Shot Mining Semantic Part Concepts in CNNs.1
A new learning strategy that incrementally embeds new object-part concepts into a pre-trained convolutional neural network in order to explore explicit semantics for the CNN units and gradually transfer the pre- trained CNN into a “white-box” model for hierarchical object understanding.
- 1
- 1
- A mathematical theory of textons and primal sketch: integrating generative and descriptive methods1
This dissertation is to study a mathematical framework for the two concepts of textons and primal sketches to build a theoretical foundation for the early vision theory and leads to generative image models for generic natural images and computing algorithms for learning, simulation, and inference.
- Stochastic Graph Partition: Generalizing the Swendsen-Wang Method1
This paper generalizes Swendsen-Wang (1987) - a well celebrated algorithm in statistical mechanics- for general graph partition and designs reversible Markov chain moves in the space of all possible partitions to search for global optimum in the Bayesian framework.
- Effective Bayesian inference by data-driven Markov chain Monte Carlo for object recognition and image segmentation1
The DDMCMC paradigm utilizes data driven techniques, such as Hough transform, edge detection, and color clustering, to design effective transition probabilities for Markov chain dynamics, which drastically improves the effectiveness of traditional MCMC algorithms in terms of two standard metrics: 'burn-in' period and 'mixing' rate.
- Modeling Decision-Making with Will for Cooperation in Social Dilemmas–
It is found that heterogeneous will strength promotes cooperation, and that agents who autonomously suspend rational re-evaluation can significantly outperform continuous optimizers, suggesting that successful cooperation relies on the cognitive capacity to strategically constrain calculation.
- Discrete Markov Bridge–
This work proposes Discrete Markov Bridge, a novel framework specifically designed for discrete representation learning, built upon two key components: Matrix Learning and Score Learning and conducts a rigorous theoretical analysis, establishing formal performance guarantees for Matrix Learning and proving the convergence of the overall framework.
- –
- Discovering "Words" in Music: Unsupervised Learning of Compositional Sparse Code for Symbolic Music–
An unsupervised machine learning algorithm that identifies recurring patterns -- referred to as ``music-words''-- from symbolic music data and indicates that minimizing code length effectively addresses semantic ambiguity, suggesting that human optimization of encoding systems shapes musical semantics.
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling–
This work presents strategies to learn an Energy-Based Model (EBM) according to the desired length of its MCMC sampling trajectories and introduces three novel methods of MCMC initialization for negative samples used in Maximum Likelihood (ML) learning.
- –
- –
- –
- –
- –
- –
- –
- Convergence Analysis of MCMC–
In this chapter, some of the most useful bounds on the convergence rate of νKn are derived and implemented through the application of randomly shuffling a deck of cards.
- –
- –
- –
- Learning with Stochastic Gradient–
In the first half of the chapter, several theorems concerning the approximation of the true gradient from per-observation gradients are presented, and an important connection between Stochastic Gradient and Langevin Dynamics is discussed.
- Metropolis Methods and Variants–
In this chapter several variations of the original algorithm are discussed and the concepts of reversible jumps and diffusion are covered.
- Gibbs Sampler and Its Variants–
The Gibbs sampler, first created by the Geman brothers Donald and Stewart, is an MCMC algorithm for obtaining samples from distributions that are difficult to sample.
- –
- –
- –
- Towards Interpretable Image Synthesis by Learning Sparsely Connected AND-OR Networks.–
The results show that meaningful and interpretable hierarchical representations are learned with better qualities of image synthesis and reconstruction obtained than state-of-the-art baselines.
- –
- –
- Inferring Human Interaction from Motion Trajectories in Aerial Videos.–
A hierarchical model was developed to account for human judgments in this task, which represents interactivity using la- tent variables, and learns the distribution of critical movement features that signal potential interactivity.
- Joint Parsing of Cross-view Scenes with Spatio-temporal Semantic Parse Graphs.–
A joint parsing method that takes view-centric proposals from pre-trained computer vision models and produces spatio-temporal parse graphs that represents a coherent scene-centric understanding of cross-view scenes is introduced.
- –
- Learning Interactive Affordance for Human-Robot Interaction–
The approach learns structural representations of human-human (and human-object-human) interactions, describing how body-parts of each agent move with respect to each other and what spatial relations they should maintain to complete each sub-event, allowing the robot to infer its own movement in reaction to the human body motion, allowing it to naturally replicate such interactions.
- Evaluating physical quantities and learning human utilities from RGBD videos–
A notion of affordance that takes into account physical quantities generated when the human body interacts with real-world objects is proposed, and a learning framework that incorporates the concept of human utilities is introduced, which in this opinion provides a deeper and finer-grained account not only of object affordance but also of people's interaction with objects.
- –
- Structure vs. Appearance and 3D vs. 2D? A Numeric Answer–
Experiments on these two competing representation pairs show that the optimal representation is data dependent, and forms a spectrum across multiple images, which shows the necessity of having numerical solutions to these problems.
- Integrating 3d and 2d representations for view invariant object recognition–
This thesis presents representations and corresponding algorithms which learn models to recognize objects in the full continuous view space, and proposes to integrate the 3D object- centered representations with 2D viewer-centered representations, which fills in the representation gap between the sparse and simple 3D shapes and their view variant appearances observed as image pixels.
- Perceptual Scale-Space and Its application–
It is argued that the perceptual scale- space enriches the conventional scale-space theory, and provides an important representation for many vision applications, such as super-resolution and multi-scale object recognition.
- Learning and-or templates for object recognition by information projection–
This work focuses on learning probabilistic generative image models as hierarchical AND-OR Templates (AOT), which have the following characteristics which are advantageous in representing visual objects: hierarchical composition (AND) and multiple ways of composition (Discrete/structural OR).
- A statistical and computational theory for the art of painting–
A statistical and computational theory for the art of painting based on studies of painterly rendering and image understanding is developed and a system for rendering abstract paintings with similar effects to those created by artists is developed.
- –
- –
- –
- CCPR 2008 Keynote Speech 3 and Keynote Speech 4–
The objective of this work is to study the structures of the image space, based on which a number of classic models used in computer vision and pattern recognition can be understood.
- A hierarchical compositional model for representation and sketching of high-resolution human images–
The representing power of the And-Or graph model is demonstrated by reconstructing high-resolution face images and generating the cartoon facial sketches, and is useful for a wide variety of applications, including recognition, non-photorealistic rendering, super-resolution, and low-bit rate face coding.
- –
- Computing three-dimensional scene from a single image by bottom-up/top-down bayesian inference–
A bottom-up/top-down Bayesian inference framework to compute the 3D structures of objects in the scene from a single image, which integrates the involved visual tasks in a principled way and incorporates the prior visual knowledge in the inference.
- Modeling complex motion: photometric, geometric, dynamic, and topological aspects–
A unified theory with a generative statistical learning and inference framework to analyze complex visual patterns in both motion and scale-space and aims to answer questions like "What do the authors perceive when they look at a video sequence?"
- Visual Inference by Markov Chain Monte Carlo Methods–
The task integrates conventional vision tasks: image segmentation, perceptual organization, object recognition, etc, and many families of generative models compete to explain the image.
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-14. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.