Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Suggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works161 from public data
- 3,070
- 2,188
- Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words1,938
A novel unsupervised learning method for human action categories that can recognize and localize multiple actions in long and complex video sequences containing multiple motions.
- Dense-Captioning Events in Videos1,676
This work proposes a new model that is able to identify all events in a single pass of the video while simultaneously describing the detected events with natural language, and introduces a new captioning module that uses contextual information from past and future events to jointly describe all events.
- Modeling Temporal Structure of Decomposable Motion Segments for Activity Classification824
A framework for modeling motion by exploiting the temporal structure of the human activities, which represents activities as temporal compositions of motion segments, and shows that the algorithm performs better than other state of the art methods.
- 508
- Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs455
This work introduces Action Genome, a representation that decomposes actions into spatio-temporal scene graphs and demonstrates the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42.7% mAP using as few as 10 examples.
- 447
- ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding443
ULIP is introduced to learn a unified representation of image, text, and 3D point cloud by pre-training with object triplets from the three modalities, using a pre-trained vision-language model that has already learned a common visual and textual space by training with massive image-text pairs.
- DAPs: Deep Action Proposals for Action Understanding425
Deep Action Proposals (DAPs), an effective and efficient algorithm for generating temporal action proposals from long videos, is introduced, which outperforms previous work on a large scale action benchmark, runs at 134 FPS making it practical for large-scale scenarios, and exhibits an appealing ability to generalize.
- Peeking Into the Future: Predicting Future Person Activities and Locations in Videos421
An end-to-end, multi-task learning system utilizing rich visual features about human behavioral information and interaction with their surroundings is proposed, providing the first empirical evidence that joint modeling of paths and activities benefits future path prediction.
- Learning to Decompose and Disentangle Representations for Video Prediction324
The Decompositional Disentangled Predictive Auto-Encoder (DDPAE) is proposed, a framework that combines structured probabilistic models and deep networks to automatically decompose the high-dimensional video that the authors aim to predict into components, and disentangle each component to have low-dimensional temporal dynamics that are easier to predict.
- Few-Shot Video Classification via Temporal Alignment300
This paper proposes the Ordered Temporal Alignment Module (OTAM), a novel few-shot learning framework that can learn to classify a previously unseen video and demonstrates that the model leads to significant improvement of few- shot video classification over a wide range of competitive baselines and outperforms state-of-the-art benchmarks by a large margin.
- 292
- ULIP-2: Towards Scalable Multimodal Pre-Training for 3D Understanding287
ULIP-2 sheds light on a new paradigm for scalable multimodal 3D representation learning without human annotations and shows significant improvements over existing baselines.
- Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers286
A computer vision based algorithm for recognizing single actions of earthmoving construction equipment, based on a multiple binary SVM classifier and spatio-temporal features, which outperforms previous algorithms for excavator and truck action recognition.
- Spatio-Temporal Graph for Video Captioning With Knowledge Distillation277
A novel spatio-temporal graph model for video captioning that exploits object interactions in space and time and an object-aware knowledge distillation mechanism, in which local object information is used to regularize global scene features are proposed.
- Connectionist Temporal Modeling for Weakly Supervised Action Labeling259
The Extended Connectionist Temporal Classification (ECTC) framework is introduced to efficiently evaluate all possible alignments via dynamic programming and explicitly enforce their consistency with frame-to-frame visual similarities.
- End-to-End, Single-Stream Temporal Action Detection in Untrimmed Videos253
This work introduces the new architecture for Single-Stream Temporal Action Detection (SS-TAD), which effectively integrates joint action detection with its semantic sub-tasks in a single unifying end-to-end framework.
- Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors249
Preliminary results on detection of standing workers, excavators and dump trucks with an average accuracy of 98.83%, 82.10%, and 84.88% respectively indicate the applicability of the proposed method for automated activity analysis of workers and equipment from single video cameras.
- Revisiting the “Video” in Video-Language Understanding247
It is found that understanding of event temporality is often not necessary to achieve strong or state-of-the-art performance, even compared with recent large-scale video-language models and in contexts intended to benchmark deeper video-level understanding.
- Align and Prompt: Video-and-Language Pre-training with Entity Prompts238
This paper proposes Align and Prompt: a new video-and-language pre-training framework (AlPro), which operates on sparsely-sampled video frames and achieves more effective cross-modal alignment without explicit object detectors.
- 210
- Leveraging Video Descriptions to Learn Video Question Answering197
A self-paced learning procedure to iteratively identify non-perfect candidate QA pairs and mitigate their effects in training is proposed and shown to be effective and the extended SS model outperforms various baselines.
- Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction195
A framework based on graph convolution to uncover the spatiotemporal relationships in the scene for reasoning about pedestrian intent is presented and a new dataset designed specifically for autonomous-driving scenarios in areas with dense pedestrian populations is introduced: the Stanford-TRI Intent Prediction (STIP) dataset.
- Action-Agnostic Human Pose Forecasting190
The triangular-prism recurrent neural network (TP-RNN) models the hierarchical and multi-scale characteristics of human dynamics and captures the latent hierarchical structure in human pose sequences by encoding temporal dependencies with different time-scales.
- Artificial Intelligence Index Report 2023188
This year's AI Index Report introduces more original data than any previous edition, including a new chapter on AI public opinion, a more thorough technical performance chapter, original analysis about large language and multimodal models, detailed trends in global AI legislation records, a study of the environmental impact of AI systems, and more.
- D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation184
The proposed Discriminative Differentiable Dynamic Time Warping (D3TW) innovatively solves sequence alignment with discriminative modeling and end-to-end training, which substantially improves the performance in weakly supervised action alignment and segmentation tasks.
- 165
- Artificial Intelligence Index Report 2024157
The 2024 Index is the most comprehensive to date and arrives at an important moment when AI's influence on society has never been more pronounced, with new estimates on AI training costs, detailed analyses of the responsible AI landscape, and an entirely new chapter dedicated to AI's impact on science and medicine.
- Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration153
Neural Task Graph Networks are proposed, which use conjugate task graph as the intermediate representation to modularize both the video demonstration and the derived policy, and can effectively predict task structure on the JIGSAWS surgical dataset and generalize to unseen tasks.
- Procedure Planning in Instructional Videos134
The experiments show that the proposed latent space planning is able to learn plannable semantic representations without explicit supervision, which enables sequential reasoning on real-world videos and leads to stronger generalization compared to existing planning approaches and neural network policies.
- Vision-based workface assessment using depth images for activity analysis of interior construction operations133
The results with an average accuracy of 76% on the testing dataset show the promise of vision-based methods using RGB-D sequences for facilitating the activity analysis workface assessment.
- Open Vocabulary Object Detection with Pseudo Bounding-Box Labels122
This work proposes a method to automatically generate pseudo bounding-box annotations of diverse objects from large-scale image-caption pairs that leverages the localization ability of pre-trained vision-language models to generate pseudo Bounding- box labels and then directly uses them for training object detectors.
- Graph Distillation for Action Detection with Privileged Modalities120
This work proposes a method termed graph distillation that incorporates rich privileged information from a large-scale multimodal dataset in the source domain, and improves the learning in the target domain where training data and modalities are scarce.
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas119
This work studies the spatial reasoning challenge from the lens of mechanistic interpretability, diving into the model's internal states to examine the interactions between image and text tokens and proposes ADAPTVIS based on inference-time confidence scores to sharpen the attention on highly relevant regions when confident, while smoothing and broadening the attention window to consider a wider context when confidence is lower.
- Adversarial Cross-Domain Action Recognition with Co-Attention118
A Temporal Co-attention Network (TCoN) is proposed, which matches the distributions of temporally aligned action features between source and target domains using a novel cross-domain co-att attention mechanism.
- Imitation Learning for Human Pose Prediction111
This paper proposes a new reinforcement learning formulation for the problem of human pose prediction, and develops an imitation learning algorithm for predicting future poses under this formulation through a combination of behavioral cloning and generative adversarial imitation learning.
- 105
- Home Action Genome: Cooperative Compositional Action Understanding103
HOMAGE is introduced: a multi-view action dataset with multiple modalities and view-points supplemented with hierarchical activity and atomic action labels together with dense scene composition labels and Cooperative Compositional Action Understanding (CCAU), a cooperative learning framework for hierarchical action recognition that is aware of compositional action elements.
- Representation Learning with Statistical Independence to Mitigate Bias101
A model based on adversarial training with two competing objectives to learn features that have maximum discriminative power with respect to the task and minimal statistical mean dependence with the protected (bias) variable(s) is proposed.
- Vision-Based Estimation of MDS-UPDRS Gait Scores for Assessing Parkinson’s Disease Motor Severity97
A computer vision-based model is proposed that observes non-intrusive video recordings of individuals, extracts their 3D body skeletons, tracks them through time, and classifies the movements according to the MDS-UPDRS gait scores, which is the first benchmark for classifying PD patients based on MDS-UPDRS gait severity.
- Socially and Contextually Aware Human Motion and Pose Forecasting96
A novel framework to tackle both tasks of human motion (or trajectory) and body skeleton pose forecasting in a unified end-to-end pipeline is proposed and achieves a superior performance compared to several baselines on two social datasets.
- Automated Vision-Based Recognition of Construction Worker Actions for Building Interior Construction Operations Using RGBD Cameras96
A novel method for reliable recognition of construction workers and their actions using color and depth data from a Microsoft Kinect sensor is presented, based on machine learning techniques, which adopt a bag-of-poses representation for worker actions and combine it with powerful discriminative classifiers to achieve accurate action recognition.
- Quantifying Parkinson’s disease motor severity under uncertainty using MDS-UPDRS videos94
An ordinal focal neural network is introduced to estimate the MDS-UPDRS scores from input videos, to leverage the ordinal nature of MDS-UPDRS scores and combat class imbalance and demonstrates how computer-assisted technologies can be used to track patients and their motor impairments, even when there is uncertainty in the clinical ratings.
- 93
- X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning92
An efficient and effective framework that integrates multiple modalities (images, 3D, audio and video) to a frozen LLM and demonstrates an emergent ability for cross-modal reasoning (2+ modality inputs).
- Agent-Centric Risk Assessment: Accident Anticipation and Risky Region Localization90
A novel soft-attention Recurrent Neural Network (RNN) which explicitly models both spatial and appearance-wise non-linear interaction between the agent triggering the event and another agent or static-region involved is proposed.
- 86
- Temporal Modular Networks for Retrieving Complex Compositional Activities in Videos81
This work takes a modular neural network approach that, given a natural language query, extracts the semantic structure to assemble a compositional neural network layout and corresponding network modules and shows that it is able to achieve state-of-the-art results on the DiDeMo video retrieval dataset.
- TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild77
A novel TRajectory and POse Dynamics (nicknamed TRiPOD) method based on graph attentional networks to model the human-human and human-object interactions both in the input space and the output space (decoded future output).
- Procedure-Aware Pretraining for Instructional Video Understanding75
The main insight is that instructional videos depict sequences of steps that repeat between instances of the same or different tasks, and that this structure can be well represented by a Procedural Knowledge Graph (PKG), where nodes are discrete steps and edges connect steps that occur sequentially in the instructional activities.
- 75
- Interpretable Visual Question Answering by Visual Grounding From Attention Supervision Mining75
This work demonstrates that it can effectively train a VQA architecture with grounding supervision that can be automatically obtained from available region descriptions and object annotations, and generates visual groundings that achieve a higher correlation with respect to manually-annotated groundings.
- Title Generation for User Generated Videos73
This work addresses video title generation for the first time by proposing two methods that extend state-of-the-art video captioners to this new task and proposes a novel sentence augmentation method to train a captioner with additional sentence-only examples that come without corresponding videos.
- 69
- Visual Forecasting by Imitating Dynamics in Natural Sequences63
A general framework for visual forecasting, which directly imitates visual sequences without additional supervision by formulating visual forecasting as an inverse reinforcement learning (IRL) problem, and directly imitate the dynamics in natural sequences from their raw pixel values.
- Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos57
This work proposes an unsupervised method for reference resolution in instructional videos, where the goal is to temporally link an entity to the action that produced it, and learns a joint visual-linguistic model, where linguistic cues can help resolve visual ambiguities and vice versa.
- Segmenting the Future54
A temporal encoder-decoder network architecture that encodes RGB frames from the past and decodes the future semantic segmentation of the scene segments while simultaneously accounting for the object dynamics to infer the future scene semantic segments is proposed.
- Disentangling Human Dynamics for Pedestrian Locomotion Forecasting with Noisy Supervision53
A framework to unify these two problems and address the practically useful task of pedestrian locomotion prediction in the wild and proposes a novel hierarchical trajectory forecasting network that utilizes low-level vision domain specific signals like egomotion and depth to predict the global trajectory.
- Sparse composition of body poses and atomic actions for human activity recognition in RGB-D videos53
The results show the benefits of using a hierarchical model that exploits the sharing and composition of body poses into atomic actions, and atomic actions into activities, and the compositional capabilities of the model bring robustness to body occlusions.
- 51
- A Hierarchical Pose-Based Approach to Complex Action Understanding Using Dictionaries of Actionlets and Motion Poselets50
A new hierarchical model for human action recognition using body joint locations that can categorize complex actions in videos, and perform spatio-temporal annotations of the atomic actions that compose the complex action being performed.
- Extracting Moving People from Internet Videos50
This work proposes a fully automatic framework to detect and extract arbitrary human motion volumes from real-world videos collected from YouTube and applies a top-down pictorial structure model to complete the extraction of the humans in arbitrary motion.
- 48
- Learning Temporal Action Proposals With Fewer Labels46
This work proposes a semi-supervised learning algorithm specifically designed for training temporal action proposal networks and shows that this approach consistently matches or outperforms the fully supervised state-of-the-art approaches.
- A Deep Learning Based Behavioral Approach to Indoor Autonomous Navigation46
The results show that using a simple sets of perceptual and navigational behaviors, the proposed approach can successfully guide the way of the robot as it completes navigational missions such as going to a specific office.
- 40
- 40
- End-to-End Joint Semantic Segmentation of Actors and Actions in Video36
This work proposes a new end-to-end architecture for tackling semantic segmentation in videos that effectively leverages multiple input modalities, contextual information, and multitask learning in the video to directly output semantic segmentations in a single unified framework.
- 36
- 35
- 29
- 29
- 29
- 29
- 29
- 28
- 26
- 23
- 22
- 20
- 19
- 18
- 18
- 17
- 17
- 17
- 13
- 12
- Camera Motion and Surrounding Scene Appearance as Context for Action Recognition12
A weak foreground-background segmentation approach to robustly extract not only foreground features that are focused on the actors, but also global camera motion and contextual scene information that interestingly is shown to be discriminative for certain action classes.
- 11
- Unifying Specialized Visual Encoders for Video Language Models11
The method, MERV, Multi-Encoder Representation of Videos, leverages multiple frozen visual encoders to create a unified representation of a video, providing the VideoLLM with a comprehensive set of specialized visual knowledge.
- 11
- 11
- 10
- 9
- 9
- 9
- 9
- 9
- 9
- 8
- 8
- 8
- 8
- 7
- 7
- 7
- 5
- 5
- 4
- 3
- 3
- 2
- 2
- 2
- SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation2
This work introduces SMILE: Semantic Metric Integrating Lexical Exactness, a novel approach that combines sentence-level semantic understanding with keyword-level semantic understanding and easy keyword matching, bridging the gap between lexical and semantic evaluation.
- 2
- 2
- 2
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-13. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.