Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Michael M. CohenSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works281 from public data
- 1,512
- Dynamic Convolution: Attention Over Convolution Kernels1,508
By simply using dynamic convolution for the state-of-the-art architecture MobileNetV3-Small, the top-1 accuracy of ImageNet classification is boosted by 2.9% with only 4% additional FLOPs and 2.8 AP gain is achieved on COCO keypoint detection.
- 1,488
- The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)1,017
This paper analyzes the latest model, GPT-4V(ision), to deepen the understanding of large multimodal models, and curate and organize a collection of carefully designed qualitative samples spanning a variety of domains and tasks to inspire future research on the next-generation multi-sensory task formulation.
- 984
- End-to-End Human Pose and Mesh Reconstruction with Transformers813
A new method to reconstruct 3D human pose and mesh vertices from a single image using a transformer encoder to jointly model vertex-vertex and vertex-joint interactions, which generates new state-of-the-art results for human mesh reconstruction on the public Human3.6M and 3DPW datasets.
- GIT: A Generative Image-to-text Transformer for Vision and Language808
This paper design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering and establishes new state of the arts on 12 challenging benchmarks with a large margin.
- Mobile-Former: Bridging MobileNet and Transformer727
The proposed light-weight cross attention to model the bridge enables bidirectional fusion of local and global features and builds an efficient end-to-end detector by replacing backbone, encoder and decoder in DETR with Mobile-Former, which outperforms DETR but saves 52% of computational cost and 36% of parameters.
- Rethinking Classification and Localization for Object Detection712
A Double-Head method is proposed, which has a fully connected head focusing on classification and a convolution head for bounding box regression, and it is found that fc-head has more spatial sensitivity than conv-head.
- End-to-End Semi-Supervised Object Detection with Soft Teacher698
This paper presents an end-to-end semi-supervised object detection approach, in contrast to previous more complex multi-stage methods, and proposes two simple yet effective techniques within this framework: a soft teacher mechanism where the classification loss of each unlabeled bounding box is weighed by the classification score produced by the teacher network.
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action649
This paper defines and explores a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models and proposes MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.
- An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA559
This work proposes PICa, a simple yet effective method that Prompts GPT3 via the use of Image Captions, for knowledge-based VQA, and treats GPT-3 as an implicit and unstructured KB that can jointly acquire and process relevant knowledge.
- Agent Laboratory: Using LLM Agents as Research Assistants534
This work introduces Agent Laboratory, an autonomous LLM-based framework capable of completing the entire research process, and finds that it significantly reduces research expenses, achieving an 84% decrease compared to previous autonomous research methods.
- 531
- Robust 3D Action Recognition with Random Occupancy Patterns501
This work extracts semi-local features called random occupancy pattern ROP features, which employ a novel sampling scheme that effectively explores an extremely large sampling space and utilizes a sparse coding approach to robustly encode these features.
- An Empirical Study of Training End-to-End Vision-and-Language Transformers482
Meter, a Multimodal End-to-end TransformER framework, is presented, through which it is investigated how to design and pre-train a fully transformer-based VL model in an end- to-end manner and provides insights on how to train a performant VL transformer.
- Mesh Graphormer444
Experimental results show that the proposed method, Mesh Graphormer, significantly outperforms the previous state-of-the-art methods on multiple benchmarks, including Human3.6M, 3DPW, and FreiHAND datasets.
- HP-GAN: Probabilistic 3D Human Motion Prediction via GAN384
A novel sequence-to-sequence model for probabilistic human motion prediction, trained with a modified version of improved Wasserstein generative adversarial networks (WGAN-GP), in which the model learns a probability density function of future human poses conditioned on previous poses.
- SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning356
SwinBERT is an end-to-end transformer-based model for video captioning, which takes video frame patches directly as inputs, and outputs a natural language description, and adopts a video transformer to encode spatial-temporal representations that can adapt to variable lengths of video input without dedicated design for different frame rates.
- 344
- Scaling Up Vision-Language Pretraining for Image Captioning342
LEMON O, a LargE-scale iMage captiONer, is presented, and the first empirical study on the scaling behavior of VLP for image captioning is provided, and it is shown LEMON can generate captions with long-tail vi-sual concepts when used in a zero-shot manner.
- 326
- 318
- 318
- STOP: Space-Time Occupancy Patterns for 3D Action Recognition from Depth Map Sequences294
This paper presents Space-Time Occupancy Patterns (STOP), a new visual representation for 3D action recognition from sequences of depth maps that preserves spatial and temporal contextual information between space-time cells while being flexible enough to accommodate intra-action variations.
- TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking269
This paper proposes TransMOT, which leverages powerful graph transformers to efficiently model the spatial and temporal interactions among the objects and achieves state-of-the-art performance on all the datasets.
- Expressive expression mapping with ratio images260
This paper captures the illumination change of one person's expression in what they call an expression ratio image (ERI), and together with geometric warping, they map an ERI to any other person's face image to generate more expressive facial expressions.
- Distributed meetings258
A system that provides speaker clustering, spatial indexing, and time compression to provide a rich experience for people who miss a meeting and want to watch it afterward and a user study evaluation of the system is described.
- 227
- Dynamic ReLU226
This paper proposes dynamic ReLU (DY-ReLU), a dynamic rectifier of which parameters are generated by a hyper function over all in-put elements, and encodes the global context into the hyper function, and adapts the piecewise linear activation function accordingly.
- ReCo: Region-Controlled Text-to-Image Generation220
The proposed model, dubbed as ReCo (Region-Controlled T2I), enables the region control for arbitrary objects described by open-ended regional texts rather than by object labels from a constrained category set, and can better control the object count, spatial relationship, and region attributes such as color/size, with the free-form regional description.
- Disco: Disentangled Control for Realistic Human Dance Generation187
Disco is introduced, which includes a novel model architecture with disentangled control to improve the compositionality of dance synthesis, and an effective human attribute pre-training for better generalizability to unseen humans.
- 181
- 179
- Human pose estimation and its application to action recognition: A survey175
This survey focuses on 2D skeleton-based action recognition where the human poses are estimated from regular RGB images instead of depth images, and summarizes the performance of recent action recognition methods that use pose estimated from color images as input, showing that there is much room for improvements in this direction.
- 173
- Rapid modeling of animated faces from video173
A system that constructs textured 3D face models from videos with minimal user interaction and the goal is to allow an untrained user with a PC and an ordinary camera to create and instantly animate his/her face model in no more than a few minutes.
- 168
- GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation166
The findings demonstrate that large multimodal models, specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities.
- Discriminative Orderlet Mining for Real-Time Recognition of Human-Object Interaction160
This paper presents a novel visual representation, called orderlets, for real-time human action recognition with depth sensors, that is insensitive to small noise since an orderlet only depends on the comparative relationship among individual features.
- Sparse representation and learning in visual recognition: Theory and applications155
A survey of some recent work on sparse representation, learning and modeling with emphasis on visual recognition, and the applications of sparse theory to various visual recognition tasks are introduced.
- 154
- UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling154
UniTAB’s unified multi-task network and the task-agnostic output sequence design make the model parameter efficient and generalizable to new tasks, and could provide a more comprehensive and interpretable image description.
- ARTiFACIAL: Automated Reverse Turing test using FACIAL features142
This paper designs attacks using the best existing face detectors and demonstrated the challenge they presented to bots, and proposes a new HIP algorithm based on detecting human face and facial features.
- Injecting Semantic Concepts into End-to-End Image Captioning134
This paper proposes a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features, and introduces a novel Concept Token Network (CTN) to predict the semantic concepts and then incorporate them into the end-to-end captioning.
- 133
- 128
- Compressing Visual-linguistic Model via Knowledge Distillation127
This paper uses the mean square error loss to mimic the attention distribution inside the transformer block, and presents a token-wise noise contrastive loss to align the hidden state by contrasting with negative representations stored in a sample queue.
- MicroNet: Improving Image Recognition with Extremely Low FLOPs113
This paper found that two factors, sparse connectivity and dynamic activation function, are effective to improve the accuracy and proposes micro-factorized convolution, which factorizes a convolution matrix into low rank matrices, to integrate sparse connectivity into convolution.
- Image Ratio Features for Facial Expression Recognition Application107
Experimental results show that the proposed image ratio feature is more robust to albedo and lighting variations, and the combination of image ratio features and FAPs outperforms each feature alone.
- 105
- LAVENDER: Unifying Video-Language Understanding as Masked Language Modeling104
Surprisingly, experimental results show that this unified VidL framework LAVENDER achieves competitive performance on 14 VidL benchmarks, covering video question answering, text-to-video retrieval and video captioning.
- Eye gaze tracking using an RGBD camera: a comparison with a RGB solution99
A non-infrared based approach to track one's eye gaze with an RGBD camera (in this case, Kinect) that adopts a personalized 3D face model constructed off-line and finds that the use of depth information directly from Kinect achieves more accurate tracking.
- 99
- MM-VID: Advancing Video Understanding with GPT-4V(ision)98
MM-VID is an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video understanding and demonstrates the effectiveness of MM-VID in handling distinct video genres with various video lengths.
- An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling95
This work systematically examines the potential of MVM in the context of VidL learning, and conducts comprehensive experiments and provides insights into the factors leading to effective MVM training, resulting in an enhanced model VIOLETv2.
- 94
- 91
- 87
- 87
- Real time gaze estimation with a consumer depth camera87
A low-cost, non-intrusive, simple-setup gaze estimation system that can estimate the gaze direction under free head movement and is robust against large head movements is presented.
- Robust and Rapid Generation of Animated Faces from Video Images: A Model-Based Modeling Approach87
An easy-to-use and cost-effective system to construct textured 3D animated face models from videos with minimal user interaction, which makes full use of generic knowledge of faces in head motion determination, head tracking, model fitting, and multiple-view bundle adjustment.
- 86
- 84
- 83
- MPS-NeRF: Generalizable 3D Human Rendering From Multiview Images82
This article proposes a simple yet surprisingly effective method to train a generalizable NeRF with multiview images as conditional input and leverages the parametric 3D human model fitted on the input images to derive the deformation.
- 82
- An image-to-class dynamic time warping approach for both 3D static and trajectory hand gesture recognition81
The experiment results show that the proposed I2C-DTW approach significantly improves the recognition accuracy on both static gestures and trajectory gestures.
- 80
- 79
- Equivariant Similarity for Vision-Language Foundation Models74
This study proposes EqSim, a regularization loss that can be efficiently calculated from any two matched training pairs and easily pluggable into existing image-text retrieval fine-tuning and presents a new challenging benchmark EqBen, the first to focus on "visual-minimal change".
- 72
- 72
- 68
- Playing Lottery Tickets with Vision and Language64
This work performs the first empirical study to assess whether trainable subnetworks also exist in pre-trained VL models, and finds that LTH remains relevant when using other training methods (e.g., adversarial training).
- VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning64
The results show that the VIsual VOcabulary pre-training model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects.
- Propagative Hough Voting for Human Activity Recognition63
The proposed activity recognition method does not rely on human detection and tracking, and can well handle the scale and intra-class variations of the activity patterns, and outperforms the state-of-the-art techniques when there is sufficient training data.
- On the improvement of human action recognition from depth map sequences using Space–Time Occupancy Patterns60
A new visual representation for 3D action recognition from sequences of depth maps that preserves spatial and temporal contextual information between space and time cells while being flexible enough to accommodate intra-action variations and combines depth maps with skeletons to obtain view invariance.
- Predicting human activities using spatio-temporal structure of interest points60
A new random forest structure is proposed, called multi-class balanced random forest, which makes a good trade-off between the balance of the trees and the discriminative abilities and significantly outperforms the state of the arts for the human activity prediction problem.
- VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training.59
The results show that the VIsual VOcabulary pre-training model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects.
- A Tube-and-Droplet-Based Approach for Representing and Analyzing Motion Trajectories59
This paper addresses the problem of representing motion trajectories in a highly informative way, and utilizes the droplet-based process for analyzing trajectories and applies it to trajectory analysis applications including trajectory clustering, trajectory classification & abnormality detection, and 3D action recognition.
- ImmerseBoard: Immersive Telepresence Experience using a Digital Whiteboard56
User studies involving three tasks show that compared to standard video conferencing with a digital whiteboard, ImmerseBoard provides participants with a quantitatively better ability to estimate their remote partners' eye gaze direction, gesture direction, intention, and level of agreement.
- Lifelong Unsupervised Domain Adaptive Person Re-identification with Coordinated Anti-forgetting and Adaptation53
An effective scheme for this task, dubbed CLUDA-ReID, where the anti-forgetting is harmoniously coordinated with the adaptation, and a meta-based Coordinated Data Replay strategy is proposed to replay old data and update the network with a coordinated optimization direction for both adaptation and memorization.
- 53
- 52
- 51
- 50
- 49
- 48
- MMPTRACK: Large-scale Densely Annotated Multi-camera Multiple People Tracking Benchmark47
This work appeals to an auto-annotation system to reduce the cost, which uses overlapped and calibrated depth and RGB cameras to build a 3D tracker and automatically generates the 3D tracking results, and collects and annotates a large-scale densely labeled multi-camera tracking dataset from five different environments.
- 47
- 46
- Reinforced Temporal Attention and Split-Rate Transfer for Depth-Based Person Re-identification45
This work introduces split-rate RGB-to-Depth transfer, which leverages large RGB datasets more effectively than popular fine-tuning approaches and enhances re-identification for video by implementing temporal attention as a Bernoulli-Sigmoid unit acting upon frame-level features.
- 42
- 41
- Human Action Recognition with Depth Cameras39
The authors provide in-depth descriptions of their recently developed feature representations and machine learning techniques, including lower-level depth and skeleton features, higher-level representations to model the temporal structure and human-object interactions, and feature selection techniques for occlusion handling.
- 38
- Crossmodal Representation Learning for Zero-shot Action Recognition38
A cross-modal Transformer-based frame-work, which jointly encodes video data and text labels for zero-shot action recognition (ZSAR), which employs a conceptually new pipeline by which visual representations are learned in conjunction with visual-semantic associations in an end-to-end manner.
- 37
- 37
- 37
- 35
- 34
- 32
- 32
- 31
- Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization28
An algorithm named pairwise sample optimization (PSO) is presented, which enables the direct fine-tuning of an arbitrary timestep-distilled diffusion model, and can directly adapt distilled models to human-preferred generation with both offline and online-generated pairwise preference image data.
- 28
- 28
- 27
- 26
- 26
- 25
- 25
- 24
- 23
- 23
- 22
- 22
- Real-time human action search using random forest based hough voting21
This paper proposes a novel framework for action retrieval which performs pattern matching at subvolume level and is very efficient in handling large corpus of videos and presents action search experiments on challenging datasets to validate the efficiency and effectiveness of the system.
- 21
- 21
- 20
- 20
- 20
- 19
- 19
- Speeding up spatio-temporal sliding-window search for efficient event detection in crowded videos19
This work proposes a heuristic branch-and-bound solution for event detection in videos by directly finding subvolumes whose scores are higher than the threshold, which dramatically reduces computational cost without significantly degrading the quality of the detection results.
- OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation18
A new interleaved generation framework based on prompting large-language models (LLMs) and pre-trained text-to-image (T2I) models, namely OpenLEAF is proposed, which can generate high-quality image-text content for various domains and applications.
- 18
- 18
- Efficient Scale-Space Spatiotemporal Saliency Tracking for Distortion-Free Video Retargeting18
This paper proposes a novel approach to distortion-free video retargeting by scale-space spatiotemporal saliency tracking, and develops an efficient coarse-to-fine solution that combines exhaustive coarse and gradient-based fine search.
- 18
- 17
- 17
- 16
- 16
- 16
- 16
- Adaptive Human Matting for Dynamic Videos15
This work introduces Adaptive Matting for Dynamic Videos, termed AdaM, which is a framework designed for simultaneously differentiating foregrounds from backgrounds and capturing alpha matte details of human subjects in the foreground, and achieves new best-in-class generalizability.
- 15
- 14
- 14
- 13
- 13
- 12
- 12
- 12
- 12
- 12
- Spatial-Frequency U-Net for Denoising Diffusion Probabilistic Models11
This paper designs a novel architecture SFUNet to effectively capture the correlation for both spatial and frequency domains in wavelet data and finds the model is able to generate images with higher quality on CIFAR-10, FFHQ, LSUN-Bedroom, and LSun-Church datasets, than the pixel-based counterpart.
- Neural Voting Field for Camera-Space 3D Hand Pose Estimation11
The proposed Neural Voting Field (NVF) fully models 3D dense local evidence and hand global geometry, helping to alleviate common 2D-to-3D ambiguities and outperforms existing state-of-the-art algorithms on FreiHAND dataset for camera-space 3D hand pose estimation.
- 11
- 11
- 11
- 10
- 10
- 10
- 10
- 9
- 9
- 9
- 8
- 8
- 8
- 8
- 7
- 7
- 7
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 6
- 5
- 5
- 5
- 5
- 5
- 5
- 5
- 5
- 5
- 5
- OpenLEAF: A Novel Benchmark for Open-Domain Interleaved Image-Text Generation4
This work introduces OpenLEAF, a new benchmark designed for the emerging open-domain interleaved image-text generation task, and proposes a novel evaluation pipeline named "detection-summarization-scoring," which breaks down the evaluation into multiple reasoning steps.
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- 3
- 3
- 3
- OVIS: Open-Vocabulary Visual Instance Search via Visual-Semantic Aligned Representation Learning3
Through extensive experiments, ViSA's ability to search for visual instances in images not available during training given a wide range of textual queries including those composed of uncommon words is demonstrated.
- 3
- 3
- Unsupervised Trees for Human Action Search3
This chapter proposes a very fast action retrieval system which can effectively locate the subvolumes similar to the query video and proposes a coarse-to-fine subvolume search scheme, which results in a dramatic speedup over the existing video branch-and-bound method.
- 3
- 3
- 3
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- Propagative Hough Voting to Leverage Contextual Information1
The superior performances on benchmarked datasets validate that the propagative Hough voting can outperform state-of-the-art techniques in various action analysis tasks, such as action search and recognition.
- Introduction to Human Action Analysis1
This chapter starts from an overview of human action analysis problems, and then discusses the recent development of the four humanaction analysis applications as well as tree-based approaches.
- 1
- Human Action Prediction with Multiclass Balanced Random Forest1
This chapter develops a spatial-temporal implicit shape model (STISM), which characterizes the space-time structure of the sparse local features extracted from a video, and proposes a new random forest structure, called multiclass balanced random forest, which makes a good trade-off between the balance of the trees and the discriminative abilities.
- 1
- Human Action Analysis with Randomized Trees1
This book describes how the randomized trees can be used for action classification, action detection, action search, and action prediction, and techniques for space-time action localization including branch-and-bound sub-volume search and propagative Hough voting.
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- Supervised Trees for Human Action Recognition and Detection–
In this chapter, a random forest is constructed to efficiently generate discriminative votes from individual interest points, and a fast top-K subvolume search algorithm is developed to find all action instances in a single round of search.
- –
- –
- –
- –
- –
- –
- –
- –
- –
- Introduction to the ICME2010 Special Issue–
The 15 papers in this special issue are extended versions of papers presented at the 2010 IEEE International Conference on Multimedia and Expo (ICME), held in Singapore on July 19-23, 2010.
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-13. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.