Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Matthew TurkSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works219 from public data
- A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection1,570
A unified deep neural network, denoted the multi-scale CNN (MS-CNN), is proposed for fast multi- scale object detection, which is learned end-to-end, by optimizing a multi-task loss.
- CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning672
This work proposes to learn a set of prompt components which are assembled with input- Conditioned weights to produce input-conditioned prompts, resulting in a novel attention-based end-to-end key-query scheme.
- BlockDrop: Dynamic Inference Paths in Residual Networks554
BlockDrop, an approach that learns to dynamically choose which layers of a deep network to execute during inference so as to best reduce total computation without degrading prediction accuracy, is introduced.
- SpotTune: Transfer Learning Through Adaptive Fine-Tuning525
In SpotTune, given an image from the target task, a policy network is used to make routing decisions on whether to pass the image through the fine-tuned layers or the pre-trained layers, which outperforms the traditional fine- Tuning approach on 12 out of 14 standard datasets.
- 460
- A Broader Study of Cross-Domain Few-Shot Learning436
The proposed Broader Study of Cross-Domain Few-Shot Learning (BSCD-FSL) benchmark, consisting of image data from a diverse assortment of image acquisition methods, demonstrates that state-of-art meta- learning methods are surprisingly outperformed by earlier meta-learning approaches, and all meta- Learning methods underperform in relation to simple fine-tuning.
- Cross-Domain Image Retrieval with a Dual Attribute-Aware Ranking Network431
This work proposes a Dual Attribute-aware Ranking Network (DARN) for retrieval feature learning, consisting of two sub-networks, one for each domain, whose retrieval feature representations are driven by semantic attribute learning.
- Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification410
Evaluation on person attributes classification tasks involving facial and clothing attributes suggests that the models produced by the proposed method are fast, compact and can closely match or exceed the state-of-the-art accuracy from strong baselines by much more expensive models.
- Image ranking and retrieval based on multi-attribute queries393
This work proposes a principled approach for multi-attribute retrieval which explicitly models the correlations that are present between the attributes in the vocabulary, and integrates ranking and retrieval within the same formulation.
- RepMet: Representative-Based Metric Learning for Classification and Few-Shot Object Detection386
This work proposes a new method for DML that simultaneously learns the backbone network parameters, the embedding space, and the multi-modal distribution of each of the training categories in that space, in a single end-to-end training process.
- 341
- Targeted Supervised Contrastive Learning for Long-Tailed Recognition281
Targeted supervised contrastive learning (TSC) is proposed, which improves the uniformity of the feature distribution on the hypersphere, and achieves state-of-the-art performance on long-tailed recognition tasks.
- Revisiting RCNN: On Awakening the Classification Power of Faster RCNN272
The potential of detector classification power is demonstrated by a simple, effective, and widely-applicable Decoupled Classification Refinement (DCR) network and failure cases of state-of-the-art detectors are analyzed.
- 266
- 264
- Differential Treatment for Stuff and Things: A Simple Unsupervised Domain Adaptation Method for Semantic Segmentation256
The reason why the current adversarial loss is often unstable in minimizing the distribution discrepancy is revealed and it is shown that the method can help ease this issue by minimizing the most similar stuff and instance features between the source and the target domains.
- 230
- Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation226
The key idea is to design a generator that competes against a discriminator that explores weaknesses of the discriminators, while the discriminator learns from hard augmentations to achieve better performance.
- Diversity in Faces.222
Diversity in Faces (DiF) provides a data set of one million annotated human face images for advancing the study of facial diversity, and believes that by making the extracted coding schemes available on a large set of faces, can accelerate research and development towards creating more fair and accurate facial recognition systems.
- 221
- 215
- 210
- 198
- AR-Net: Adaptive Frame Resolution for Efficient Action Recognition188
A novel approach, called AR-Net (Adaptive Resolution Network), that selects on-the-fly the optimal resolution for each frame conditioned on the input for efficient action recognition in long untrimmed videos.
- Everything at Once – Multi-modal Fusion Transformer for Video Retrieval173
This work presents a multi-modal, modality agnostic fusion transformer that learns to exchange information between multiple modalities, such as video, audio, and text, and integrate them into a fused representation in a joined multi- modal embedding space.
- Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning172
This work proposes multitask prompt tuning (MPT), which first learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts, then learns multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task.
- AVLnet: Learning Audio-Visual Language Representations from Instructional Videos149
This work introduces Audio-Video Language Network (AVLnet), a self-supervised network that learns a shared audio-visual embedding space directly from raw video inputs, and performs analysis of AVLnet's learned representations, showing the model has learned to relate visual objects with salient words and natural sounds.
- A Recurrent Encoder-Decoder Network for Sequential Face Alignment146
A novel recurrent encoder-decoder network model for real-time video-based face alignment that predicts 2D facial point maps regularized by a regression loss, while uniquely exploiting recurrent learning at both spatial and temporal dimensions is proposed.
- 139
- A Broad Study on the Transferability of Visual Representations with Contrastive Learning134
A comprehensive study on the transferability of learned representations of different contrastive approaches for linear evaluation, full-network transfer, and few-shot recognition on 12 downstream datasets from different domains, and object detection tasks on MSCOCO and VOC0712 shows that the contrasts learn representations that are easily transferable to a different downstream task.
- 134
- LaSO: Label-Set Operations Networks for Multi-Label Few-Shot Learning132
This work proposes a novel technique for synthesizing samples with multiple labels for the (yet unhandled) multi-label few-shot classification scenario, and proposes to combine pairs of given examples in feature space so that the resulting synthesized feature vectors will correspond to examples whose label sets are obtained through certain set operations on the label sets of the corresponding input pairs.
- Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition125
An unified framework is developed for both 2D-CNN and 3D- CNN action models, which enables us to remove bells and whistles and provides a common ground for fair comparison, and reveals a significant leap is made in efficiency for action recognition, but not in accuracy.
- Baby steps towards few-shot learning with multiple semantics116
This paper offers a new result on the one-shot test of the popular miniImageNet benchmark, comparing favorably to the previous state-of-the-art results for both visual only and visual plus semantics-based approaches.
- Walk and Learn: Facial Attribute Representation Learning from Egocentric Video and Contextual Data116
This work tracks the faces of casual walkers on more than 40 hours of egocentric video and automatically extracts nearly 5 million pairs of images connected by or from different face tracks, along with their weather and location context, under pose and lighting variations, to learn a rich feature representation for facial attribute classification.
- Semi-Supervised Action Recognition with Temporal Contrastive Learning115
This work proposes a two-pathway temporal contrastive model using unlabeled videos at two different speeds lever-aging the fact that changing video speed does not change an action to maximize the similarity between encoded representations of the same video atTwo different speeds as well as minimize the similarities between different videos played at different speeds.
- Detection and Tracking of Facial Features in Video Sequences110
A real time system for detection and tracking of facial features in video sequences, using a statistical skin-color model to segment face-candidate regions in the image, based on an efficient template matching scheme.
- 107
- Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos105
A self-supervised training framework that learns a common multimodal embedding space that enforces a grouping of semantically similar instances that enables retrieval of samples across all modalities, even from unseen datasets and different domains is proposed.
- 105
- Learning to Grow Pretrained Models for Efficient Transformer Training102
This paper describes an approach for accelerating transformer training by learning to grow pretrained transformers, where it is learned to linearly map the parameters of the smaller model to initialize the larger model.
- Teaching Structured Vision & Language Concepts to Vision & Language Models101
This paper introduces the collective notion of Structured Vision & Language Concepts (SVLC) which includes object attributes, relations, and states which are present in the text and visible in the image and does not require any additional data.
- 95
- Attribute-based People Search: Lessons Learnt from a Practical Surveillance System94
It is shown that a novel set of multimodal integral filters and proper normalization of attribute scores are critical to obtain good performance in the problem of attribute-based people search in real surveillance environments.
- The Fashion IQ Dataset: Retrieving Images by Combining Side Information and Relative Natural Language Feedback.87
A novel approach is proposed and empirically demonstrated that combining natural language feedback with visual attribute information results in superior user feedback modeling and retrieval performance relative to using either of these modalities.
- S3Pool: Pooling with Stochastic Spatial Sampling87
This work proposes a novel pooling strategy with stochastic spatial sampling (S3Pool), where the regular downsampling is replaced by a more general stochastics version that acts as a strong regularizer and can be seen as doing implicit data augmentation by introducing distortions in the feature maps.
- TAFSSL: Task-Adaptive Feature Sub-Space Learning for Few-Shot Classification86
It is shown that on the challenging miniImageNet and tieredImageNet benchmarks, TAFSSL can improve the current state-of-the-art in both transductive and semi-supervised FSL settings by more than $5\%$, while increasing the benefit of using unlabeled data in FSL to above $10\%$ performance gain.
- Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding85
The Multi-Moments in Time dataset (M-MiT) is presented which includes over two million action labels for over one million three second videos and introduces novel challenges on how to train and analyze models for multi-action detection.
- 79
- 75
- 75
- 74
- Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions73
The Spoken Moments (S-MiT) dataset of 500k spoken captions each attributed to a unique short video depicting a broad range of different events is presented and a novel Adaptive Mean Margin (AMM) approach to contrastive learning is presented to evaluate the authors' models on video/caption retrieval on multiple datasets.
- AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition73
An adaptive multi-modal learning framework, called AdaMML, is proposed that selects on-the-fly the optimal modalities for each segment conditioned on the input for efficient video recognition.
- OnlineAugment: Online Data Augmentation with Less Domain Knowledge69
An orthogonal online data augmentation scheme together with three new augmentation networks, co-trained with the target learning task, that is both more efficient, in the sense that it does not require expensive offline training when entering a new domain, and more adaptive as it adapts to the learner state.
- 67
- Alleviating Semantic-level Shift: A Semi-supervised Domain Adaptation Method for Semantic Segmentation67
A semi-supervised approach named Alleviating Semantic-level Shift (ASS) is proposed, which can promote the distribution consistency from both global and local views and can beat the oracle model trained on the whole target dataset.
- Fine-grained Angular Contrastive Learning with Coarse Labels66
A novel ’Angular normalization’ module is introduced that allows to effectively combine supervised and self-supervised contrastive pre-training to approach the proposed C2FS task, demonstrating significant gains in a broad study over multiple baselines and datasets.
- 64
- Real-Time Detection of Abandoned and Removed Objects in Complex Environments63
A new framework to robustly and efficiently detect abandoned and removed objects in complex environments for real-time video surveillance is presented, significantly outperforming previous edge-based techniques.
- MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge61
This work adapts a VL model for zero-shot and few-shot action recognition using a collection of unlabeled videos and an unpaired action dictionary, and leverages Large Language Models and VL models to build a text bag for each unlabeled video via matching, text expansion and captioning.
- VALHALLA: Visual Hallucination for Machine Translation59
A visual hallucination framework, called VALHALLA, which requires only source sentences at inference time and instead uses hallucinated visual representations for multi-modal machine translation, and demonstrates the effectiveness over both text-only baselines and state-of-the-art methods.
- Dynamic Network Quantization for Efficient Video Inference58
A dynamic network quantization framework, that selects optimal precision for each frame conditioned on the input for efficient video recognition is proposed, that provides significant savings in computation and memory usage while outperforming the existing state-of-the-art methods.
- Unsupervised Domain Generalization by Learning a Bridge Across Domains56
This work shows how using an edge-regularized BrAD the approach achieves significant gains across multiple benchmarks and a range of tasks, including UDG, Few-shot UDA, and unsupervised generalization across multi-domain datasets (including generalization to unseen domains and classes).
- 56
- Going Beyond Nouns With Vision & Language Models Using Synthetic Data52
This work investigates to which extent purely synthetic data could be leveraged to teach large-scale pre-trained Vision & Language models to overcome shortcomings without compromising their zero-shot capabilities, and contributes Synthetic Visual Concepts - a million-scale synthetic dataset and data generation codebase allowing to generate additional suitable data to improve VLC understanding and compositional reasoning of VL models.
- Fast Neural Networks with Circulant Projections.50
This work proposes to replace the conventional linear projection with the circulant projection, which enables the use of the Fast Fourier Transform to speed up the computation of a fully-connected neural network layer.
- Task2Sim: Towards Effective Pre-training and Transfer from Synthetic Data47
Task2Sim, a unified model mapping downstream task representations to optimal sim-ulation parameters to generate synthetic pre-training data for them, is introduced, which results in significantly better downstream per-formance than non-adaptively choosing simulation param-eters on both seen and unseen tasks.
- Collaborative Human-AI (CHAI): Evidence-Based Interpretable Melanoma Classification in Dermoscopic Images47
A novel hierarchical triplet logic is implemented to jointly learn an embedding according to disease labels and non-expert similarity, and results are improved over baselines trained on disease labels alone, as well as standard multiclass loss.
- Attribute-based vehicle search in crowded surveillance videos46
A novel application for searching for vehicles in surveillance videos based on semantic attributes that has the ability to handle challenging urban conditions such as high volumes of activity and environmental factors.
- A New Benchmark for Evaluation of Cross-Domain Few-Shot Learning.45
This paper proposes the cross-domain few-shot learning (CD-FSL) benchmark, consist-ing of images from diverse domains with varying similarity to ImageNet, ranging from crop disease images, satellite images, and medical images, to serve as a challenging platform to guide future research on cross- domain few- shot learning.
- Learning Motion in Feature Space: Locally-Consistent Deformable Convolution Networks for Fine-Grained Action Detection44
A novel locally-consistent deformable convolution is proposed, which utilizes the change in receptive fields and enforces a local coherency constraint to capture motion information effectively to produce features capable of modeling fine-grained motion more efficiently.
- 44
- How Transferable Are Video Representations Based on Synthetic Data?43
This work proposes SynAPT, a novel benchmark for action recognition based on a combination of existing synthetic datasets, in which a model is pre-trained on synthetic videos rendered by various graphics simulators, and then transferred to a set of downstream action recognition datasets, containing different categories than the synthetic data.
- Active Wavelet Networks for Face Alignment42
A new method for face alignment called active wavelet networks (AWN) is proposed, which replaces the AAM texture model by a wavelet network representation, which shows more robustness against partial occlusions and some illumination changes.
- Separating Skills and Concepts for Novel Visual Question Answering40
This work presents a novel method for learning to compose skills and concepts that separates these two factors implicitly within a model by learning grounded concept representations and disentangling the encoding of skills from that of concepts.
- 40
- Adaptive Memory Replay for Continual Learning39
This work introduces a framework of adaptive memory replay for continual learning, where sampling of past data is phrased as a multi-armed bandit problem and utilizes Bolzmann sampling to derive a method which dynamically selects past data for training conditioned on the current task, assuming full data access and emphasizing training efficiency.
- 38
- Learning Object Detectors from Scratch with Gated Recurrent Feature Pyramids.38
This study proposes a recurrent feature-pyramid structure to squeeze rich spatial and semantic features into a single prediction layer that further reduces the number of parameters to learn, and is the best performed model of learning object detection from scratch.
- 38
- 37
- 36
- 34
- Select, Label, and Mix: Learning Discriminative Invariant Feature Representations for Partial Domain Adaptation33
A novel ‘Select, Label, and Mix’ (SLM) framework that aims to learn discriminative invariant feature representations for partial domain adaptation and demonstrates the superiority of the proposed framework over state-of-the-art methods.
- Detector-Free Weakly Supervised Grounding by Separation33
The key idea behind the proposed Grounding by Separation (GbS) method is synthesizing ‘text to image-regions’ associations by random alpha-blending of arbitrary image pairs and using the corresponding texts of the pair as conditions to recover the alpha map from the blended image via a segmentation network.
- 33
- ConStruct-VL: Data-Free Continual Structured VL Concepts Learning*32
This work proposes a data-free method comprised of a new approach of Adversarial Pseudo-Replay (APR) which generates adversarial reminders of past tasks from past task models and proposes a continual parameter-efficient Layered-LoRA (LaLo) neural architecture allowing no-memory-cost access to all past models at train time.
- Pseudo-IoU: Improving Label Assignment in Anchor-Free Object Detection31
PseudoIntersection-over-Union (PSEudo-IoU): a simple metric that brings more standardized and accurate assignment rule into anchor-free object detection frameworks without any additional computational cost or extra parameters for training and testing, making it possible to further improve anchor- free object detection.
- 31
- Sim VQA: Exploring Simulated Environments for Visual Question Answering30
This work explores using synthetic computer-generated data to fully control the visual and language space, allowing us to provide more diverse scenarios on VQA, and proposes Feature Swapping (F-SWAP) - where it is shown that F- SWAP is effective for improving V QA models on real images without compromising on their accuracy to answer existing questions in the dataset.
- Advances in Visual Computing28
This paper proposes a novel hybrid example-based single image superresolution approach which integrates learning from both external and internal exemplars and outperforms the recent state-of-the-art single imagesuperresolution approaches both quantitatively and qualitatively.
- Eigenfaces Versus Eigeneyes: First Steps Toward Performance Assessment of Representations for Face Recognition26
The main contributions of the present paper are the description of the performance assessment framework (which is still under development), the results of the two experiments and a discussion of some possible reasons for them.
- Grounding Spoken Words in Unlabeled Video24
Deep learning models that learn joint multi-modal embeddings in videos where the audio and visual streams are loosely synchronized are explored, and with weak supervision the authors see significant amounts of cross- modal learning.
- 23
- RED-Net: A Recurrent Encoder–Decoder Network for Video-Based Face Alignment23
This work proposes a novel method for real-time face alignment in videos based on a recurrent encoder–decoder network model that predicts 2D facial point heat maps regularized by both detection and regression loss, while uniquely exploiting recurrent learning at both spatial and temporal dimensions.
- 23
- Benchmarking Datasets for Human Activity Recognition22
This chapter gives an overview of different benchmarking datasets, summarizes the performances of the-state-of-the-art algorithms, and analyzes these datasets.
- 22
- Shape-Enhanced Surgical Visualizations and Medical Illustrations with Multi-flash Imaging22
A novel approach for enhancing images and video used in endoscopic surgery so that they are better able to convey shape, based on multi-flash imaging, in which multiple light sources are strategically positioned to cast shadows along depth discontinuities.
- 21
- Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs20
- MetAdapt: Meta-learned task-adaptive architecture for few-shot classification20
This work proposes to employ tools inspired by the Differentiable Neural Architecture Search (D-NAS) literature in order to optimize the architecture for FSL without over-fitting, and proposes the concept of `MetAdapt Controller' modules, added to the model and meta-trained to predict the optimal network connections for a given novel task.
- Automatic Curation of Golf Highlights Using Multimodal Excitement Features20
This work proposes a novel approach for auto-curating sports highlights, and uses it to create a real-world system for the editorial aid of golf highlight reels, and introduces new techniques for learning classifiers with reduced manual training data annotation.
- 20
- Event Detection, Query, and Retrieval for Video Surveillance20
This chapter explores video parsing techniques that automatically extract index data from video, indexing which stores data in relational tables, retrieval which uses SQL queries to retrieve events of interest and the software architecture that integrates these technologies.
- 19
- StarNet: towards Weakly Supervised Few-Shot Object Detection17
Being a few-shot detector, StarNet does not require any bounding box annotations, neither during pre-training nor for novel classes adaptation, and can be applied to the previously unexplored and challenging task of Weakly Supervised Few-Shot Object Detection (WS-FSOD), where it attains significant improvements over the baselines.
- 17
- 16
- 16
- 16
- Fine registration of 3D point clouds fusing structural and photometric information using an RGB-D camera14
A new cost function with dynamic weights for the ICP algorithm that balances the significance of structural and photometric features with dynamically adjusted weights to improve the error minimization process and includes a novel outlier rejection method, based on structural andPhotometric cue.
- Large Scale Generative AI Text Applied to Sports and Music13
An AI commentary system was deployed to produce automated narrations for highlight packages at the 2023 US Open, Wimbledon, and Masters tournaments and was extended to create personalized content for ESPN Fantasy Football and stories about music artists for the GRAMMY awards.
- Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths13
A collaborative strategy to choose a high-precision "teacher" for transferring knowledge to the low-precision "student" while jointly optimizing the model with all bit-widths is proposed and a dynamic block swapping method is developed.
- 13
- 13
- Segmentation of Both Diseased and Healthy Skin From Clinical Photographs in a Primary Care Setting.12
The stark performance improvement with fine-tuning compared to direct transfer and direct training emphasizes both the need for adequate representative data of diseased skin, and the utility of other publicly available data sources for this task.
- 11
- Fusing well-crafted feature descriptors for efficient fine-grained classification11
It is shown that a combination of carefully designed features, including a robust shape descriptor to capture fine morphological structures of objects, as well as traditional color and texture features, is essential for obtaining good performance in this novel approach for classification of subordinate categories.
- Practical computer vision: Example techniques and challenges11
A set of challenging unresolved problems that if solved could spur great progress in practical computer vision and start to extend initially limited areas of competence into a more general-purpose vision toolkit are illustrated.
- 10
- 9
- The Excitement of Sports: Automatic Highlights Using Audio/Visual Cues.9
This work proposes a novel approach for auto-curating sports highlights, and demonstrates it to create a first of a kind, real-world system for the editorial aid of golf and tennis highlight reels.
- IBM Research and Columbia University TRECVID-2013 Multimedia Event Detection (MED), Multimedia Event Recounting (MER), Surveillance Event Detection (SED), and Semantic Indexing (SIN) Systems.9
The current IBM-Columbia MER system exploits nine observations about human cognition, language, and visual perception in order to produce an effective video recounting of an event, and achieves better results on the retrospective and interactive tasks than last year.
- 9
- 9
- Specular reflection reduction using a multi-flash camera9
A novel method to reduce the effect of specularities in digital images is described, based on taking successive images of the scene with different light sources and exploiting image reconstruction from gradient fields to obtain a specular-reduced image.
- 8
- 8
- 8
- 8
- 8
- 8
- 7
- 7
- 7
- 7
- 7
- 7
- 7
- 7
- 7
- 6
- 6
- 5
- 5
- 5
- 5
- 5
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- 3
- 3
- 3
- 3
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- IBM-Northwestern@Trecvid 2014: Surveillance event detection1
An interactive approach to visualize data with temporal relations and a novel risk ranking method to differentiate detection results and present more informative ones to the user for better interaction are proposed.
- 1
- 1
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-16. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.