Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Tamara L. BergSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works124 from public data
- UNITER: UNiversal Image-TExt Representation Learning2,731
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- Modeling Context in Referring Expressions1,889
This work focuses on incorporating better measures of visual context into referring expression models and finds that visual comparison to other objects within an image helps improve performance significantly.
- MAttNet: Modular Attention Network for Referring Expression Comprehension1,003
This work proposes to decompose expressions into three modular components related to subject appearance, location, and relationship to other objects, which allows for flexibly adapt to expressions containing different types of information in an end-to-end framework.
- TVQA: Localized, Compositional Video Question Answering821
This paper presents TVQA, a large-scale video QA dataset based on 6 popular TV shows, and provides analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVZA task.
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training573
HELP, a novel framework for large-scale video+language omni-representation learning that achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains is presented.
- UNITER: Learning UNiversal Image-TExt Representations479
UNITER, a UNiversal Image-TExt Representation, learned through large-scale pre-training over four image-text datasets is introduced, which can power heterogeneous downstream V+L tasks with joint multimodal embeddings.
- Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout422
This paper presents a generalizable navigational agent, trained in two stages via mixed imitation and reinforcement learning, outperforming the state-of-art approaches by a large margin on the private unseen test set of the Room-to-Room task, and achieving the top rank on the leaderboard.
- TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval388
The proposed XML model uses a late fusion design with a novel Convolutional Start-End detector (ConvSE), surpassing baselines by a large margin and with better efficiency, providing a strong starting point for future work.
- A Joint Speaker-Listener-Reinforcer Model for Referring Expressions295
A unified framework for the tasks of referring expression comprehension and generation is proposed, composed of three modules: speaker, listener, and reinforcer, which achieves state-of-the-art results for both comprehension andgeneration on three referring expression datasets.
- TVQA+: Spatio-Temporal Grounding for Video Question Answering293
By performing this joint task, the proposed Spatio-Temporal Answerer with Grounded Evidence with STAGE model is able to produce insightful and interpretable spatio-temporal attention visualizations.
- 184
- 148
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models144
VALUE (Vision-And-Language Understanding Evaluation), a set of meticulously designed probing tasks generalizable to standard pre-trained V+L models, aiming to decipher the inner workings of multimodal pre-training.
- Multi-Target Embodied Question Answering137
This work presents a generalization of EQA -- Multi-Target EQA (MT-EQA), and proposes a modular architecture composed of a program generator, a controller, a navigator, and a VQA module that can outperform previous methods and strong baselines by a significant margin.
- Physics-Inspired Garment Recovery from a Single-View Image115
This work proposes a method that is able to compute a 3D model of a human body and its outfit from a single photograph with little human interaction and demonstrates the effectiveness of the algorithm by re-purposing the reconstructed garments for virtual try-on and garment transfer applications and for cloth animation on digital characters.
- AVID: Any-Length Video Inpainting with Diffusion Model105
Any-Length Video Inpainting with Diffusion Model, dubbed as AVID, is introduced, equipped with effective motion modules and adjustable structure guidance, for fixed-length video inpainting and a novel Temporal MultiDiffusion sampling pipeline with middle-frame attention guidance mechanism, facilitating the generation of videos with any desired duration.
- Apollo: An Exploration of Video Understanding in Large Multimodal Models101
A comprehensive study that helps uncover what effectively drives video understanding in Large Multimodal Models (LMMs), and introduces Apollo, a state-of-the-art family of LMMs that achieve superior performance across different model sizes.
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis94
A consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video to encode the optical flow via warping from the first frame and serve it as a supplementary reference in the diffusion model.
- What is More Likely to Happen Next? Video-and-Language Future Event Prediction89
This work collects a new dataset, named Video-and-Language Event Prediction (VLEP), with 28,726 future event prediction examples (along with their rationales) from 10,234 diverse TV Show and YouTube Lifestyle Vlog video clips, and presents a strong baseline incorporating information from video, dialogue, and commonsense knowledge.
- FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks86
This work proposes a novel FAshion-focused Multi-task Efficient learning method for Vision-and-Language tasks (FAME-ViL), which applies a single model for multiple heterogeneous fashion tasks, therefore being much more parameter-efficient.
- Single NIR Laser-Activated Multifunctional Nanoparticles for Cascaded Photothermal and Oxygen-Independent Photodynamic Therapy85
The results exhibit that Bi2Se3@AIPH NPs possesses 31.2% photothermal conversion efficiency for enhanced PTT, ideal free radical generation for oxygen-independent PDT, and 37.77 HU mL mg−1 X-ray attenuation coefficient for CT imaging with high quality.
- Violin: A Large-Scale Dataset for Video-and-Language Inference81
A new large-scale dataset, named Violin (VIdeO-and-Language INference), is introduced for this task, which consists of 95,322 video-hypothesis pairs from 15,887 video clips, spanning over 582 hours of video.
- 74
- 70
- Hierarchically-Attentive RNN for Album Summarization and Storytelling69
This work addresses the problem of end-to-end visual storytelling by using the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets to encode the album photos, select representative photos, and compose the story.
- Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations66
This work proposes to learn video representation that encodes both action steps and their temporal ordering, based on a large-scale dataset of web instructional videos and their narrations, without using human annotations, and significantly advances the state-of-the-art results on step classification and fore-casting.
- 62
- 53
- CommerceMM49
The CommerceMM model achieves state-of-the-art performance on 7 commerce-related downstream tasks after fine-tuning and proposes another 9 novel cross-modal and cross-pair retrieval tasks, called Omni-Retrieval pre-training.
- Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation44
Multimodal Masked Video Generation (MMVG) is proposed, which can address all 3 cases of TVC, including video prediction, rewind, and infilling, by applying corresponding masking conditions.
- 44
- BachGAN: High-Resolution Image Synthesis From Salient Object Layout44
The proposed Background Hallucination Generative Adversarial Network (BachGAN), which leverages a background retrieval module to first select a set of segmentation maps from a large candidate pool, then encodes these candidate layouts via a background fusion module to hallucinate a suitable background for the given objects.
- 43
- 39
- Unsupervised Vision-and-Language Pretraining via Retrieval-based Multi-Granular Alignment38
This paper proposes a novel unsupervised V + L pre-training curriculum for non-parallel texts and images and constructs a weakly aligned imagetext corpus via a retrieval-based approach, then applies a set of multi-granular alignment pre- training tasks to bridge the gap between the two modalities.
- 37
- 34
- Connecting What to Say With Where to Look by Modeling Human Attention Traces33
A novel model architecture that is jointly trained on dual tasks (controlled trace generation and controlled caption generation) is presented and a local bipartite matching (LBM) distance metric which allows the comparison of two traces of different lengths is proposed.
- 33
- 30
- GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval30
This paper introduces a new dataset called Kinetic-GEB+.
- 29
- Tumor acid microenvironment-activated self-targeting & splitting gold nanoassembly for tumor chemo-radiotherapy28
The results show that Au-NNP(RTX), a pH-responsive gold nanoassembly, possesses of high tumor accumulation and effective tumor penetration, thereby enhancing the tumor chemo-radiotherapy efficiency.
- 27
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models25
The novelty of SceneTextGen lies in its integration of three key components: a character-level encoder for capturing detailed typographic properties, coupled with a character-level instance segmentation model and a word-level spotting model to address the issues of unwanted text generation and minor character inaccuracies
- 25
- In situ self-assembled biosupramolecular porphyrin nanofibers for enhancing photodynamic therapy in tumors24
This nanosystem effectively solves the problem of the low retention abilities leading to a poor PS concentration in a tumor, prolonging the treatment time window efficiently after only a single administration and achieving the purpose of PDT enhancement.
- An oxidation responsive nano-radiosensitizer increases radiotherapy efficacy by remolding tumor vasculature22
A strategy to inhibit oxidative stress in tumors by generating chelating agents in response to hydrogen peroxide, thereby inhibiting multi-angiogenic factors simultaneously to normalize blood vessels is developed.
- LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval18
Experiments show the proposed LoopITR approach achieves state-of-the-art dual encoder performance when compared with approaches using a similar amount of data, and highlights the benefits of training the two encoders in the same network.
- 17
- A pH-responsive Pt-based nanoradiosensitizer for enhanced radiotherapy via oxidative stress amplification17
In vivo results indicated that the NPs could increase the tumor inhibition rate to 91.2% with negligible side effects to normal tissues, and results demonstrate that Pt@HSA/CA NPs had outstanding tumor curative efficacy and hypotoxicity.
- 14
- 14
- 14
- 13
- 12
- Construction of a pH/TGase “Dual Key”-Responsive Gold Nano-radiosensitizer with Liver Tumor-Targeting Ability12
A "dual key"-responsive strategy is utilized to construct a transglutaminase/pH-responsive radiosensitizer (Au@TAcoGal) with specific aggregation behavior in hepatic tumor cells with a higher therapeutic effect and tumor inhibition rate.
- 11
- 11
- MCMG simulator: A unified simulation framework for CPU and graphic GPU11
MCMG (Multi-CPU Multi-GPU) simulator is introduced, a cycle accurate, modular and open-source toolset that enables simulating x86 CPUs and Nvidia G80 like GPUs simultaneously, targeting heterogeneous architectural exploration.
- Interfacial Rheological Property and Rheokinetics of a Novel Photoreversible Micellar System10
The interfacial tension and modulus of the micellar system change significantly after UV irradiation and show the reversible interfacial rheological properties when irradiated by visible light.
- ROICtrl: Boosting Instance Control for Visual Generation9
This work enhances diffusion models by introducing regional instance control, where each instance is governed by a bounding box paired with a free-form caption, and proposes ROICtrl, an adapter for pretrained diffusion models that enables precise regional instance control.
- 9
- 9
- 9
- 9
- 9
- 8
- 8
- 8
- 8
- 8
- 8
- 7
- 7
- 7
- 7
- 6
- 6
- 5
- A quantitative quality control method of big data in cancer patients using artificial neural network5
- 5
- 5
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- 4
- 3
- 3
- 3
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 2
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- 1
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
- –
Publication data from OpenAlex, with missing venues and authors filled in from Crossref; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-16. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.