Is this you? Claim this profile to correct it, add a bio and choose the work people see first.
Claim this profileAcademic lineage
Possible advisorsa guess from early papers, not confirmed
- Ran HeSuggested from co-authorship
Is this you? Claim this profile to confirm or dismiss it.
Works82 from public data
- A survey on multimodal large language models1,696
This paper presents the basic formulation of the MLLM and delineates its related concepts, including architecture, training strategy and data, as well as evaluation, and introduces research topics about how MLLMs can be extended to support more granularity, modalities, languages and scenarios.
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis1,687
Video-MME is introduced, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis, and it is revealed that Gemini 1.5 Pro is the best-performing commercial model, significantly outperforming the open-source models with an average accuracy of 75%, compared to 71.9% for GPT-4o.
- Woodpecker: hallucination correction for multimodal large language models296
Implemented in a post-remedy manner, Woodpecker can easily serve different MLLMs, while being interpretable by accessing intermediate outputs of the five stages, and shows the huge potential of this new paradigm.
- Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM215
This paper proposes a novel speech-text multimodal LLM architecture called Freeze-Omni, holding great potential to conduct speech-to-speech dialogue based on a multimodal LLM under the condition of a frozen LLM, avoiding the catastrophic forgetting problem caused by limited data and training resources.
- VITA: Towards Open-Source Interactive Omni Multimodal LLM204
VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience is introduced.
- CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identification174
This paper systematically study the manually designed architectures, and identifies that appropriately separating Batch Normalization layers is the key to bring a great boost towards cross-modality matching, and proposes a novel method, named Cross-Modality Neural Architecture Search (CM-NAS).
- 148
- Aligning and Prompting Everything All at Once for Universal Visual Perception100
APE advances the convergence of detection and grounding by reformulating language-guided grounding as open-vocabulary detection, which efficiently scales up model prompting to thousands of category vocab-ularies and region descriptions while maintaining the ef-fectiveness of cross-modality fusion.
- A Survey on Benchmarks of Multimodal Large Language Models90
A comprehensive review of 200 benchmarks and evaluations for MLLMs, focusing on perception and understanding, recognition and reasoning, specific domains, key capabilities, and other modalities, concludes that evaluation should be regarded as a crucial discipline to support the development of MLLMs better.
- MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs85
This work aims to offer researchers an easy grasp of how to effectively evaluate MLLMs according to different needs and to inspire better evaluation methods, thereby driving the progress of MLLM research.
- Cantor: Inspiring Multimodal Chain-of-Thought of MLLM74
It is argued that converging visual context acquisition and logical reasoning is pivotal for tackling visual reasoning tasks and an innovative multimodal CoT framework, termed Cantor, characterized by a perception-decision architecture is proposed, demonstrating the efficacy of the proposed framework.
- A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise68
A preliminary exploration of Gemini Pro's visual understanding proficiency, which comprehensively covers four domains: fundamental perception, advanced cognition, challenging vision tasks, and various expert capacities, demonstrates the potential of Gemini to be a strong challenger to GPT-4V.
- Cross-Spectral Face Hallucination via Disentangling Independent Factors67
A Pose Aligned Cross-spectral Hallucination (PACH) approach to disentangle the independent factors and deal with them in individual stages to achieve complexion control and consequently generate more realistic VIS images than existing methods.
- 62
- Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding53
This work introduces Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding, and designs a progressive tri-level hierarchy that incrementally increases the complexity of video comprehension.
- VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model42
VITA-Audio is the first multi-modal large language model capable of generating audio output during the first forward pass, enabling real-time conversational capabilities with minimal latency and significantly outperforms open-source models of similar model size on multiple benchmarks.
- Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy40
An effective multi-modal training schema that starts with large language models and proceeds through vision-language alignment, general knowledge learning, and two sequential stages of long-sequence fine-tuning to serve as a competitive baseline and offer valuable insights for the open-source community in advancing long-context multi-modal understanding.
- No Time to Train: Empowering Non-Parametric Networks for Few-Shot 3D Scene Segmentation39
A Non-parametric Network for few-shot 3D Segmentation, Seg-NN, and its Parametric variant, Seg-PN, and its Parametric variant, Seg-PN are proposed, which can alleviate the domain gap issue and save a substantial amount of time.
- Heterogeneous Face Recognition via Face Synthesis With Identity-Attribute Disentanglement31
A new HFR method from the perspective of heterogeneous data augmentation, named Face Synthesis with Identity-Attribute Disentanglement (FSIAD) is proposed, which obtains superior performance than previous HFR approaches.
- 26
- LUCY: Linguistic Understanding and Control Yielding Early Stage of Her23
LUCY is a E2E speech model that senses and responds to user's emotion, senses and responds to user's emotion, delivers responses in a succinct and natural style, and uses external tool to answer real-time inquiries.
- Audio-Driven Dubbing for User Generated Contents via Style-Aware Semi-Parametric Synthesis20
This paper investigates an audio-driven dubbing method that is more feasible for User Generated Content (UGC) production, and develops a semi-parametric video renderer, which takes full advantage of the limited training data of the unseen speaker via a video-level retrieve-warp-refine pipeline.
- InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption19
This work proposes a novel instance-aware structured caption framework, termed InstanceCap, to achieve instance-level and fine-grained video caption for the first time, and designs an auxiliary models cluster to convert original video into instances to enhance instance fidelity.
- Deep momentum uncertainty hashing19
A novel Deep Momentum Uncertainty Hashing (DMUH) that explicitly estimates the uncertainty during training and leverages the uncertainty information to guide the approximation process, and achieves best performance on all datasets and surpasses existing state-of-the-arts by a large margin.
- 18
Publication data from OpenAlex; citation counts are the higher of OpenAlex and Semantic Scholar, last synced 2026-10-16. One-sentence summaries under some papers are written by Semantic Scholar’s model. Citation counts may be lower than on Google Scholar, which indexes more sources.
Report an error
Wrong papers, two people merged into one, or a profile that should not be here? Tell us and we will fix or hide it.