Muse Image and Video
In Meta AI Blog, 2026
Meta’s first in-house image and video generation models, with agentic reasoning, ranking #2 on Arena leaderboards.
rohitgirdhar.github.io
Rohit Girdhar
Research Scientist · AMI Labs
I am a Research Scientist at AMI Labs. My current research focuses on multimodal understanding, generation and world modeling. I obtained a PhD from Carnegie Mellon University (here’s a link to my dissertation), where I worked on learning from and understanding videos. I was previously part of the Meta Superintelligence Labs (MSL) and Facebook AI Research (FAIR) at Meta, and have spent time at DeepMind, Adobe and Facebook as an intern. See here for a formal bio.
News
- Jun 2026
After 7 incredible years, I have moved from Meta to AMI Labs! See more here.
- Jun 2026
Mark Zuckerberg announced Muse Image and Muse Video, the first image/video generation models from Meta Superintelligence Labs, ranked 2nd on multiple Arena evals. I drove key efforts in multi-image editing.
- Jun 2025
EgoVis 2023-24 Distinguished Paper Award for HierVL.
- Oct 2024
- Jul 2024
Mark Zuckerberg announced Llama 3.1, along with our state-of-the-art video recognition capabilities!
- Jun 2024
Invited panelist for the AI for Content Creation (AI4CC) workshop at CVPR 2024 (along with Cynthia Lu and Robin Rombach).
- Jun 2024
LaViLa and Ego4D among the winners of the EgoVis 2022-23 Distinguished Paper Awards!
- Apr 2024
- Apr 2024
Presented Emu Video at RunwayML’s inaugural Research and Art (RNA) event.
- Feb 2024
Invited judge for the MIT Filmmaking Hackathhon 2024.
- Nov 2023
Mark Zuckerberg announced our state-of-the-art video generation work, Emu Video! Also see coverage by TechCrunch, TheVerge, VentureBeat, Reuters, and others!
- Jun 2023
Giving a talk at HVU Workshop and presenting 5 papers at CVPR 2023!
- May 2023
Mark Zuckerberg announced our multimodal embedding work, ImageBind! Also see coverage by TheVerge, Engadget, SiliconANGLE, maginative and others!
- Jun 2022
Presenting 3 papers at CVPR 2022, including Omivore, a single model that obtains state-of-the-art results across 3 different modalities: images, videos and single-view 3D!
- Oct 2021
We announced Ego4D, the largest egocentric video dataset to date! See this video for a quick intro, and see coverage from TechCrunch, TheVerge, Axios, Fast Company, and others!
Career
2026 — Present
AMI Labs
Research Scientist · New York
Building world models that understand real-world sensor data. Research on multimodal understanding, generation and world modeling.
2019 — 2026
Meta FAIR, GenAI & MSL
Research Scientist · New York
Core contributor to Meta’s media generation and multimodal efforts, including Movie Gen, Muse Image, Emu Video, ImageBind and Llama 3. Earlier at FAIR, worked on self-supervised, multimodal and omnivorous representations for video understanding.
Fast, High-Fidelity Video Editing
In Meta AI & Instagram Edits, 2026
Text-prompted, high-fidelity video edits in under 30 seconds — local edits, scene transformations, and composite requests.
LLMs can see and hear without any training
In ICML, 2025
Pure text-only LLMs can use off-the-shelf multimodal embedding models to do various multimodal tasks!
Movie Gen: A Cast of Media Foundation Models
In arXiv, 2024
State-of-the-Art Video (+Audio) Generation Model
The Llama 3 Herd of Models
In arXiv, 2024
State-of-the-Art open-source LLM with multimodal capabilities
Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning
In ECCV, 2024
A simple and effective approach to high-quality video generation by learning to animate high quality images.
ImageBind: One Embedding Space To Bind Them All
In CVPR, 2023 (Highlighted Presentation)
One embedding space for 6 different modalities, enables zero-shot recognition on all modalities!
Learning Video Representations from Large Language Models
In CVPR, 2023 (Highlighted Presentation)
Leveraging LLMs to auto-annotate videos for representation learning.
Omnivore: A Single Model for Many Visual Modalities
In CVPR, 2022 (Oral Presentation)
A single model for images, video and single-view 3D.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
In CVPR, 2022 (Best paper finalist)
The largest egocentric video dataset.
Masked-attention Mask Transformer for Universal Image Segmentation
In CVPR, 2022
Single architecture state-of-the-art in instance, semantic and panoptic segmentation.
Anticipative Video Transformer
In ICCV, 2021
An autoregressive video transformer architecture for action anticipation in videos.
- 2026Muse Image and Video In Meta AI Blog, 2026
- 2026Fast, High-Fidelity Video Editing In Meta AI & Instagram Edits, 2026
- 2026Human detectors are surprisingly powerful reward models In arXiv, 2026
- 2025
- 2025Diffusion Autoencoders are Scalable Image Tokenizers In arXiv, 2025
- 2025LLMs can see and hear without any training In ICML, 2025
- 2024
- 2024Movie Gen: A Cast of Media Foundation Models In arXiv, 2024
- 2024The Llama 3 Herd of Models In arXiv, 2024
- 2024
- 2024
- 2023Generating Illustrated Instructions In CVPR, 2024
- 2023Motion-Conditioned Image Animation for Video Editing In arXiv, 2023
- 2023
- 2023
- 2023ImageBind: One Embedding Space To Bind Them All In CVPR, 2023 (Highlighted Presentation)
- 2023
- 2023
- 2023HierVL: Learning Hierarchical Video-Language Embeddings In CVPR, 2023 (Highlighted Presentation)
- 2022Learning Video Representations from Large Language Models In CVPR, 2023 (Highlighted Presentation)
- 2022
- 2022Omnivore: A Single Model for Many Visual Modalities In CVPR, 2022 (Oral Presentation)
- 2022
- 2022
- 2021Mask2Former for Video Instance Segmentation In arXiv, 2021
- 2021
- 20213DETR: An End-to-End Transformer Model for 3D Object Detection In ICCV, 2021 (Oral Presentation)
- 2021Anticipative Video Transformer In ICCV, 2021
- 20213D Spatial Recognition without Spatially Labeled 3D In CVPR, 2021
- 2021
- 2021Physical Reasoning Using Dynamics Aware Embeddings In ICML Workshops, 2021
- 2020Forward Prediction for Physical Reasoning In ICML Workshops, 2021
- 2026
Summer 2018
DeepMind
Research Scientist Intern · London
Worked with Andrew Zisserman, João Carreira and Carl Doersch on attention-based architectures for recognizing and localizing human actions in video.
Video Action Transformer Network
In CVPR, 2019 (Oral Presentation)
Among the first applications of Transformers to model videos. SOTA results: close 2nd at AVA Challenge, CVPR'18.
Summer 2017
Facebook AI Research
Research Scientist Intern · Menlo Park
Worked with Lorenzo Torresani, Georgia Gkioxari and Du Tran on detecting and tracking people for keypoint estimation in video.
Detect-and-Track: Efficient Pose Estimation in Videos
In CVPR, 2018
Human keypoint tracking approach that ranked first in ICCV 2017 PoseTrack keypoint tracking challenge!
Summer 2016
Adobe Research
Research Scientist Intern · San Francisco
Worked with Josef Sivic and Bryan Russell on learnable spatio-temporal aggregation for action classification.
ActionVLAD: Learning spatio-temporal aggregation for action classification
In CVPR, 2017
Aggregating visual features for action recognition.
2014 — 2019
Carnegie Mellon University The Robotics Institute
PhD in Robotics · Pittsburgh
PhD with Deva Ramanan on learning from and understanding videos; MS with Martial Hebert, Abhinav Gupta, Kris Kitani and David Fouhey as a Siebel Scholar.
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning
In ICLR, 2020 (Oral Presentation)
A dataset to evaluate temporal reasoning in video models.
Attentional Pooling for Action Recognition
In NeurIPS, 2017
Among the first applications of attention for contemporary video/action understanding.
Binge Watching: Scaling Affordance Learning from Sitcoms
In CVPR, 2017 (Spotlight Presentation)
Learning how humans interact with their environment by watching TV.
Learning a Predictable and Generative Vector Representation for Objects
In ECCV, 2016 (Spotlight Presentation)
A single embedding space, good for both generating and understanding 3D models
- 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning In ICLR, 2020 (Oral Presentation)
- 2019MetaPix: Few-Shot Video Retargeting In ICLR, 2020
- 2019
- 2017Attentional Pooling for Action Recognition In NeurIPS, 2017
- 2016Binge Watching: Scaling Affordance Learning from Sitcoms In CVPR, 2017 (Spotlight Presentation)
- 2016Learning a Predictable and Generative Vector Representation for Objects In ECCV, 2016 (Spotlight Presentation)
- 2019
Summer 2013
Facebook
Software Engineering Intern · Menlo Park
First stint at Facebook, working on Graph Search.
2010 — 2014
IIIT Hyderabad
B.Tech in Computer Science · Hyderabad, India
Undergraduate research with C. V. Jawahar at CVIT on large-scale image retrieval and computer vision.
Code
meta-llama/llama3
The official Meta Llama 3 release
29k
facebookresearch/ImageBind
One embedding space for six modalities
9.1k
facebookresearch/Mask2Former
Universal image segmentation
3.4k
facebookresearch/Detic
Detecting twenty-thousand classes
2.0k
facebookresearch/CutLER
Unsupervised object detection and segmentation
1.1k
facebookresearch/DetectAndTrack
Pose detection and tracking in video
998
facebookresearch/3detr
End-to-end transformer for 3D detection
711
facebookresearch/Ego4D
The largest egocentric video ML dataset
621
frank-xwang/InstanceDiffusion
Instance-level control for image generation
614
facebookresearch/omnivore
One model for images, videos and 3D
573
facebookresearch/LaViLa
Video-language models with LLM narrators
535
facebookresearch/MILS
LLMs can see and hear without any training
460
facebookresearch/DepthContrast
Self-supervised pretraining for 3D point clouds
274
rohitgirdhar/AttentionalPoolingAction
Attentional pooling for action recognition
259
rohitgirdhar/ActionVLAD
Spatio-temporal aggregation for action classification
218
yinboc/dito
Diffusion tokenizers for image generation
168
facebookresearch/AVT
Anticipative Video Transformer
154
rohitgirdhar/CATER
Diagnostic dataset for temporal reasoning
112