Segment anything model 3

The core innovation of SAM 3.1 is object multiplexing, which allows the model to simultaneously track up to 16 objects in a single forward pass. Previously, each object required an independent forward pass, causing computational resource consumption to scale linearly with the number of objects. By multiplexing, redundant computation is eliminated and memory bottlenecks are alleviated.

This change directly brings speed improvements: for videos with a moderate number of objects, processing throughput increases from 16 frames per second to 32 frames per second (on a single H100 GPU), doubling the speed. Meta states that this makes real-time object tracking possible in complex videos while reducing GPU resource requirements, enabling high-performance applications to run on smaller, more accessible hardware.

SAM 3.1 adopts a global reasoning approach, processing all tracked objects as a whole rather than independently. This holistic perspective not only simplifies the computational pipeline but also improves tracking accuracy in crowded scenes.

Global reasoning works synergistically with object multiplexing: multiplexing reduces redundant computation, while global reasoning ensures the model does not lose the ability to distinguish individual objects when sharing features. Meta emphasizes that this combination significantly improves processing efficiency while maintaining high accuracy.

SAM 3.1 is an updated version of SAM 3, which itself introduces promptable concept segmentation capabilities, supporting text prompts (open-vocabulary short noun phrases) and image example prompts, breaking the limitation of fixed label sets. This enables the model to segment nuanced concepts like 'striped red umbrella' rather than just common categories like 'person'.

To evaluate this capability, Meta created the Segment Anything with Concepts (SA-Co) benchmark dataset, covering promptable concept segmentation in images and videos. SAM 3 achieves a 2x performance improvement over existing systems on this benchmark. Additionally, SAM 3 retains visual prompts (masks, boxes, points) from SAM 1 and SAM 2, further enhancing flexibility.

Meta also built a novel data engine combining AI and human annotation to acquire large-scale, multi-category, high-quality data with segmentation masks and text labels. This engine addresses the scarcity of such annotations in web data, especially the time-consuming task of obtaining complete mask annotations for each object category in videos.

SAM 3 and SAM 3D have been used in Facebook Marketplace's 'View in Room' feature, helping users visualize how home decor items (like lamps or tables) will look and fit in their own space before purchasing.

In creative tools, SAM 3 will soon be integrated into Instagram's Edits app, allowing creators to apply dynamic effects to specific people or objects in videos with a single tap. Additionally, the Vibes feature in Meta AI app and the meta.ai website will introduce new creation experiences based on SAM 3, including AI visual creation tools and remixing features for AI-generated videos.

Meta has also partnered with Conservation X Labs and Osa Conservation to release the first publicly available wildlife monitoring video dataset using SAM 3.

SAM 3.1 is released as a drop-in replacement for SAM 3, with model weights, updated codebase, and research paper made publicly available. Users can experience the model's capabilities online through the Segment Anything Playground.

SAM 3's model weights, evaluation dataset (SA-Co), and fine-tuning code have also been released. Meta encourages the community to download and use them, and to explore applications in creative media editing.