Transformers v5.19.0 Ships Google's Multimodal EmbeddingGemma 2

A multimodal embedding model from Google built on the Gemma 4 architecture. It encodes text, images, audio, and video individually or combined into a shared 768-dimensional vector space, and supports truncation to 512, 256, or 128 dimensions via Matryoshka Representation Learning.

Several breaking changes apply: MoE models that compute router logits now return them when output_router_logits=True, the \"paged|\" prefix for SDPA and Flash Attention is deprecated, and OWLv2 image query selection changed. Review your code before upgrading.

Yes. Unified multimodal embeddings simplify cross-modal retrieval for government archives and media search, while expert parallelism and continuous batching improvements lower the cost of running large MoE models on local infrastructure.

A token-dispatch implementation was added, selected via the ep_dispatch_experts plan rule and now the default for Qwen3 MoE and Mellum. It removes the requirement that EP size equal TP size, and the Trainer was adapted to work with expert parallelism.