The 4M‑21 framework supports 21 different vision modalities, including SAM, bbox, pose, and others, in a single model.
It can function as a generative model in addition to being a predictive vision model.
The framework is fine‑tuned for unseen downstream tasks or for new input modalities without extra engineering.
I chose a unified “any‑to‑any” architecture to simplify development and enable cross‑modal transfer, showcasing the breadth of 21 modalities and adding generative capability to broaden applicability.
I discovered the 4M framework – Massively Multimodal Masked Modeling – which tokenizes every modality into a shared space and trains a Transformer encoder‑decoder with a multimodal masked‑pre‑training objective.
4M stands for Massively Multimodal Masked Modeling. Massively Multimodal Masked Modeling