Beyond CLIP

TL;DRFour claims

Part IThe C–L–I Framework

A multimodal retriever maps a heterogeneous item (image, text, page, clip, audio, …) into a geometry in which semantically similar ≈ geometrically close. Every architectural choice trades along three near-orthogonal axes:

Four representative systems, projected on the triangle:

Three load-bearing properties of the frame, each with its falsifier. The axes are near-orthogonal but not independent — pushing C weakly helps L and I; pushing I barely helps C or L; most papers concentrate gains on one axis. A single architectural change that lifts all three by comparable magnitudes at fixed budget would refute this; none has been reported. Every benchmark selects an axis. MMEB-V2 is I-heavy; ViDoRe is L-heavy; MIEB tries to balance C across eight categories; Flickr/MSCOCO weight C almost alone. A benchmark whose per-task scorecard cleanly separates C/L/I contributions would falsify the selectivity claim; MIEB gestures toward this but does not close it. Production stacks always hybridize. The 2026 retrieval pipeline is C-axis first stage (SigLIP 2 / jina-clip-v2 / MetaCLIP 2) + L-axis structured stage for document corpora (ColQwen3 / Nemotron-ColEmbed-V2) + I-axis reranker or reasoner (Qwen3-VL-Embedding / Think-Then-Embed / MLLM-as-judge). A single-model deployment that dominates heterogeneous workloads without a separate reranker or dual index would falsify this; its appearance would reorganize the field.

Five years of history, in three moves

Part IIThe Two Unlocks: MLLM Encoders & Late Interaction

Era 3 is defined by two structural changes that happen to be near-orthogonal on C–L–I: a causal decoder turning out to be a better C+I encoder than any bidirectional alternative, and late interaction turning out to be affordable at scale. Each solves a problem the other cannot touch. The modality gap — a long-standing worry about the CLIP geometry — is dissolved as a side effect of both.

2.1 Why a causal MLLM is secretly a good encoder

On paper, using a causal decoder for retrieval is strange. Attention is triangular; the last token sees everything but nothing sees the last token; retrieval similarity is symmetric and causal LMs are asymmetric by construction. Yet E5-V, VLM2Vec, GME, Seed1.6-Embedding, Qwen3-VL-Embedding, KaLM-Embedding, MoCa, RzenEmbed — all follow the same recipe (run a forward pass, pool the last hidden state, done) and beat every bidirectional alternative at comparable scale. Four mechanisms explain why; a fifth explains where it breaks.

The last-token bottleneck is a feature, not a bug. Next-token prediction forces every preceding token to contribute signal to the final position — otherwise the predictor cannot be coherent. A well-pretrained causal LM therefore already contains, hidden inside it, an encoder whose output is the last hidden state: a task-agnostic compressed summary. Contrastive fine-tuning rotates this summary into retrieval-friendly subspace; it rarely has to build the summary from scratch. This is Jiang et al.'s argument for text (LLM2Vec); it generalizes to multimodal inputs because cross-modal tokens pass through the same residual stream.

Cross-modal interleaving shares the compression subspace across modalities. In CLIP, image and text hidden states are produced by different networks glued by a shared projection. In an MLLM embedder, they pass through the same transformer stack, interleaved in a single residual stream, with attention free to mix them at every layer. The functional subspace used for retrieval is therefore shared by construction. This is the C-axis property that dissolves the modality gap (Liang et al., NeurIPS 2022) at the pooled token: CLIP's narrow disjoint cones are a free-energy minimum of the dual-tower contrastive loss landscape — no gradient reason to close them. MLLMs do not abolish the gap (hidden states still cluster by modality layer-by-layer) but they relocate it to the subspace used for retrieval, which is all retrieval needs. The papers that explicitly address modality geometry — MoCa, UniME, Kuaishou's Modality Curation — pick up additional points on cross-modal compositional queries where the residual gap still bites.

Instruction prompts act as soft task heads. The E5-V prompt Summarize the above image in one word: does the job of a "task head" in CLIP's text tower — it tells the model which retrieval task it is performing and empirically routes information through different parts of the last hidden state. MMEB-style per-task instructions generalize this: the encoder learns a small, fast router from instruction to retrieval subspace. This is pure I-axis capability with no CLIP analogue.

The minority report: bidirectional beats causal at the margin. Two independent groups now show that flipping continual pretraining to bidirectional attention plus a masked or EOS-reconstruction pretext recovers measurable points. MoCa (arXiv 2506.23115, Renmin U. + Microsoft Research Asia) runs a modality-aware continual pretraining stage with bidirectional attention + text MLM + image MAE, then contrastive fine-tuning: MoCa-7B reaches 71.5 on MMEB-V1 vs 67.1 for a bidirectional-only baseline (≈ +4.4) and vs 69.8 for mmE5. CoCoA (arXiv 2603.01471, ICT/CAS + Baidu) replaces the pretext with collaborative attention + EOS-based reconstruction, reaching 70.6 at 7B. The effect is real, reproducible across objectives, and cheap — roughly one extra pretraining epoch. Open base models (Qwen3-VL, InternVL3) are still purely causal. This is low-hanging fruit; see §VI for why it lands by Q4 2026.

The 2026 turn: compute before the embedding. A newer family shows the next axis beyond bigger encoders. Think-Then-Embed (TTE) (arXiv 2510.05014, Meta + UCF + NYU, ICLR 2026) first generates a reasoning trace with a small MLLM reasoner, then conditions the embedder on both the query and the trace — reported 7% absolute MMEB improvement over recently proposed baselines on the 78-task average. UniME-V2 (arXiv 2510.13515, Alibaba, AAAI 2026) replaces the brittle similarity heuristic in contrastive hard-negative mining with MLLM-as-a-judge soft scoring. RzenEmbed (arXiv 2510.27350, Qihoo 360) adds hardness-weighted contrastive loss; AutoThinkRAG (arXiv 2603.05551) routes by query complexity and reports 82.13% on DocBench with 18.9% fewer per-query tokens than naïve multimodal RAG. Each is a migration of compute from training to inference, from encoder to reasoner — the same dynamic that played out in chat LLMs with o1/Claude reasoning modes, arriving in retrieval roughly one year later. The 2026–2027 recipe almost certainly combines causal MLLM backbone + bidirectional continual pretraining + inference-time reasoning trace + MLLM-as-judge hard-negative mining. Anyone selling one lever is selling a quarter of the answer.

2.2 Late interaction as within-item localization

ColPali / ColQwen3 / ColNomic / Nemotron-ColEmbed-V2 dominate visual-document retrieval (Nemotron-ColEmbed-V2-8B leads ViDoRe V3 at 63.42 NDCG@10 per NVIDIA's Feb-2026 pipeline snapshot) and barely move Flickr30K / MSCOCO text↔image retrieval. Under C–L–I this is exactly what you expect: a PDF page rendered to pixels produces ~1,000 patches, each aligned to a retrievable sub-unit (word, cell, axis label). MaxSim over query tokens against these patch tokens is ColBERT on a document that happens to be pixels. On photographs the same patches correspond to texture and lighting — units that are not individually retrievable. Page queries are multi-concept ("Q3 revenue in Europe") and MaxSim aggregates multi-token relevance; photo queries are typically single-concept and a single global vector losslessly compresses them.

This is a framing point the community keeps missing: the architecture community calls this "late interaction"; the retrieval community should call it within-item localization. It is a different problem that happens to share the dense-vector toolkit. L-axis is useful only when items have queryable sub-units; on anything else, the ~1000× storage tax is unrecoverable.

2.3 The MUVERA era

Late interaction's Achilles' heel was always storage. At 10M pages × ~1,030 tokens × 128-d fp16 ≈ ~2.6 TB of raw vectors before any compression, and MaxSim compute scales linearly in tokens. Three waves of compression closed the gap:

MUVERA's mechanism is worth stating plainly: it projects a variable-length multi-vector set into a single fixed-dimensional vector compatible with any standard ANN index, preserving MaxSim-style relevance under inner product. The breakthrough was not published in 2024 — it was that 2025–2026 DB integration turned the model-side choice into a configuration toggle. Weaviate 1.31 exposes it as a first-class Encoding.muvera(...) option; Qdrant's FastEmbed line treats it as an encoder-side post-processing step; Vespa provides native tensor MaxSim; LanceDB and Milvus are catching up. Before this wave, L-axis retrieval was a tool for medium, static, high-value document corpora. After it, L-axis retrieval is a candidate for large, dynamic corpora — and the C-axis single-vector default starts looking weaker on document-heavy workloads.

The 2026 practitioner defaults fall out cleanly. Natural-image retrieval (photos, social content, products) stays single-vector; late interaction offers no material gain and still costs ~30× storage even post-MUVERA. Visual-document retrieval (PDFs, slides, UI screenshots, scans, charts, diagrams) is now decisively late-interaction; ColQwen3-7B or Nemotron-ColEmbed-V2-8B is mainstream where it was niche 18 months ago. Mixed corpora (a web page with both a photo and a table) demand a two-track index — single-vector for the photo, multi-vector for the document segments. No single model handles both optimally, and MUVERA does not change that — it just makes the different problem affordable at scale.

Part IIIData, Instruction, Benchmarks: Why the Numbers Lie

This section merges three debates that are usually treated separately but are in fact one: how do we know a model is good, and what does "good" reward? Data, instruction conditioning, and benchmark design are the three knobs through which a paper's headline number gets manufactured, and all three are now contaminated enough that a single score without provenance is indistinguishable from a marketing claim.

3.1 The data stratification, and its ceiling

Every paper since 2024 attributes >80% of gains to data engineering. This is correct in a narrow regime and overclaimed everywhere else. The decomposition that holds across regimes:

This resolves the apparent contradiction between "scale is all you need" (DataComp, LAION) and "data quality is all you need" (DFN, Seed1.6). Both are right at different altitudes. The MegaPairs → mmE5 → Modality-Curation arc is the one that matters for 2026 practitioners: MegaPairs (BAAI, Dec 2024) synthesized hundreds of millions of image-image-text triplets with no human labelers, ending the "labelling-bounded" myth; mmE5 (Renmin U. + Microsoft, Feb 2025) showed multilinguality is nearly free if you synthesize; Modality Curation (NEU + Kuaishou, May 2025) showed batch composition matters, yielding free MMEB points. The collective — and rarely stated — implication: the data pipeline has replaced the architecture as the primary IP of an embedding lab. "Reproduce from our weights and code" has quietly become the gold standard, because "reproduce from scratch" is no longer feasible outside well-funded labs.

The ceiling nobody quite admits: once the 78-task MMEB mixture is saturated with synthetic + mined pairs, the marginal point gets expensive enough that inference-time compute (reasoning, MLLM-as-judge rerank, test-time retrieval) is now cheaper per point than more training data. This is the regime TTE, UniME-V2, AutoThinkRAG exploit. Expect the 2026–2027 paper balance to tilt back toward architecture + inference, away from data volume — a full cycle from 2022.

3.2 Instruction conditioning: product feature disguised as modeling technique

Every competitive 2025–2026 model exposes an instruction field. MMEB-V2 bakes 78-task instructions into the eval. The quiet, ugly consequence: the model learns to branch on the instruction, and leaderboard numbers start reflecting how well it switches tasks rather than how well it represents content. Strip the instruction and many 2026 SOTA models drop several points on zero-shot — a gap that rarely appears in press releases. Small prompt rewordings can move NDCG@5 by several points; this is under-reported. Three observations that practitioners keep relearning the hard way: instructions help only if your workload actually provides them; zero-shot and instructed retrieval are different skills (MagicLens, UniME, ColPali do both well; heavily-instructed models lean on the prompt branch); style drift is a production risk. Under C–L–I, instruction conditioning is a pure I-axis lever — valuable where your workload has task heterogeneity, a net loss where it does not.

3.3 Benchmarks: saturation, contamination, and what a score actually means

From MMEB-V1 (Nov 2024) to MMEB-V2 (Apr 2026), the top score moved explosively in the first year and is now saturating. Three symptoms. Per-task ceiling effects — on the easier MMEB-V2 tasks, the top-20 models are within 1–2 points, and further progress is noise. Instruction-format overfitting — several submissions gained points by tuning the MMEB task-prompt formatting without retraining the encoder. Data leakage — the current generation is trained on corpora that plausibly include MMEB-related train splits (MMEB reuses M-BEIR, UniIR, and public retrieval sets). Not fraud; the normal hygiene problem every benchmark eventually faces (ImageNet, GLUE, SuperGLUE). But in a field moving this fast, leaderboards overstate generalization; an external production deployment typically sees the public MMEB-V2 score minus 3–5 points.

Under C–L–I, the differences between benchmarks stop being mysterious:

Cross-leaderboard snapshot — April 2026

Every number is a single-configuration snapshot from a primary source (leaderboard CSV, model card, or technical report). They move week-to-week. Gaps <2 points are noise.

Why two MMEB-V2 columns for some rows? The leaderboard CSV penalizes models that skip subsets more than the authors' own Table numbers do. For a fair within-column comparison, pick the CSV. For a paper-claim comparison, pick the Table. Always name which. Corrections to community folklore: RzenEmbed and UniME-V2 do not sit at MMEB-V2 ~77; TTE is mid-70s, not 76; the 76.9 sometimes quoted for Seed1.6 is the Qwen3 paper's 78-task recomputation, not a MIEB ranking. Never quote a single number without naming the axis, the CSV-vs-paper distinction, and whether the evaluation used held-out splits. This is not pedantry — it is the only defense against a saturating benchmark.

Part IVDeployment Economics

Leaderboards never show the numbers that dominate a production budget. What follows is a single-configuration baseline on one H100 SXM, fp16, batch tuned per model, CUDA 12.x, realistic image distribution. Treat ±20% as noise.

Three rules of thumb. Every 2× in encoder parameters → roughly 2.5–3× in indexing and 1.5–2× in query latency — MLLMs pay in dynamic resolution and variable-length attention. Multi-vector without compression ≈ 1000× storage vs single-vector; MUVERA cuts this to ~30×, which is the regime where hybrid C+L stacks become a default rather than an exotic choice. Query latency in MLLM encoders is decode-bound — they are still autoregressively producing special embedding tokens, and vLLM/SGLang-style batched decoding offers several-× headroom that most stacks do not yet use. This is the first target of 2026-H2 inference-infra work (see §VI-Near).

Matryoshka + binary is not a free lunch. The marketing claim — "cut storage 64× with <1% quality loss" — holds only on the easiest workloads. MRL truncation (3584 → 512) costs ~1–2% on easy tasks and materially more on compositional / multi-entity / visual-document workloads; 1-bit binary quantization adds percentage points more. Published combined-loss-under-5% results typically come from classification-style probes, not retrieval on hard heterogeneous data. The way to recover the marketing claim is not to deny it — it is to binary-recall + full-precision rerank: retrieve top-100 with MRL+binary, re-score those 100 with fp16. Hybrid Recall@10 sits within ~1% of full precision at ≥32× smaller storage. Matryoshka + binary is a first-stage technique, not a full-stack technique; if you are not reranking, do not binarise.

What breaks when you put ColPali in production. Storage explosion is mitigated by MUVERA but the compressed index still costs ~30× a single-vector corpus — budget accordingly. Serving MaxSim is the DB question, not the model question: Qdrant ships native multivector + MaxSim and is the default for ColPali/ColQwen deployments; Weaviate 1.31 adds native MUVERA via Encoding.muvera(...); Vespa exposes tensor MaxSim; LanceDB has multi-vector columns with client-side MaxSim; Milvus 2.5 supports multi-vector fields with MaxSim as a group_by workaround, with 3.0 on the roadmap; FAISS reduces client-side. Pick the DB on this feature, not on headline benchmarks. Freshness is hard — each document is thousands of inserts, and bulk-reindex cadences that worked for single-vector break here, although MUVERA's compressed encoding updates incrementally like a normal vector. Chart and screenshot brittleness dominate over model choice on low-DPI inputs; input preprocessing matters more than which ColQwen you picked. And MaxSim at high QPS can become the bottleneck, not the encoder — MUVERA's ~90% latency reduction on the BEIR subsets is the only reason this is tractable at scale.

Part VOpen Weights, Closed APIs, and the Moats That Will Remain

The top of almost every 2026 multimodal leaderboard is populated by Chinese labs — Alibaba (Qwen3-VL, GME, UniME), ByteDance (Seed1.6), Tencent (KaLM, LLaVE), BAAI (MegaPairs, BGE), Ant (M2-Encoder, GroupRank), Kuaishou (Modality Curation, UniECS), Xiaohongshu (LamRA), Qihoo 360 (RzenEmbed). This is structural, not cyclical. Three reasons, stated without hyperbole. Strong open-weights MLLM bases in the 2–14B class — Qwen2.5-VL, Qwen3-VL, InternVL3, GLM-4.1V, DeepSeek-VL2 are strong open starting points, and embedding quality inherits directly from base quality. Vertical integration of data pipelines — ByteDance, Alibaba, Kuaishou, Xiaohongshu operate among the world's largest image-text interactive corpora; "data > architecture" structurally favors them. Incentive asymmetry — Chinese labs tend to publish open weights to stake international-reputation claims, while several Western labs gate their best models behind APIs. Public leaderboards therefore systematically over-represent open-weights Chinese output and undercount closed Western models (OpenAI, Anthropic, Cohere) that do not submit.

The narrative has to be stated carefully. The Western open-weights scene is not empty. Meta released Llama 4 (Scout / Maverick) as natively multimodal open weights in April 2025, and Llama 3.2 Vision earlier; a Llama-4-based embedder is obvious next. LLaVA-OneVision-1.5 (arXiv 2509.23661) competes with Qwen2.5-VL on several benchmarks. Microsoft's Harrier-OSS-27B tops text MTEB v2 at 74.3. Meta FAIR / NVIDIA / Google DeepMind still lead on narrow fronts (Perception Encoder, SigLIP 2, PS3, Nemotron-ColEmbed-V2). Salesforce + Waterloo (VLM2Vec), IBM (DAC), Apple (DFN, VeCLIP), Jina AI, Snowflake (Arctic-Embed 2.0), Cohere (Embed v4), Voyage (voyage-multimodal-3), Nomic all ship production-grade multimodal or text embedders outside China. The honest summary: the 2–14B open-weights multimodal embedding space is disproportionately Chinese, and the production-API breadth space is disproportionately Western. Neither dominates the other end.

Where closed APIs will keep their moats into 2027–2028 is the set of modalities that are data-gated rather than compute-gated. Audio embeddings and unified T+I+V+Audio+PDF: Gemini Embedding is the only credible unified multimodal product as of April 2026 — the experimental gemini-embedding-exp-03-07 reaches mean-task ≈ 68.32 on the Multilingual MTEB leaderboard, and no open-weights peer matches its modality breadth. Long-context PDFs (>200 pages, mixed text + figures) — Gemini and Voyage-multimodal-3 have bespoke tokenizers; open weights lag. 3D / CAD / point clouds — almost entirely absent from the open ecosystem. Tabular + image joint retrieval — Cohere Embed v4 handles text + images + mixed PDFs; truly native tabular encoding is rare on both sides. These are smaller markets than photo search, but they are where the closed-API moat compounds, because the training corpora themselves are not open-sourceable.

Part VIForecast 2026 → 2030 · Near, Mid, Far

Forecasts are only useful if they are falsifiable. Each bet below names a mechanism, a signal to track, and a falsifier. Probabilities are subjective and calibrated against the author's track record in adjacent fields. Brier-score them in 2027, 2028, 2030. Many bets in the far tier are load-bearing for the thesis of the guide itself: if the far tier is wrong, the C–L–I frame is probably over-stated.

This horizon is about where compute moves and which plumbing ships. Bets here are dominated by mechanisms already demonstrated in papers and awaiting productization.

The load-bearing implication of the near tier is that the 2026 recipe (causal MLLM + bidirectional continual + reasoning rerank + MUVERA) crystallizes as the default rather than an edge configuration. After that happens, further gains on MMEB-V2 are bounded by benchmark hygiene, not by model capability.

This horizon is about what replaces the leaderboard and what squeezes out of long-context LLMs. Bets here are structural, not model-level.

M1 · Public multimodal benchmarks bifurcate into a "trust tier" (held-out, private split, C/L/I scorecard) and a "marketing tier" (public static leaderboards). Mechanism: saturation + contamination + instruction-format gaming have crossed the threshold where a score is uninterpretable without provenance; institutional buyers (cloud vendors, regulated industries) pay for trusted eval. P ≈ 65%. Falsifier: public MMEB / MTEB remain the referenced source for enterprise RFPs through 2028.

M2 · Long-context LLMs (>4M tokens) absorb a meaningful slice of "retrieval" tasks that are currently RAG. Mechanism: the point where dumping a 500-page document into context beats retrieving three chunks has moved from "theoretical" to "Gemini 2.5 in production" — the remaining economic constraint is inference cost, which halves roughly every 9–12 months. P ≈ 55% that >20% of current small-corpus RAG workloads migrate to in-context by end-2028. Falsifier: token-per-dollar inference cost fails to drop by 4× vs April 2026.

M3 · The line between "retrieval" and "tool use" dissolves at the agent layer. Mechanism: MCP-style tool protocols already treat "search the KB" as one capability among many; reasoning-augmented retrievers (§II.1) already run inference-time logic that is indistinguishable from a tool call. P ≈ 70% that the default agent framework by 2028 treats retrieval as a capability rather than a separable pipeline stage. Falsifier: standalone RAG frameworks (LlamaIndex, Haystack) continue as dominant paradigm and agent frameworks stay separate.

M4 · A unified open-weights T + I + V + Audio embedder reaches within 2 pts of Gemini Embedding's multilingual MTEB score on held-out splits. Mechanism: the data gap closes slowly but the model-side gap is one pretraining run. P ≈ 45% by end-2028. Falsifier: no open-weights model matches Gemini on any unified eval by 2028.

M5 · Embodied retrieval — for robots, AR glasses, and offline agents — becomes a distinct sub-field. Mechanism: the scene in front of a wearable device is a new kind of "corpus" (spatially indexed, time-streamed, identity-bound), and existing embedding stacks do not model the spatial or temporal structure. Early signals: Meta Aria, visual SLAM + VLM hybrids, Twelve Labs' video work. P ≈ 60% that a benchmark for this lands by 2028. Falsifier: no such benchmark exists and the use case remains niche research.

M6 · "Retrieval as preference alignment" — the retriever itself is RLHF'd on human or LLM judge signal, not just contrastive-trained. Mechanism: MLLM-as-judge in UniME-V2 is already a weak version; the next step is closing the loop with user-interaction data. Large consumer platforms (search, social, e-commerce) have the data moat; small labs do not. P ≈ 70% for at least one major retrieval paper per year in this direction starting 2027. Falsifier: contrastive InfoNCE remains the de-facto training objective across all top-10 papers through 2028.

M7 · Chip and energy geography splits the embedding stack into two non-interoperable lineages. Mechanism: export controls, national-AI policies, and domestic chip supply lines (Ascend / Cambricon vs NVIDIA) combine with sovereign data rules (EU AI Act, China's generative-AI provisions) to make cross-border deployment of certain embedder + corpus combinations legally or operationally infeasible. P ≈ 55% that by end-2028 a "sovereign-embedding" tier of vendors explicitly markets on compliance + locality. Falsifier: one global open-weights stack continues to dominate without legal fragmentation.

This horizon is where the C–L–I frame itself comes under pressure. Far-tier bets are deliberately large-scope — falsifying them would reshape the field, not just reorder a leaderboard.

F1 · Embedding as a standalone artifact enters its late phase; "retrieval" becomes a subset of agent memory. Mechanism: four convergent trends kill the standalone embedder for new projects while leaving installed fleets alive for years. First, long-context LLMs swallow small-corpus RAG. Second, reasoning-augmented retrievers push the embedder into the reasoner's latent state — Think-Then-Embed generalizes to "embedding is a function of the reasoner's working memory at the moment of retrieval", not of the document alone. Third, agent memory architectures (episodic + semantic + procedural, à la Voyager, MemGPT, Letta) treat dense vectors as one substrate among several — structured KV, symbolic indices, code-as-memory. Fourth, MLLM-as-judge reranking eats the top of the quality distribution, leaving the embedder as a cheap candidate-gen layer. The practical effect: by 2030, greenfield retrieval projects will look like agent + tool registry + hybrid memory rather than encoder + ANN + reranker; "embedding model" becomes a commodity inside the agent, not the center of the stack. P ≈ 60%. Falsifier: standalone embedding APIs (OpenAI, Cohere, Voyage, Jina) grow their embedding-specific revenue share through 2030.

F2 · The CLIP / SigLIP / ColPali generation becomes "classic layer" infrastructure — paid for, under-maintained, everywhere. Mechanism: even as the frontier moves to agent memory, the installed base of vector indices, product search engines, recommendation pipelines, and academic benchmarks running on single-vector and ColPali encoders is measured in exabytes of persisted vectors. Migration cost is gigantic. Every previous ML generation (word2vec, ELMo, BERT base, ResNet-50) survived a decade past its SOTA obsolescence in production; this one will too. P ≈ 85%. Falsifier: a migration event (a major cloud deprecating CLIP-era encoders, a forced standardization) wipes the installed base by 2030.

F3 · Evaluation moves from static public leaderboards to living benchmarks and private evals; publication norms follow. Mechanism: (M1) plus the gradual realization that any public test set, once named in a tweet, is compromised within one training cycle. Living benchmarks — dynamically generated queries against held-out corpora, with periodic rotation — become the only reputationally defensible eval. Academic papers start citing distributions of scores rather than single numbers. P ≈ 55%. Falsifier: MMEB-V4 / MTEB v3 dominate publications in 2029 on a static-leaderboard model.

F4 · Governance and auditability become first-class features of embedding stacks. Mechanism: embeddings leak memorized content (membership inference is easier than on generative LLMs); vector databases hold personal data; multimodal embedders have copyright exposure on training images. Enterprise procurement already asks "is your training data auditable"; regulators will. By 2030, a production embedder ships with a training-data provenance document, a membership-inference bound, and a differential-privacy story, or it does not get bought by regulated industry. P ≈ 70%. Falsifier: no major enterprise procurement in 2029–2030 requires audit trails for embedding models.

F5 · The "one embedding per item" assumption dies for a large class of items. Mechanism: a single-vector representation of "a video", "a codebase", "a person", "a scientific paper" is a dimensional straitjacket. The successor is a bundle of representations (spatial, temporal, semantic, stylistic, metadata) whose relevant subspace is selected dynamically by the querying agent — late interaction generalized to arbitrary attribute axes. MUVERA-style compression extends from "multi-token" to "multi-facet". P ≈ 50%. Falsifier: single-vector remains the dominant format for 90%+ of new production deployments in 2030.

F6 · An embedding-free retrieval architecture beats a dense one on a major benchmark. Mechanism: hybrid stacks of structured indices (code ASTs, scene graphs, symbolic memory), LLM-generated query-specific indices, and neural attention over raw corpora — rather than pre-computed dense vectors — reclaim the top of some leaderboard. Precedent: SPLADE in text showed sparse can beat dense; the multimodal analogue has not landed but is plausible. P ≈ 40%. Falsifier: every top-5 entry on every major retrieval benchmark through 2030 uses pre-computed dense vectors as the primary primitive.

F7 · Embedding geography bifurcates into distinct technical lineages, not just distinct vendors. Mechanism: M7 compounds over five years. By 2030, the Western lineage optimizes for API breadth + governance; the Chinese lineage optimizes for open-weights scale + domestic deployment; a third lineage (EU / sovereign clouds) optimizes for auditability + privacy. Interoperability exists at the interface level (vectors are vectors) but not at the training-data or compliance level. P ≈ 45%. Falsifier: a single global embedding stack (most likely a hyperscaler's) dominates all three geographies in 2030.

F8 · Human-in-the-loop retrieval is the dominant consumer paradigm; vector similarity is invisible to the user. Mechanism: the "search bar returns ten links" UX is already eroding — Perplexity, ChatGPT Search, Google AI Mode replace it with a conversational reformulation loop. In that loop, embedding is a background primitive, not a product surface; the product is the conversation. P ≈ 75%. Falsifier: classical search UX (query → ranked list) remains the dominant consumer modality in 2030.

Bets I am not making

For calibration, three non-bets for the 12–24 month window. No single MLLM encoder will "rule them all" — the C–L–I triangle is real and stacks will hybridize, not collapse. No >82 MMEB-V2 model will win on all tasks equally — saturation plus contamination make this a diminishing-returns problem; specialised models will win specialised axes. Open-weights audio-text embeddings will not reach Gemini Embedding parity by end-2027 — the data gap is too wide.

Where the field is actually stuck — the open problems that would move the far tier

If you want a paper that is read in 2028 rather than filed, pick one of these and actually solve it. These are not "gaps" — they are unsolved problems whose solutions would change the probabilities in the far-tier table above.

1. Long-video retrieval (>30 min). R@1 on LongVideoBench-Retrieval is materially below short-video baselines. Temporal reasoning, scene-change detection, efficient video-token compression are all open. Closed APIs (Twelve Labs Marengo-2.7, Pegasus-1.2) are partial answers.
2. Fine-grained visual grounding at retrieval time. ColPali localizes within a document; nothing localizes within a photo or a long video frame well. Snappy (arXiv 2512.02660) is a patch-to-region propagation partial answer.
3. Calibration across modalities. Cosine similarity of T-T, I-I, and T-I pairs is not on the same scale even after training. Thresholds must be per-modality. No paper formalizes this.
4. Incremental / streaming multi-vector indexing. MUVERA helps, but "add 10K pages to a 10M collection" is still thousands of inserts per document and requires periodic recentering.
5. Compositional generalization evaluation. ARO, MMVP, SPEC are small and somewhat adversarial; a real-world compositional-generalization benchmark would cost more to build than any single lab has funded.
6. Embodied / spatial retrieval. No consensus representation for "the scene in front of me, indexed by location and time".
7. Preference-aligned retrievers. Contrastive InfoNCE is a weak proxy for user utility. Replacing it with a judge-in-the-loop or RL-from-interaction objective at scale is open.

Part VIIReference

7.1 Paper list — Era 1–2 foundations (2021 – mid-2025)

7.2 Paper list — Era 3: MLLM-native + reasoning-augmented (Aug 2025 – Apr 2026)

7.3 Ecosystem matrix — choose by corpus × modality × freshness × axis

Vector DBs & ANN indexes

The question that matters is "multi-vector + MUVERA", not headline QPS.

Rerankers — use one, always

Multimodal RAG frameworks

Stack recipes · April 2026

Each annotated with the axes it pushes.

• 100K product images, millions of queries/day. C only jina-clip-v2 → FAISS IndexHNSWFlat → mxbai-rerank-large-v2 → LLM answer. Skip MLLM encoders and ColPali.
• 100K enterprise PDFs, chart/table/slide retrieval. L+I ColQwen2.5-multilingual / Nemotron-ColEmbed-V2 → Qdrant multivector (or Weaviate 1.31 with MUVERA) → Claude / GPT-4o/5 rerank → VLM answer. The killer app, now affordable at scale.
• 100M social-media images, compositional queries. C first+I rerank SigLIP 2 first stage (MRL + binary) → Qwen3-VL-Embedding-8B rerank on top-1K → VLM answer. Reserve the MLLM encoder for rerank.
• Heterogeneous instruction-heavy RAG traffic. I dominant Think-Then-Embed or UniME-V2 reasoning rerank on top of Qwen3-VL-Embedding / RzenEmbed first stage. Monitor zero-shot vs instructed drift.
• Long-form videos (≥1 hr). C+L open. Marengo-2.7 / Pegasus-1.2 (Twelve Labs) for now; open weights not yet competitive. Re-evaluate Q3 2026.
• Mixed audio + PDF + image. C breadth Gemini Embedding is the credible unified option in April 2026. Budget for API spend.

7.4 Methodology & epistemic hygiene

Primary sources: arXiv papers (linked), Hugging Face model cards, MMEB / MTEB / ViDoRe / MIEB public leaderboards (snapshotted April 2026), official vendor docs. Secondary: author-run benchmarks on an internal H100 SXM node for §IV; single-configuration, ±20% noise. Tertiary: vendor blogs, release notes, conference talks — linked, used only where primary references are unavailable.

Load-bearing vs illustrative. The C–L–I frame and the §VI forecasts are load-bearing — the thesis depends on them. Specific numeric claims about individual models (MMEB scores, latency, storage) are illustrative — accurate to the snapshot, aging within 6–12 months.

Known gaps. A guide that hides its gaps is worse than one that names them. As of April 2026, this document does not cover: adversarial robustness (typographic attacks, watermark poisoning, OOD shift — relevant for moderation, T&S, content-auth pipelines); privacy / membership-inference on contrastive models; audio-text embedding in the depth applied to vision; 3D / point-cloud / CAD; on-device / edge embedding; non-search uses of embeddings (RL reward models, moderation, dedup, near-dup); legal / licensing analysis of training-data provenance. If you build in any of these areas, supplement this guide; do not substitute it.

Freshness contract. Embeddings move fast; so do benchmarks. Treat every number older than this edition as stale. Living leaderboards linked in §7.1–7.2 are authoritative; this guide is a frame for reading them, not a replacement.

AckAcknowledgements

Thanks to the authors of every paper, model, and leaderboard linked here. Particular debt to the MTEB / MMEB / MIEB / ViDoRe benchmark teams for transparent leaderboards; the ColBERT / ColPali lineage for making late interaction legible; the MUVERA authors for making it affordable; the LLM2Vec / E5-V / VLM2Vec / GME / Qwen-VL-Embedding / Seed1.6 teams for the MLLM-as-encoder recipe; and the many practitioners whose deployment post-mortems informed §IV.

Errors and opinions are the author's alone.

CiteCitation

@misc{li2026beyondclip,
  author       = {Wei Li},
  title        = {Beyond CLIP: A First-Principles Field Guide to Multimodal Embedding, from CLIP to the Post-Embedding Agent Stack},
  howpublished = {\url{https://github.com/BIGBALLON/BeyondCLIP}},
  year         = {2026},
  note         = {2nd edition, April 2026}
}