Definition
The process of querying a vector database using one data modality (e.g., text) to retrieve semantically relevant assets in a different modality (e.g., images or audio) via a shared latent embedding space. This approach facilitates Multimodal RAG but requires a trade-off between the high computational cost of dual-encoder alignment and the loss of modality-specific granular features.
Finding non-text assets using text, rather than just generating text descriptions of images.
"A Rosetta Stone for a library where a single index system uses the same coordinates to map a written word, a photograph, and a sound recording."
Conceptual Overview
The process of querying a vector database using one data modality (e.g., text) to retrieve semantically relevant assets in a different modality (e.g., images or audio) via a shared latent embedding space. This approach facilitates Multimodal RAG but requires a trade-off between the high computational cost of dual-encoder alignment and the loss of modality-specific granular features.
Disambiguation
Finding non-text assets using text, rather than just generating text descriptions of images.
Visual Analog
A Rosetta Stone for a library where a single index system uses the same coordinates to map a written word, a photograph, and a sound recording.