Embeddings
Understanding embeddings for AI applications: text, image, and multi-modal embeddings with similarity search and clustering.
Last updated
Embeddings
Embeddings turn messy, unstructured data — text, images, code — into dense vectors where meaning becomes geometry. They are the backbone of RAG, search, recommendations, and clustering, so getting them right sets the ceiling for everything downstream.
An embedding places each piece of content at a point in high-dimensional space so that similar meaning becomes nearby position. Once meaning is geometry, hard problems turn easy: “find related docs” becomes “find nearby points,” and “is this on-topic?” becomes “measure the angle.” Everything in this section is a consequence of that one idea.
17.1 — What Are Embeddings?
An embedding is a learned mapping from a high-dimensional, discrete input space (words, sentences, images) into a continuous, lower-dimensional vector space. Each input becomes a fixed-size array of floating-point numbers — typically 256 to 3072 dimensions — where geometric proximity encodes semantic similarity.
The geometry of meaning: In a well-trained embedding space, vectors for "dog" and "puppy" sit close together, while "dog" and "refrigerator" are far apart. This isn't hand-coded — it emerges from training on massive corpora where the model learns co-occurrence patterns, contextual relationships, and latent structure.