Home / Articles / Engineering
Engineering

RAG and Vector Search: Grounding LLMs in Your Data

Embeddings, chunking, hybrid retrieval and reranking — a practical guide to retrieval-augmented generation.

By SiliconMind EditorialNov 202612 min read
Key takeaways
  • RAG retrieves relevant documents at query time and adds them to the prompt so answers are grounded and current.
  • Embeddings map text to vectors; semantically similar text lands close together.
  • Chunking strategy, hybrid search (vectors + keywords) and reranking matter more than the choice of vector database.
  • Long context windows complement RAG rather than replace it: retrieval still controls cost, latency and relevance.
  • Evaluate retrieval and generation separately — recall@k for search, faithfulness for answers.

A language model knows what was in its training data — up to a cutoff date, and nothing about your private documents. Retrieval-Augmented Generation (RAG) fixes both problems by searching a knowledge base at query time and handing the most relevant passages to the model. It is the most widely deployed enterprise AI pattern, powering internal search assistants, support bots and documentation Q&A.

The RAG pipeline

INDEXING (offline) Documents Chunk Embed Vector + keyword index QUERY (online) Question Retrieve Rerank LLM answers + cites top-k chunks are inserted into the prompt with instructions to answer only from them
Fig. 1 — Offline indexing and online retrieval in a RAG system.

Embeddings: meaning as geometry

An embedding model converts text into a vector of a few hundred to a few thousand numbers. Training pushes semantically similar texts close together, so "How do I reset my password?" and "I forgot my login credentials" end up near each other even though they share few words. Similarity is usually measured with cosine similarity or dot product.

python
import numpy as np def cosine(a, b): return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))) # vectors come from an embedding model, e.g. via your cloud provider's embeddings API query_vec = embed("How do I rotate an AWS access key?") scores = [(cosine(query_vec, c.vector), c) for c in chunks] top_k = sorted(scores, key=lambda s: s[0], reverse=True)[:5]

At scale, comparing against every vector is too slow, so vector indexes use approximate nearest neighbor (ANN) algorithms such as HNSW graphs or IVF partitioning, trading a little recall for orders-of-magnitude speedups.

Chunking: the underrated decision

Documents are split into chunks before embedding. Too large, and a chunk mixes topics and dilutes relevance; too small, and it loses context.

  • Structure-aware splitting — split on headings, sections and paragraphs rather than fixed character counts.
  • Overlap — repeat a little text between adjacent chunks so ideas aren't cut in half.
  • Contextual chunks — prepend a short, generated description of where the chunk sits in its document ("From the 2026 security policy, section on key rotation…"). Anthropic's published "contextual retrieval" technique reported large reductions in retrieval failures with this approach.
  • Metadata — store source, date, owner and access permissions alongside each chunk for filtering.

Hybrid search and reranking

Pure vector search struggles with exact identifiers — error codes, product SKUs, names. Keyword search (BM25) excels at those but misses paraphrases. Hybrid search runs both and merges results, for example with reciprocal rank fusion.

A reranker then rescores the top 50–100 candidates with a more expensive cross-encoder model that reads the query and passage together, keeping only the best handful for the prompt. Reranking is often the single biggest quality improvement in a RAG system.

Where to store vectors

OptionExamplesGood when
Existing database extensionPostgreSQL + pgvector, OpenSearch, ElasticsearchYou want one system and moderate scale
Dedicated vector databasePinecone, Weaviate, Qdrant, MilvusLarge scale, advanced filtering, managed service
Managed cloud RAGBedrock Knowledge Bases, Vertex AI Search, Azure AI SearchYou want ingestion, chunking and retrieval handled for you
Object-store vectorsAmazon S3 VectorsVery large, cost-sensitive, less latency-critical corpora

Does long context kill RAG?

With million-token context windows, you can sometimes just put the whole knowledge base in the prompt — and with prompt caching that can be economical for small corpora. But for most systems retrieval still wins on three counts: cost (you pay for every input token), latency (prefill time grows with input length) and scale (enterprise corpora are far larger than any context window). The two approaches combine well: retrieve generously, then let a long-context model reason over more material than older models could.

Agentic RAG

Instead of a single retrieve-then-answer step, agentic systems give the model search as a tool. It can reformulate queries, search multiple sources, read full documents when snippets are insufficient and stop when it has enough evidence. This handles multi-hop questions ("Which of our customers in Germany are affected by last week's outage?") much better, at the cost of more tokens.

Evaluating RAG

  • Retrieval — recall@k and MRR on a labeled set of questions with known relevant documents.
  • Faithfulness — is every claim in the answer supported by the retrieved text?
  • Answer relevance and completeness — does it actually answer the question?
  • Citations — ask the model to cite sources; many APIs support structured citations that point to exact passages.
Security noteEnforce document-level permissions at retrieval time. If a user can't open a document directly, the assistant must not retrieve it for them either.