RAG is often described as “chat with your documents.” In practice, useful enterprise retrieval is a product and data architecture problem involving source quality, permissions, retrieval behavior, citations and evaluation.
Retrieval quality starts before embeddings
Documents need useful structure, ownership and metadata. Chunking cannot compensate for outdated, duplicated or permission-blind source content.
A better knowledge base often improves RAG more than a more expensive model.
Design retrieval around the user task
Search for a policy, compare contracts and synthesize a project history are different retrieval problems. Each may need different ranking, filters and context composition.
The unit of retrieval should match the unit of work.
Evaluation needs real business questions
Build a regression set from actual user questions, expected sources and acceptable answer behavior. Track retrieval recall, groundedness, refusal behavior, latency and cost.
Without an evaluation set, RAG tuning becomes opinion rather than engineering.