RAG Pipeline Architecture
Built a distributed, asynchronous RAG pipeline using Celery, Redis, and Unstructured for document ingestion, with independently retryable stages and exponential backoff for fault isolation. Used Amazon S3 for artifact storage and Qdrant for vector retrieval, enabling independent scaling across ingestion, storage, and search. Implemented a hybrid retrieval service combining vector search, keyword search, metadata filtering, and cross-encoder reranking to improve context quality before LLM generation.