A common data environment that reads your documents and writes new ones — ingest, chunk, embed, retrieve, then synthesise against a template. Every stage is containerised and independently observable.
- Built a containerised RAG backend on LlamaIndex for document chunking and metadata extraction, with each stage isolated behind its own service boundary.
- Used Qdrant — a Rust vector database — for high-dimensional similarity search, keeping k-NN latency off the request path of ingestion.
- Decoupled storage from day one: MinIO for raw objects, PostgreSQL for metadata, so write scaling and blob scaling are independent problems.
- Wired the whole pipeline into Promtail → Loki → Grafana, because a synthesis step that fails silently is worse than one that fails loudly.
- Python
- Qdrant
- LlamaIndex
- MinIO
- PostgreSQL
- Docker Compose