Overview

Teams sit on folders of PDFs — policies, manuals, contracts — and need answers from them without a general-purpose model hallucinating around the gaps. This is a deliberately simple RAG chatbot: point it at a documents/ folder, and it answers strictly from what's in those files, grounded and citable, with a hard fallback when the answer genuinely isn't in the corpus.

Tech Stack

Challenges

• Keeping the model from answering beyond the documents — hallucinating a plausible answer that isn't in the corpus.
• Running entirely against an internal, OpenAI-compatible company gateway behind a firewall.
• Keeping embedding and chat model names in sync so the index and queries never mismatch.
• Staying simple enough to set up with no database or hosted vector service.

Solution

ingest.py loads every PDF, splits text into ~1000-character overlapping chunks, embeds each via the company's embedding API, and saves a FAISS index locally. app.py loads that index and, for each question, retrieves the top-k relevant chunks and passes them to the LLM with strict instructions to answer only from that context — returning "I could not find this information in the provided documents" otherwise. Both scripts read the embedding model from a shared .env so they can't drift, and the OpenAI-compatible client works with any internal gateway (or OpenAI, Azure, or Gemini with a one-line swap).

Outcome

A clean, minimal Streamlit chat window with no login, no database and no extra setup: drop PDFs in a folder, run ingest, and ask away. Because nothing is pulled from HuggingFace and both embeddings and answers come from the configured gateway, it runs happily behind a corporate firewall — provider-agnostic by design, with settings exposed near the top of each script.

What I'd do differently

The value here is restraint: a RAG system people actually trust is one that says "I don't know" cleanly, so the grounding prompt and the fallback message matter more than any clever retrieval trick. Keeping the model and endpoint fully config-driven (no code changes to switch providers) made it genuinely portable across company gateways.