LLM-Based Document Search Assistant for a Security Preparedness Company

About Our Client

The Client is a GCC-based company that supports organizations operating in security-sensitive environments. Through advisory services, specialized training, and personnel readiness assessments, it helps its clients improve their preparedness for complex operational scenarios.

Need for Transparent AI Search in Sensitive Document Workflows

The Client’s internal users needed a faster and more reliable way to search the company’s large knowledge base of security-related documents. Conventional keyword search was insufficient for complex queries: users had to know the exact terms used in the documents and often still had to review multiple files manually to locate the right information.

For a company working with security-sensitive materials, search results also had to be transparent and easy to verify. The Client needed users not only to receive relevant answers but also to see which source documents the answers were based on.

To explore this opportunity, the Client decided to check whether an AI assistant could support free-form search across the available documents and return grounded answers with links to the original sources. Before committing to a full-scale implementation, the Client engaged ScienceSoft’s AI development team to test the feasibility, accuracy, and transparency of AI-powered search via a proof of concept.

RAG-Based AI Assistant With Hybrid Document Search and Source-Linked Answers

ScienceSoft developed the PoC as a retrieval-augmented generation (RAG) assistant designed to help users search a large knowledge base of security-related documents using free-form questions. To make the assistant’s responses transparent and easy to verify, the generated answers included links to the source documents that informed them.

Using the document set provided by the Client as the knowledge source, ScienceSoft evaluated several open-source foundation models for the assistant. Given the Client’s regional context, the team considered Falcon-family models alongside Llama-family models. During testing, Llama 2 produced stronger and more consistent responses, so ScienceSoft selected it as the foundation model and deployed it via a Text Generation Inference (TGI) server for PoC evaluation.

To compare model outputs and retrieval configurations consistently, ScienceSoft applied a metric-based evaluation approach using LLM-as-a-judge metrics, including faithfulness, answer relevance, context relevance context recall, and answer correctness. This helped the team assess which setup retrieved the most relevant context, generated accurate answers, and kept responses grounded in the source materials.

As the main way to improve retrieval quality, ScienceSoft implemented a hybrid search pipeline combining BM25 and vector search. BM25 supported exact keyword matching, which was important for queries involving dates, quantities, names, or specific terms. Vector search complemented this approach by retrieving semantically relevant content even when users phrased questions differently from the source documents. Combining the two search methods helped the assistant handle both precise factual queries and broader questions requiring information from several sources.

ScienceSoft also fine-tuned the embedding model used for semantic retrieval to improve how the system matched user queries with relevant document fragments. This helped adapt vector search to the terminology, abbreviations, and phrasing found in the Client’s documents. The fine-tuning had a positive effect, but its impact was limited compared with the improvement brought by hybrid retrieval. For this reason, ScienceSoft treated the hybrid search setup as the main driver of retrieval quality while keeping the fine-tuned embeddings as part of the PoC.

The team also evaluated an agentic architecture but found that the available open-source models did not provide the level of predictability and answer quality required for the PoC. ScienceSoft therefore settled on a controlled RAG-based assistant that could retrieve source content, generate document-based answers, and make the output easier to verify.

ScienceSoft delivered the PoC as a chatbot application where users could ask free-form questions, refine their search with follow-up prompts, review chat history, and leave feedback on the assistant’s responses.

AI-Assisted Document Search Ready for Enterprise Scaling

The proof of concept demonstrated that AI-assisted search could help the Client’s staff navigate the large knowledge base of security-related documents more efficiently. Using a chatbot interface, users could ask retrieve relevant information through free-form questions, follow-up prompts, or exact keyword queries for names, dates, quantities, and specialized terms. The AI assistant kept responses transparent by linking each answer to the source documents it relied on, giving users a faster way to verify information and reducing the need to manually review multiple documents.

The PoC gave the Client a validated technical foundation for an enterprise rollout, including the target AI architecture, a working retrieval approach, and quality benchmarks for production implementation.

Technologies and Tools

Llama 2, RAG, BM25, FAISS, SFT, LLM-as-a-judge, TGI.