AI Support Assistant Optimization Raises Response Accuracy From 56% to 80%

About Our Client

The Client is a European imaging technology company with over 20 years of industry experience, known for releasing several digital tools that have achieved global recognition.

AI Customer Support Assistant Showed Subpar Accuracy

The Client had developed a pilot AI assistant to help its customer support team draft responses. But the pilot version needed improvement before the solution could be used confidently in day-to-day operations. The Client wanted to raise the pilot’s overall response accuracy from about 56% toward an 80% target. The company turned to ScienceSoft, trusting its experience in AI engineering and AI-driven operational transformation.

Optimizing the AI Support Pipeline End to End

To improve the pilot, ScienceSoft’s team of senior data scientists treated it as an end-to-end AI system rather than focusing only on the language model. The work covered five connected stages: preparing knowledge for vector search, improving the response drafting system, creating an evaluation dataset, running controlled experiments, and deploying the improved solution.

Setting up knowledge ingestion pipeline for vector search

The pilot’s knowledge sources included pre-approved support answers, FAQ content, website documentation, release notes, licensing information, and historical support tickets. ScienceSoft set up automated, scheduled procedures that extract the content, prepare it using source-specific preprocessing steps, and transfer it to the corresponding vector stores. This way, the AI assistant searches only current and relevant knowledge.

Optimizing the RAG-based response drafting system

The team set up a staged RAG pipeline: to answer a customer’s inquiry, the AI assistant retrieves information first from pre-approved support answers, then from company documentation, and finally from similar historical tickets. The first stage creates an initial draft; later stages refine it with additional evidence. This staged structure lets the assistant build up evidence progressively, giving each knowledge source a distinct role.

The team built response rules into the AI assistant’s system guidelines and response logic. The rules required it to draft replies only from retrieved company information, include only retrieved links, reply in the customer’s language, use official product names, and follow the required greeting and sign-off format. If the assistant could not find enough relevant information, it flagged this to the support agent rather than generating a reply suggestion.

A human support agent reviews the draft and can edit it before it reaches the customer.

Experiments and accuracy evaluation

ScienceSoft established a controlled experimentation process to compare and select the most effective retrieval and response generation settings for the AI assistant. The experiment protocol was to define a baseline, change one variable at a time, rerun the evaluation, and document the results.

To run the experiments, the team built a dedicated evaluation dataset: they collected historical support tickets, assigned them to predefined categories, and ensured balanced representation across categories. The dataset gave a consistent benchmark for measuring how each change in the retrieval and response drafting pipeline improved or degraded performance.

Following this process, the team tested:

• Retrieval filters — testing how restricting search by ticket category affected the relevance of retrieved information.
• Search thresholds and result limits — changing how selective each vector search was and how many results it could return.
• Query reformulation — comparing retrieval with and without an LLM step that rephrased the extracted customer query before searching company documentation and historical tickets.
• Knowledge source combinations — testing different combinations of vector stores and retrieval stages to identify which mix provided the most useful context for response generation.
• Different LLMs — comparing model performance within the same RAG workflow.

ScienceSoft used automated evaluation, including RAGAS (Retrieval-Augmented Generation Assessment) metrics and an LLM-as-a-judge approach, to compare experimental configurations. This evaluated qualities such as:

• Faithfulness — whether claims were supported by retrieved information.
• Answer relevancy — whether the reply addressed the customer’s question.
• Context precision — how much of the retrieved content was relevant.
• Context recall — whether the retrieval step found enough of the information needed to answer the question.

This gave the team a consistent way to compare retrieval and model changes and identify trade-offs between different quality dimensions.

The Client’s support team also assessed the accuracy of suggested responses, providing a separate human benchmark for how well the system performed in real support scenarios.

Deployment and integrations

Based on the evaluation results, ScienceSoft selected the final retrieval, model, and response-generation configuration and proceeded with deployment. ScienceSoft deployed the AI assistant’s back end on AWS Lambda. The team integrated it with the Client’s Zendesk ticketing platform, the vector knowledge stores, and an external LLM API so the system could receive ticket data, retrieve relevant company knowledge, generate and refine the response draft, and return the suggestion to the support workflow.

ScienceSoft’s front-end developer built a Zendesk interface for support agents to review, edit, or replace the AI-generated response suggestion, and to rate its usefulness before sending the final reply to the customer.

Higher-Quality AI Suggestions Improve Support Speed and Customer Experience

ScienceSoft helped a global imaging technology company turn a customer support AI pilot into a more reliable AI assistant for drafting support responses. The Client reached its target of 80% response accuracy and an acceptance rate of more than 85%. The acceptance rate reflects the share of response drafts that the Client’s support team used as is or with no more than 20% editing.

Stronger knowledge retrieval and response drafting mechanisms built by ScienceSoft led the following support process improvements:

• Faster responses to common questions, allowing support agents to spend more time on complex cases.
• Safer response generation, since the AI assistant answers only from the Client’s approved knowledge, does not invent facts or links, and steps aside when a request is outside its scope or cannot be answered reliably.
• Consistent customer communication thanks to standardized response rules and multilingual support.

Technologies and Tools

Python, AWS Lambda, OpenAI Agents SDK, OpenAI API, OpenAI Vector Store, Zendesk API.