Managed Testing of an Ecommerce Solution, CRM and Data Warehouse for US Multi-Industry Corporation

Customer

The Customer is a US-based multi-business corporation present in a variety of industries including hospitality, retail (ecommerce and brick-and-mortar fashion stores), and wellness.

Challenge

The Customer has a complex IT environment comprising several interconnected business-critical systems including the following core components integrated with other third-party systems:

• The ecommerce platform based on Magento.
• The Microsoft Dynamics 365 solution (web and mobile).
• The on-premises data warehouse (DWH).

These solutions were in the process of development and customization. So, The Customer was looking for a managed testing services provider to validate the smooth functioning, security, usability of the systems and to make sure that the introduced changes didn’t undermine the existing functionality.

An additional challenge was that the systems sustained numerous integrations with one another, several other enterprise systems, and internal modules and tools. Thus, the Customer was searching for a testing services provider experienced in Magento, Microsoft Dynamics, and DWH testing.

Solution

ScienceSoft’s QA team reviewed the requirements specifications to the ecommerce platform and CRM, got acquainted with each system’s existing functionality, and identified relevant stakeholders. To streamline collaboration the QA manager agreed with the Customer on a conventional test management and a continuous integration tool.

For each system, the testing objectives, scope and types were outlined, an approximate testing schedule and a test team line-up were defined.

A dedicated test team was appointed to testing of each piece of software and their smooth integrations. During the testing process, the test team size has been scaled up and down to strike a balance between the required test coverage and testing timelines.

Ecommerce platform testing

A group of six test engineers got down to testing as soon as the development process of the ecommerce platform started.

They started with exploratory testing that required minimum test documentation. This allowed for a quick overview of possibly problematic areas in the platform’s functioning and the prioritization of future more thorough testing. Exploratory testing helped ScienceSoft’s test engineers to find the problematic issues of the platform’s functioning, GUI, and integrations that were to be checked additionally.

Upon GUI testing, test engineers put forward a number of suggestions concerning the platform’s look, feel and the layout of GUI elements to the development team.

Functionality testing validated that all platform functions stated in the requirements were executed smoothly.

Integration testing formed the most time-consuming part of the testing process. ScienceSoft’s test team checked the platform’s integrations with such external systems as CMS, a PIM tool, a website payment and a POS system, a warehouse management tool, CRM, and a tax calculator. They developed and executed complex and comprehensive test scenarios based on a formalized data flow diagram to validate data flow paths across multiple interconnected systems.

Performance testing was conducted to check the platform’s endurance under peak and continuous load, validate the maximum number of simultaneous users and the system’s acceptable response time.

In addition, ScienceSoft’s team carried out security testing (including vulnerability assessment and penetration testing) to ensure that personal data of online shoppers was sufficiently protected and payment-transaction processes complied with PCI DSS.

Before the release of the ecommerce platform, the test team performed the overall user acceptance testing. It was done in the platform’s production mode to ensure that the test engineers and end users had similar experience of using the platform. The test engineers checked the platform in line with real-life scenarios encompassing all the features and functions stated in the final version of the requirements specification (comprising architectural, interface, performance, security, and other requirements).

As the ecommerce platform was regularly amended, intermittent regression testing took place to ensure the Customer’s business continuity. It was performed after the introduction of every new add-on or further customization of the platform to validate the stability of its existing functionality.

CRM testing

ScienceSoft’s team tested the Customer’s web and mobile on-premises CRM applications based on Microsoft Dynamics 365. The functionality of both applications was identical. The apps comprised the Customers module and the Employees module. Client information needed for creating a contact in CRM was uploaded from the Customer’s ecommerce platform and POS system.

The Customer’s employees from hotels, wellness centers, and brick-and-mortar stores used CRM, so it was customized to introduce numerous workflows and industry-specific functions required by the staff involved in different industries and having many access levels. Thus, two ScienceSoft’s test engineers paid special attention to functionality, performance, usability testing and role-based access control validation during web and mobile CRM testing.

The test environment was used for exploratory and performance testing. Functionality and usability testing as well as role-based access control validation were executed in the staging environment to ensure almost real-life conditions.

On the basis of the Customer’s most current software requirements, ScienceSoft’s BA and test team designed user stories with the focus on practical results beneficial for the system’s users. They fully covered specific workflows of the three industries of the Customer’s business and validated the process.

To quickly prioritize user stories and test cases, the dedicated test engineers executed exploratory testing. Afterwards, the test engineers got down to comprehensive functionality testing. With a significant number of different user stories and test cases (2,200) to implement for retail, hospitality and wellness employees, functionality testing took the majority of the testing time.

Role-based access control validation was performed to check that each employee had access to functionality and information according to their position. To manage this process, the test team created a roles matrix: for brick-and-mortar stores – 8 access levels were assigned, for hospitality employees – 7 levels, for wellness centers – 5.

During CRM performance testing, ScienceSoft’s test engineers validated the maximum number of simultaneous users, data transfer rate, and response time.

In addition, the test team checked the mobile CRM app against its compliance with requirements to minimal device processing power, random access memory, and battery consumption.

Then, ScienceSoft’s test engineers carried out usability testing that checked whether mobile CRM’s UI complied with general mobile app design guidelines and UI standards (Google Material Design for Android and Human Interface Guidelines for iOS). The mobile app was tested on three iOS devices of different versions and two Android devices. When the app’s UI was checked on an Android device, the overlarge font and the inconvenient layout of drop-down lists were detected and reported to the development team.

DWH testing

The Customer’s on-premises DWH stores data from multiple enterprise software including the warehouse management system, the ecommerce solution, and CRM. A dedicated test engineer carried out ETL (Extract-Transform-Load) testing to validate that varied product, customer, and employee data was correctly transformed into a relevant data type, uploaded to the DWH, and then extracted from it.

Besides, ScienceSoft’s test engineer checked the compatibility of data formats supported by the interrelated systems, the absence of duplicated data, and DWH response time.

It turned out that the systems integrated with the DWH had different field names, allowed varied number of characters in a field, and supported conflicting data formats. The test engineer found a number of inconsistent data types that were reported to relevant development teams to fix. For example, the POS system stored customer phone numbers in a numeric format with automatically inserted country codes. Such phone numbers were improperly rendered in CRM customers’ profiles as the respective field was designed for a smaller number of digits (without a country code).

Identifying redundant data became the most time-consuming part of DWH testing. The test engineer looked for duplicated customer accounts provided mostly by the Customer’s POS system. It turned out that the DWH was full of customer accounts with different IDs and identical name, birthday and contact information fields.

ScienceSoft’s test engineer also carried out automated DWH performance testing as ETL testing had shown that DWH response time had failed to meet the Customer’s requirements. The test team also checked the DWH load limits as it had to cope with the large volume of data (5,000+ real client records) and the maximum number of users who address the DWH through ecommerce platform and CRM simultaneously.

Results

The Customer got their ecommerce platform, CRM solution, DWH, and their integrations thoroughly tested. The Customer’s representatives could keep track of the ongoing testing activities via test management tool, and continuous communication between the parties engaged. Detailed test execution reports were regularly provided to the Customer’s representatives and test summary reports were presented after each testing stage.

The functionality of the ecommerce platform, which was improved according to the testing results, allowed for improved shopper engagement with the online store. Addressing the issues revealed during web and mobile CRM testing helped the Customer’s employees to create the customers’ profiles and manage them quicker and more effectively, facilitate the communication with their colleagues from different industries and departments, provide customers with cross-industry bonuses, thus, improve customer retention. DWH testing helped the Customer to ensure that the ecommerce platform and CRM provided clients and employees with consistent data.

Technologies and Tools

Microsoft Dynamics 365, Magento 2, Magento Test Framework, Apache JMeter, Atlassian JIRA, Atlassian Confluence, iOS 9-11, Android 7-8.