Load Testing of a Leading Edge Management Platform

Customer

Headquartered in the US, the Customer provides a leading platform for managing software and devices (sensors, smart gadgets, machines, etc.) at the edge. The Customer’s platform helps these devices talk to each other, to a distant data center and cloud services, be easily managed remotely, and stay protected from cyber threats.

Need to Ensure the Platform’s Performance with the Number of Managed Devices Growing from 100 to 10,000+ per Client

Previously, each client required the Customer’s platform to manage approximately 100 edge devices, but now many of them wanted to scale up to over 10,000 devices. This significant increase called for comprehensive platform testing and, if bottlenecks get identified, refactoring and modernization to guarantee continued performance, stability, and effective error handling at this new scale.

Given the urgency of these needs and the lack of available internal resources to tackle the challenge within a tight timeline, the Customer decided to seek external expertise. ScienceSoft, having previously completed several projects for the team, emerged as a top contender for this critical task. The Customer's confidence in ScienceSoft was bolstered by our proven track record in performance testing and our 23 years of experience in test automation.

Establishing Load and Fuzz Testing Processes

Load testing

ScienceSoft meticulously prepared a load generator. This tool was designed to simulate the volumes of data processing tasks typical for scenarios involving over 10,000 devices. To complement this, we carefully outlined the expected loading pattern that reflected typical and peak usage scenarios, giving comprehensive insights into the platform's behavior under different conditions.

The team also created sophisticated automated load tests. We ensured these tests covered all business scenarios, encompassing over 200 API points across the platform's admin and back-end parts.

By integrating these load tests into the continuous integration (CI) pipeline, ScienceSoft made the platform’s performance assessment an integral part of the platform's development lifecycle. Any changes introduced to the platform were automatically evaluated for their impact on performance, allowing for early identification and resolution of potential issues in the development process.

In the midst of an intensive testing phase, our team identified and described previously undetected areas of the platform where performance lagged. We analyzed the root causes of these issues and suggested a set of targeted short-term fixes and long-term mitigation strategies, including code and architecture adjustments. The recommended short-term adjustments were promptly and successfully implemented by the Customer’s in-house team, leading to a significant uplift in the platform's overall performance and reliability. Long-term adjustments evolved into a separate project of modernizing the Customer's software product, in which ScienceSoft's architects and developers are actively participating.

Fuzz testing

ScienceSoft’s team also designed automated fuzz tests and integrated them into the CI pipelines. These tests simulated random data inputs across various API fields, challenging the platform's data validation and filtering mechanisms in new ways. The aim was to identify any vulnerabilities that might compromise the platform's stability, ensuring it could gracefully manage even the most unpredictable data inputs.

Technology foundation

We used Apache JMeter as a core technology for both the load and fuzz tests. We also introduced a custom mutator in Golang as an extra layer of sophistication to the fuzz tests. Specially created for this project, it enabled the generation of truly random and diverse input data sets, checking the limits of what the platform was expected to endure.

Collaboration

ScienceSoft’s team maintained regular interactions with the Customer's project managers, developers, and quality assurance teams, fostering a collaborative environment essential for quickly identifying and addressing potential issues. We held daily meetings to ensure a steady flow of information and progress updates, while quarterly on-site meetings provided opportunities for in-depth discussions and strategy refinements.

Enhancing the Reliability of Edge Management Platform

In six months, ScienceSoft established thorough load and fuzz testing for the Customer’s platform. Over the next six months, the team iteratively ran the tests, analyzed the results, pinpointed bottlenecks, and recommended mitigation strategies.

Thanks to integrating the automated load and fuzz tests into the CI pipeline, the Customer can continuously monitor the platform’s reliability, responsiveness, and resilience. The enhanced platform performance and stability led to widespread user satisfaction, bolstering the Customer's reputation for providing scalable, reliable, and efficient solutions.

Technologies and Tools

Apache JMeter, Groovy, Golang, Jenkins, Python, Bash scripts, Docker, Kubernetes, Terraform, Terragrunt, Cloudflare, HashiCorp Vault, AWS (EC2, Virtual Private Cloud, Relational Database Service (RDS), S3, CloudWatch, Elastic Container Service for Kubernetes, CloudTrail, Elastic Load Balancing, etc.), Azure (Virtual Machines, Blob Storage, Databases, Virtual Network, etc.).