CI Pipeline to Enable Effortless Unit Testing of 80,000+ Lines of Code

Customer

The Customer specializes in image-processing software products for both amateurs and pros bridging the gap between science and art in photography. ScienceSoft specialists are developing an application that helps to enhance colors, remove distortions and deal with chromatic aberrations as well as add new features to images such as textures, frames and analog filters.

Challenge

Besides dealing with the challenges naturally arising from complexity of the software and product features being developed, the team which consists of 4 senior developers, 2 senior and 1 middle testers and a Scrum master must ensure continuous and timely incremental releases. The key to success is to keep the number of conflicts and bugs to the minimum and promptly react to them while merging all developer working copies with a shared mainline.

Solution

To reduce testing efforts and developer rework it has been decided to implement continuous integration throughout the project. It presupposes that a combination of automated and unit tests are run on a regular basis alongside manual testing providing developers with constant feedback about the application performance and timely alerting them of the existing problems.

Currently the following infrastructure is in place: a pool of agents with different build tools for Windows and Mac OS are deployed and customized on an Atlassian Bamboo server. Several agents are used to run tests in parallel to speed up test execution, for example, for automated acceptance tests such an approach helps to reduce the test time from 4-5 to 1.5-2 hours, and currently, there are the following agents in the pool:

• 1 Windows agent with InstallShield
• 2 regular Windows agents without InstallShield
• 3 Mac OS agents

Such an infrastructure makes it possible to create three types of builds. The system automatically runs unit tests (570 tests to check unit logics) and acceptance tests (470 different user scenarios operating at a higher level than unit tests). The light test plan is triggered within 15 minutes after every commit with the number of commits amounting up to 20-30 a day per developer. A heavy test plan includes all available tests and takes a longer time (1.5-2 h) and therefore is started once a day, usually during the nighttime.

Apart from heavy and light test plans the master test plan is used to assist manual testing and prepare final delivery of the team. It is started by testers on average 2-4 times per day and its output is two installation packages (for Windows and Mac OS) while light and heavy tests assemble only a binary file package.

Heavy test plan includes, in particular, the following tests:

• Code Complexity test to pinpoint recently added complex functions which need to be simplified;
• Static Analysis test to analyze the code and track unused variables, incorrect memory allocation, dead code and so on;
• Performance tests to measure the duration of some important operations and compare them with the results of the previous builds. It helps to identify improvements or declines in performance;
• Unit test coverage analysis to measure the amount of code covered by unit tests (for the current UI application it amounts to 45%).

Results

Continuous integration allows to efficiently test a large amount of software (currently exceeding 400 files with 80,000+ lines of code) without attracting extra human resources. The team constantly receive feedback from the server in the pass/fail format for each test with additional specifications for failed automated tests informing about the step at which the problem occurred. Thus, the developers can attend to failed unit tests on the go, or – in case of conflicts during the acceptance tests – fix them in the course of one working day.

As a result the implemented solution has proved to be efficient for accelerating the product time to market and has also helped to establish a smooth workflow within the team.

Technologies and Tools

The project – and consequently the continuous integration – is realized on two platforms: Windows and Mac OS. The main technologies used include C++, Qt and Lua scripts. Essential to continuous integration, such tools as Atlassian Bamboo, Atlassian Stash, Visual Build, InstallShield and cmake enable ScienceSoft developers and build masters to easily create an automated, repeatable build process.