How to Test Software for Different Types of Bugs
A Detailed Testing Plan and Bug Classification
ScienceSoft provides comprehensive software testing services for over 33 years.
Why Do You Need Bug Classification in Software Testing?
During software testing, accurate bug classification and prioritization is crucial for assigning the found defects to the responsible project teams, thus streamlining not only the bug-fixing process, but also the testing and development workflows.
Relying on decades-long experience in software QA, ScienceSoft’s testing engineers outline the key steps of the testing process for different types of bugs and share best practices for correct defect prioritization.
Bugs by nature of testing
The first type of defect classification is based on the nature of testing. At ScienceSoft, we usually single out the following key types of defects:
Performance bugs are found during stress, load, stability, and scalability testing. This kind of testing is fully automated. The most popular performance testing tools are Apache JMeter and LoadRunner.
Usability defects are revealed during usability testing, UX audit, or UX research. ScienceSoft applies expert-based (e.g., heuristic evaluation and cognitive walkthrough) and user-based (interviews and surveys of the TA, executing scenarios by members of the TA) techniques to promptly detect usability issues.
Detecting security defects requires vulnerability assessment, penetration testing, security code review, and more. Software compliance assessment (e.g., for HIPAA, PCI DSS, GDPR) can be considered a part of security testing as well. Security testing can be both manual and automated, depending on the needs of each specific project.
Bugs by severity and priority
The next type of bug classification is based on how seriously the defect affects the software and how promptly it needs to be fixed.
Defects by severity
To determine bug severity, test engineers consider how strongly it impacts the software functionality, performance, usability, etc. and how frequently it occurs. According to this classification, bugs can be critical, high-, medium-, and low-severity.
Defects by priority
To prioritize bugs, test engineers look at the business impact of a defect (including the number of users affected, the threat to the company’s image, the resulting business disruptions, losses, etc.). Thus, bugs can be classified as urgent, high-, medium-, and low-priority.
Viktor Sachuk, Test Manager and QA Consultant at ScienceSoft:
Classifications by severity and priority are not necessarily interconnected. For example, a low-severity defect can be of high priority, like a broken website layout or typos on the landing page. Such bugs do not affect functionality, but they are likely to undermine the brand image and customer experience. Another example is a critical bug of low priority: a layout is displayed incorrectly in legacy browsers. If only a few occasional users access the website via those outdated browsers, fixing the defect will not be of high importance, even though the whole app is affected.
While it is essential to classify and prioritize defects correctly, establishing a comprehensive testing process that will allow you to timely reveal and eliminate any type of bug is even more important. Below, ScienceSoft’s testing experts outline a high-level software testing process that stays largely the same regardless of the testing type.
Requirements analysis and test planning
- Analyzing the existing software requirements or helping draft testable specifications.
- Deciding on testing objectives and the types of testing to perform (functional, performance, security, etc.).
- Defining the criteria for assessing the severity and priority of found defects.
- Defining the required team composition, estimating the testing cost, timelines, choosing the optimal testing and defect tracking tools, etc.
- Defining the share of test automation (if feasible).
- Outlining the KPIs to measure the testing effectiveness: e.g., average test execution time, the number of created test cases, average and total number of found defects, defects rejected by the dev team, requirements covered by test cases, test automation rate, and more.
I strongly recommend involving QA specialists as early as at the requirements specification stage. A QA team needs to make sure that the requirements are fully testable from the very beginning of the SDLC: it will help perform comprehensive testing and avoid costly redevelopments later on.
- Creating test cases and writing automated test scripts.
- Preparing the required test environment and test data.
- Configuring the relevant test automation frameworks and tools, if needed.
Test execution and defects reporting
- Executing the test cases and running automated test scripts.
- Submitting the found bugs into the defect tracking tool, classifying and prioritizing them.
- Retesting the software to validate the fixed defects.
- Running regression testing to make sure no related functionality has been affected by recent fixes.
- Providing test results reports based on the chosen KPIs and analyzing the effectiveness of the QA process to implement the necessary improvements.
Not every project needs 4+ severity or priority levels for efficient defect classification. Smaller projects usually do well enough with just 3 levels (critical, medium, low), while large-sized projects need a more elaborate system to clearly differentiate between high- and maximum-urgency tasks.
What’s more, most bug tracking systems offer severity and priority settings by default. Still, I recommend setting an additional classifier for the nature of the defect (e.g., functional, performance, usability, etc.) as this simplifies the assignment of bug-fixing tasks to the responsible teams.
Our Customers Trust Us
QA Manager, ATR
We are satisfied with the black-box penetration testing services provided by ScienceSoft and with their team’s attention to detail and proactive approach to collaboration. They were also very responsive and eagerly suggested security enhancements. We highly recommend ScienceSoft as a reliable cybersecurity partner.
Co-Founder and CEO
After looking through numerous candidates, we set our choice upon ScienceSoft and, over the course of our close cooperation, we haven’t regretted the decision. In terms of … automated testing services, they fulfilled our expectations and became very valuable for the projects. We would definitely recommend working with ScienceSoft as a long-term partner.
Co-Founder and COO
ScienceSoft’s team has carried out manual functional testing of our products and cross-browser testing for Windows, Mac, Android, and iOS. They also created test cases for regression testing of certain modules of the products. We appreciate ScienceSoft’s scrupulousness and proactive approach – the team has regularly suggested improvements to the products’ functionality based on the found issues.
COO / Global Managing Director
I hired ScienceSoft to perform stress testing of our software. The team at ScienceSoft built a solid platform for stressing out servers, and it helped us uncover a variety of bugs and revisions that needed to built to our code. The team at ScienceSoft is professional and organized in every way possible. I look forward to working with this team again.
- 33 years in software testing and 21 years in test automation services.
- Trusted by global brands, including Deloitte, IBM, Walmart, eBay, Nestle, NASA JPL, Viber, and more.
- ISTQB-certified testing experts on board.
- ISO 9001- and ISO 27001-certified to prove our commitment to providing quality services and ensuring full security of our customers’ data.
- Experience in 30+ industries, including healthcare, manufacturing, finance, retail, telecommunications, and more.
- Standardized defects description, test cases design, and test reporting in accordance with ISO/IEC/IEEE 29119-3:2013.
Managed Testing of Health Information Exchange Software for US Healthcare Providers
As a result of comprehensive managed testing performed by ScienceSoft’s team, the customer ensured smooth software functionality, proper integration with multiple EHR systems, compatibility with the required web browsers and versions of mobile OSs, and the system’s high performance under continuous load.
Testing of an MSP Application for a US-based IT Company
ScienceSoft performed functional, regression, and performance testing of the customer’s app. As a result of ScienceSoft’s well-coordinated teamwork, the customer obtained the needed MSP solution offering fully functional services to their end users.
Ecommerce Solution Pentesting for a Company Providing Software for Airlines
ScienceSoft revealed several security vulnerabilities of low and medium severity and provided a list of corrective measures to eliminate the security weaknesses and increase the ecommerce platform protection level.
UX Audit to Help a Website Prepare for WCAG Conformance Certification
ScienceSoft performed comprehensive usability testing and the website’s UX design audit against the WCAG requirements. We provided a detailed UX report describing the revealed issues and recommended actions and helped fix serious discovered issues.
The testing team composition will vary depending on the project scope and the type of testing you need to perform. Still, to form an all-around QA team that will ensure comprehensive software testing, ScienceSoft usually engages the following talents:
- Helps design testable software requirements and establishes a robust QA process from the beginning of SDLC.
- Defines the testing scope and develops a comprehensive testing strategy and plan, including bug classification and prioritization criteria.
- Supervises the test engineers’ activities and introduces the necessary improvements to the QA and testing processes.
- Reports on the QA progress and ensures testing KPIs fulfillment.
- Prepare the test environment and data.
- Design, maintain, and execute test cases to test the software against functional requirements.
- Report on found bugs in the defects management tool.
- Verify the fixed defects.
- Design performance testing scenarios and scripts.
- Prepare the test data and configure the test environment.
- Run load, stress, scalability, stability tests.
- Identify performance bottlenecks using performance metrics (latency, throughput, etc.).
- Provide recommendations on how to eliminate performance issues.
- Study the needs of the potential users, create personas to represent potential users.
- Map out user journeys and create user scenarios.
- Assess the app’s navigation and structure, check how the UI/UX affects the intended user workflows.
- Conduct usability testing to ensure that personas can complete their scenarios.
- Assess the GUI compliance with WCAG and Section 508 recommendations (if applicable).
- Perform vulnerability assessment and penetration testing of the software under test.
- Report on the discovered vulnerabilities.
- Provide recommendations on remediating the security issues.
- Conduct retesting to confirm that the changes in the software didn’t create new vulnerabilities.
- Prepare the environment for the required type of automated testing (e.g., regression, performance testing).
- Configure test automation tools.
- Write, execute, and maintain automated test scripts.
- Report on the found defects and verify the fixes.
In-house software testing
- Full control and responsibility over the testing process.
- High recruitment and training costs.
- Low team scalability.
Turn to ScienceSoft if you need professional advice on establishing a mature software testing process.
- Possibility to scale the testing team up and down on demand.
- Balanced software testing costs.
- All managerial efforts are on your side.
Turn to ScienceSoft if you are looking for experienced test engineers to expand your team.
Automated UI testing tools
Automated mobile testing tools
Performance testing tools
Security testing tools
Test management and defect tracking software
ScienceSoft is a global QA consulting and software testing company headquartered in McKinney, TX. Since 1989, we deliver professional testing services to validate every aspect of our customers’ software quality, from functionality to usability and regulatory compliance. Being ISO 9001 and ISO 27001-certified, we can ensure maturity of our services and complete security of the data entrusted to us.