Common Problems with Testing?
There are major problems with the efficiency and effectiveness of testing as it is currently performed in practice. In the course of three decades of developing systems and software—as well my involvement in numerous independent technical assessments of development projects—I have identified and analyzed testing-related problems that other engineers, managers, and I have observed to commonly occur during testing.
The large number of testing problems necessitated that they be categorized. At the top level, these problems were organized into two groups
general testing problems that are not specific to any type of testing, but apply to all different types of testing
test type-specific problems that are specific to a single type of testing such as unit testing, integration testing, and system testing.
• Test planning and scheduling problems often occur when there is no separate test plan, but rather highly incomplete and superficial summaries in other planning documents. Test plans are often ignored once they are written, and test case descriptions are often mistaken for overall test plans. The schedule of testing is often inadequate for the amount of testing that should be performed, especially when testing is primarily manual. Significant testing is often postponed until too late in the development process, especially on projects using traditional sequential development cycles.
• Stakeholder involvement and commitment problems include having the wrong testing mindset (that the purpose of testing is to show that the software works instead of finding defects), having unrealistic testing expectations (that testing will find all of the significant defects), and having stakeholders who are inadequate committed to and supporting of the testing effort.
• Management-related testing problems involve the impact of inadequate management. For example, management can fail to supply adequate test resources or place inappropriate external pressures one testing. There may be inadequate test-related risk management or test metrics. Testing lessons learned are far too often ignored, so the same problems are repeated project after project.
• Test organizational and professionalism problems include a lack of independence, unclear testing responsibilities, and inadequate testing expertise.
• Test process problems often occur when testing and engineering processes are poorly integrated. Organizations sometimes take a “one-size-fits-all” approach taken to testing, regardless of the specific needs of the project. Testing may not be adequately prioritized so that functional testing, black-box system testing, orwhite-box unit and integration testing may be overemphasized. Testing of components, subsystems, or the system may begin before they are sufficiently mature for testing. Other problems include inadequate test evaluations and inadequate test maintenance.
• Test tools and environments problems include an over-reliance on manual testing or COTS testing tools. Often, there are an insufficient number of test environments. Some of the test environments may also have poor quality (excessive defects) or insufficient fidelity to the actual system being tested. Moreover, the system and software under test may behave differently during testing than during operation. Other common problems are that tests were not delivered or the test software, test data, and test environments were not under sufficient configuration control.
• Test communication problems primarily involve inadequate test documentation. These types of problems often occur when test documents are not maintained or inadequate communication concerning testing is taking place.
• Requirements-related testing problems are related to the requirements that should be driving testing. Often, the requirements are ambiguous, missing, incomplete, incorrect, or unstable. Lower-level requirements may be improperly derived from their higher-level sources. Likewise, verification methods may be unspecified and the tracing between requirements and tests may be lacking.
There are major problems with the efficiency and effectiveness of testing as it is currently performed in practice. In the course of three decades of developing systems and software—as well my involvement in numerous independent technical assessments of development projects—I have identified and analyzed testing-related problems that other engineers, managers, and I have observed to commonly occur during testing.
The large number of testing problems necessitated that they be categorized. At the top level, these problems were organized into two groups
general testing problems that are not specific to any type of testing, but apply to all different types of testing
test type-specific problems that are specific to a single type of testing such as unit testing, integration testing, and system testing.
The remainder of this post will focus on general testing problems, which
can be divided into eight categories:
• Test planning and scheduling problems often occur when there is no separate test plan, but rather highly incomplete and superficial summaries in other planning documents. Test plans are often ignored once they are written, and test case descriptions are often mistaken for overall test plans. The schedule of testing is often inadequate for the amount of testing that should be performed, especially when testing is primarily manual. Significant testing is often postponed until too late in the development process, especially on projects using traditional sequential development cycles.
• Stakeholder involvement and commitment problems include having the wrong testing mindset (that the purpose of testing is to show that the software works instead of finding defects), having unrealistic testing expectations (that testing will find all of the significant defects), and having stakeholders who are inadequate committed to and supporting of the testing effort.
• Management-related testing problems involve the impact of inadequate management. For example, management can fail to supply adequate test resources or place inappropriate external pressures one testing. There may be inadequate test-related risk management or test metrics. Testing lessons learned are far too often ignored, so the same problems are repeated project after project.
• Test organizational and professionalism problems include a lack of independence, unclear testing responsibilities, and inadequate testing expertise.
• Test process problems often occur when testing and engineering processes are poorly integrated. Organizations sometimes take a “one-size-fits-all” approach taken to testing, regardless of the specific needs of the project. Testing may not be adequately prioritized so that functional testing, black-box system testing, orwhite-box unit and integration testing may be overemphasized. Testing of components, subsystems, or the system may begin before they are sufficiently mature for testing. Other problems include inadequate test evaluations and inadequate test maintenance.
• Test tools and environments problems include an over-reliance on manual testing or COTS testing tools. Often, there are an insufficient number of test environments. Some of the test environments may also have poor quality (excessive defects) or insufficient fidelity to the actual system being tested. Moreover, the system and software under test may behave differently during testing than during operation. Other common problems are that tests were not delivered or the test software, test data, and test environments were not under sufficient configuration control.
• Test communication problems primarily involve inadequate test documentation. These types of problems often occur when test documents are not maintained or inadequate communication concerning testing is taking place.
• Requirements-related testing problems are related to the requirements that should be driving testing. Often, the requirements are ambiguous, missing, incomplete, incorrect, or unstable. Lower-level requirements may be improperly derived from their higher-level sources. Likewise, verification methods may be unspecified and the tracing between requirements and tests may be lacking.
