Test Design Review Basics

What Is a Test Design Review?

A test design review refers to a process in which multiple people check the test design itself—such as the test plan and testing methods—before executing the tests, in order to identify issues and areas for improvement.

In software development, it is common to review design documents before writing programs, and similarly, a “review at the design stage” is extremely important for testing as well. If you create test cases (procedural manuals for each test, specifying what to test and how) and only notice problems or missing test cases after executing them, it will result in significant rework, costing extra time and money. In some cases, it could even lead to missing deadlines. Therefore, by finding and correcting issues during the test design review stage, you can achieve efficient and effective testing.

Furthermore, embedded software is used in many products that affect human lives, such as automotive control systems and medical devices. If a defect is overlooked during testing, it could lead to a serious accident, which is why improving the quality of the testing itself is extremely critical.

Purpose of Test Design Reviews

Test design reviews primarily have the following purposes:

Confirming Test Coverage

The most important purpose of a test design review is to confirm whether the test cases sufficiently cover the items that require testing. It involves checking whether all functions described in the design documents and specifications are tested, and whether not only normal cases but also abnormal cases and error handling are appropriately tested.

In embedded software, there are a wide variety of abnormal patterns to consider, such as integration with hardware, sensor input anomalies, and communication errors. Through reviews, it is important to ensure that easily overlooked boundary value tests and stress tests are not missing.

On the other hand, it is impossible to conduct tests that cover all conditions, and it may take too much time relative to the required quality. In such cases, the number of test cases may be reduced within a range that does not degrade quality. It is important to confirm during the review that “even with this level of reduction, there are no issues with the final quality.”

Verifying Test Validity

Confirm whether the content of the test cases is designed to correctly verify the items to be checked. For example, if there is a test item like “Confirm that the temperature sensor value can be read correctly,” the validity of the test will be questionable unless the specific temperature range and conditions for the test are clear. In the review, we confirm whether the test objectives and methods correspond correctly and whether the expected results are clearly defined.

Early Detection of Issues

By finding issues at the test design stage, you can prevent rework after the tests are executed. Rather than realizing after running the tests that “this case was missing” or “the necessary preparation for testing was insufficient,” identifying these at the design stage can significantly reduce the cost of additional testing and corrections.

Additionally, in the process of reviewing the test design, ambiguities or contradictions in the original specifications or design documents may be found. This also leads to improvements in the development process and product quality.

Knowledge Sharing Across the Entire Team

Through reviews, members other than the testers can also understand the test content, promoting knowledge sharing across the entire team. When designers or implementers check the test cases, it can lead to insights such as “this test content might cause an issue.”

• View PDF for details

Timing of Test Design Reviews

It is effective to consider the timing for conducting test design reviews using the “V-model” explained in the article “Waterfall vs. Agile Development.” In the V-model, test processes are defined corresponding to each development process, and test design is performed at each stage.

A test design document is created at each stage, such as system testing corresponding to requirements definition, integration testing corresponding to basic design, and unit testing corresponding to detailed design, and reviews are conducted before implementation. By conducting reviews at this timing, test quality can be efficiently ensured while maintaining consistency between the development and testing processes.
Especially in embedded development, preparing hardware often takes time, so it is desirable to complete the test design and finish the review before the hardware is completed.

How to Conduct a Test Design Review

To conduct test design reviews effectively, it is best to proceed with the following steps.

Preparation

Before conducting the review, all participants should thoroughly read the test design document to be reviewed. In addition, they should also check the related specifications and design documents to ensure they are ready to correctly check that the tests cover what is defined in the specifications.
It is also important to share the purpose of the review and the points to be checked in advance. If what should be checked with emphasis and who is appropriate to check it are clear, the efficiency of the review will be significantly improved.

Conducting the Review

In the review meeting, the creator of the test design document explains the content, and the participants ask questions and make comments. Rather than just making superficial checks, it is important to have essential discussions, such as “What kind of defect are we trying to detect in this test case?” “Is this condition insufficient?” or “Conversely, are we spending too much time testing this part?”
In the case of embedded development, it is also necessary to consider hardware constraints and limitations of the actual hardware environment. For example, feasibility should also be discussed, such as “Testing under this condition is difficult to reproduce with the actual hardware,” and the testing method should be decided.

From the designer’s perspective, it is also advisable to use the review to confirm that testing will focus on areas likely to be weak points where defects may occur. Examples include cases where input interrupts from external signals occur, or where modules sharing memory or other resources operate in coordination.

Recording and Addressing Feedback

All feedback raised during the review is recorded, and the action plan is clarified. Critical issues require immediate correction, while minor improvement suggestions are addressed after determining their priority.
Once the feedback has been addressed, a re-check is performed to confirm that the issues have been resolved.

Points to Check in the Review

In a test design review, checks are specifically conducted from the following perspectives.

Traceability to Requirements

Whether test cases exist for all requirement specifications, and conversely, whether it is clear which part of the specifications each test case verifies.

Coverage

Whether all necessary perspectives, such as normal cases, abnormal cases, boundary values, and error handling, are covered. Especially in embedded development, cases such as power-on, power-off, sensor anomalies, and communication errors are important.

Specificity

Whether test procedures, input conditions, and expected results are described specifically and clearly. Ambiguous expressions may lead to different interpretations depending on the test executor.

Feasibility

Whether the test can be executed in the actual hardware environment, and whether the necessary hardware and tools are available.

Efficiency

Whether there are duplicate test cases verifying the same thing, and whether the test sequence is efficient.

• View PDF for details

Test Design Reviews Are the Key to Product Quality

Test design reviews are an important process that improves test quality and leads to the improvement of overall product quality. Especially in embedded software development, where product safety and reliability are of paramount importance, careful reviews at the test design stage are indispensable.
Rather than treating the review as a mere formality, verifying the validity and specificity of the test through essential discussions can achieve more effective testing. By understanding the importance of test design reviews and actively putting them into practice, you will gain a deeper understanding of product quality as a whole.