Test Coverage Criteria
In software testing, test coverage is commonly used as an objective indicator of the extent to which testing has been completed. Particularly in embedded systems, it is important to understand and appropriately use coverage criteria such as C0, C1, and MC/DC to meet quality requirements and safety standards.
On the other hand, it is important to note that “100% coverage = 100% quality” is not true. Coverage is merely a metric for visualizing the extent to which testing has been performed, and it only becomes effective when combined with requirements-based testing and reviews.
This article clarifies the differences between C0, C1, and MC/DC, and explains how to appropriately use each in embedded development.
- What Is Test Coverage?
- What Is C0 (Statement Coverage)? The Basics of Coverage
- C1 (Branch Coverage): Common in Practice
- MC/DC (Modified Condition/Decision Coverage): Areas Requiring High Quality
- Choosing Between C0, C1, and MC/DC: Selecting Based on Quality Requirements
- Points for Utilizing Coverage Measurement
What Is Test Coverage?
Test coverage is a commonly used quantitative indicator for objectively evaluating the status of test execution. First, we will review the basic concept of coverage and the reasons why it is emphasized in embedded development.
Overview and Importance of Test Coverage
Test coverage is a metric that quantitatively indicates the extent to which testing has been performed on software code or specifications. Because it can reveal gaps in testing, it plays an important role in fulfilling accountability for quality.
However, even if coverage is 100%, it does not mean all defects can be detected. This is because errors in specifications or deficiencies in the test cases themselves cannot be determined by coverage alone.
Why It Is Needed in Embedded Development
In recent years, the reliability requirements for embedded systems used in automobiles, industrial equipment, and medical devices have become more stringent. Therefore, coverage measurement, which enables the quantitative identification of testing gaps, is adopted in many development environments.
In addition, integration into CI/CD pipelines is also becoming common.
Types of Coverage
Common types of code coverage include the following:
- C0: Statement coverage—executes all executable statements at least once
- C1: Branch (decision) coverage—executes all branches with both true and false
- C2: Condition coverage—evaluates all conditions with both true and false
- MC/DC: Modified condition/decision coverage—confirms that each condition independently affects the decision outcome
The higher the quality requirements, the more advanced coverage criteria are required.
What Is C0 (Statement Coverage)? The Basics of Coverage
C0 (statement coverage) is the most basic coverage criterion and is often the first criterion introduced in a project. By understanding its characteristics and suitable use cases, it becomes easier to grasp the differences from other coverage criteria.
Definition and Features of C0
C0 is the most basic coverage criterion and can be achieved by executing all statements at least once. Because it is easy to measure and requires relatively little testing effort, it is introduced first in many development environments.
However, because C0 is an indicator that executes executable statements in the source code at least once, it cannot guarantee the validity of execution paths or the evaluation results of conditional expressions. In addition, in embedded development, differences may occur depending on the measurement method (source-code measurement/object-code measurement), and handling dead code or initialization code can sometimes be an issue.
Cases Where C0 Is Effective
C0 is suitable for situations where you want to prevent missed testing first, such as in the early stages of development or in small-scale projects. It is also effective as a monitoring indicator for regression testing using CI/CD.
However, in systems where safety is emphasized, quality assurance using only C0 is difficult, and combination with higher coverage criteria is desirable.
C1 (Branch Coverage): Common in Practice
C1 (branch coverage) is a practical coverage criterion widely adopted in embedded software development. This section explains the differences from C0 and the reasons why it is chosen in practice.
Definition and Features of C1
In C1, both true and false are executed for all branches. Therefore, branch processing and error paths that are missed in C0 can also be checked.
For example, for a condition like “if (A)”, it is necessary to test both the case of “A = true” and the case of “A = false”.
Differences from C0
While the purpose of C0 is to execute statements, C1 requires passing through both the true and false sides of branches. Since error conditions and exception handling are also naturally covered, this leads to improved quality.
However, in the case of a compound condition like “if (A && B)”, C1 can satisfy branch coverage with only two cases, “A = true, B = true” and “A = false, B = true”, so the behavior when “B = false” cannot be confirmed. Furthermore, in programming languages often used in embedded development such as C/C++, short-circuit evaluation is performed, so when “A = false”, B is not evaluated. For this reason, higher coverage is required to guarantee the evaluation inside conditional expressions.
Cases Where C1 Is Effective
In general embedded software, C1 is considered the most cost-effective coverage criterion. Because it is often adopted for unit testing of mass-produced devices or as a quality gate for CI/CD (a criterion for stopping merging or releasing if a specified threshold is not met), it is widely adopted as a standard criterion in practice.
C1 is considered a highly cost-effective criterion because the increase in test cases is relatively small and it is effective in detecting defects in exception handling and fail-safe processing.
MC/DC (Modified Condition/Decision Coverage): Areas Requiring High Quality
In systems where safety is required, higher coverage than C1 may be necessary. This section introduces the features of MC/DC and the background of its adoption.
Definition and Features of MC/DC
MC/DC is coverage that confirms that each condition independently affects the decision outcome. Rather than simply covering the true/false of conditions, it is necessary to show in tests that “the decision outcome changes when only the target condition is changed while other conditions are fixed”.
Example: In the case of A && B
- Cases where only A is changed and the result changes
- Cases where only B is changed and the result changes
This concept of “fixing other conditions” is the essence of MC/DC, and is clearly different from simple condition coverage (evaluating true/false once each). Therefore, in DO-178C for aircraft software, structural coverage equivalent to MC/DC is required at the highest level (Level A/DAL A). In ISO 26262 for the automotive field, higher-level coverage criteria (branch coverage, MC/DC, etc.) are recommended as the ASIL increases, and in ASIL D in particular, the application of MC/DC is often strongly required in practice.
Note that the required coverage varies depending on the standard level. For example, DO-178C (Level B) requires Decision Coverage (C1: branch coverage).
Differences from C2 (Condition Coverage)
While C2 (condition coverage) evaluates the true/false of each condition once, MC/DC verifies that each condition independently affects the decision outcome. For example, in the case of A && B, C2 is satisfied even with two cases of (A, B) = (true, false) and (false, true), but MC/DC requires test pairs that show the decision outcome changes when only the target condition is changed while other conditions are fixed, such as (true, true), (false, true), and (true, false).
Note that even if C2 is 100%, C1 is not necessarily 100%.
Reasons for Requiring MC/DC and Challenges
The biggest advantage of MC/DC is that it makes it easy to detect defects hidden in complex conditional expressions. In addition, high coverage can be achieved with fewer test cases than testing complete combinations of conditions.
On the other hand, test design is generally difficult and the target code is limited. In practice, dedicated tools are often used to apply it mainly to safety-related functions.
Choosing Between C0, C1, and MC/DC: Selecting Based on Quality Requirements
Which coverage criterion to adopt depends on the product’s quality requirements, risks, and development costs. It is important to select the appropriate coverage after understanding the characteristics of each.
Relationship Between Required Quality and Coverage
Coverage is selected according to the quality required for the product.
- Small-scale, low-risk products: C0
- General embedded products: C1
- Products compliant with functional safety standards: MC/DC (DO-178C, ISO 26262, etc.)
However, it is not necessary to aim for the highest level of coverage for all products, and selection according to risk is important.
Balance Between Cost and Effectiveness
Generally, test effort increases as you progress from C0 to C1 and MC/DC. Therefore, adopting higher coverage than necessary may significantly increase development costs.
In practice, hybrid approaches such as “using C1 as the standard and applying MC/DC only to critical functions” are common and are often adopted as a way to balance quality and cost.
Target Values for Coverage
While 100% coverage is one of the goals, it does not guarantee quality by itself. Areas that do not meet the target are classified into (1) unreachable (defensive code, etc.), (2) difficult to test (hardware-dependent), and (3) simple omissions, and for (1) and (2), the rationale and review records are kept. It is also important to supplement this by performing additional testing as necessary.
Points for Utilizing Coverage Measurement
Coverage measurement is not intended simply to increase numerical values. This section explains the approach to improving quality and efficient operation methods.
Do Not Rely Solely on Coverage
Coverage is one of the indicators used to measure quality, and is not a sufficient condition. Higher quality can be achieved by combining it with requirements-based testing, testing under abnormal conditions, and design reviews.
Efficiency Improvement by Utilizing Measurement Tools
Today, almost all coverage measurement tools can integrate with CI/CD. Utilizing automated measurement and report generation features leads to more efficient quality control and reduces the burden of creating records.
Start with C1
In general embedded development, it is realistic to first introduce C1 as a quality standard. As mentioned above, C1 is a cost-effective coverage criterion, with a relatively small increase in test cases, and branch coverage is highly efficient in bug detection. In addition, step-by-step introduction of C0 → C1 → MC/DC is realistic in practice, and C1 is particularly effective in covering exception handling and fail-safe processing.
Furthermore, limiting MC/DC to functions where safety is required makes it easier to balance quality and development costs.
