Structural Challenges in Embedded Development

Why Is the Scale of Embedded Software Expanding?

Embedded software is developed for the purpose of controlling specific hardware. Its underlying premises differ from those of general-purpose software (see “Embedded vs. General-Purpose Software”).

In recent years, many products, such as home appliances, industrial equipment, and medical devices, have been equipped with sensing and communication functions. As a result, the role played by software is rapidly expanding. In addition to simple control, software is increasingly being tasked with data collection and decision-making. The features visible to users are also becoming more abundant year by year. Even in machine tools and medical devices, the inclusion of safety and communication functions is becoming standard.

The Information-technology Promotion Agency, Japan (IPA), conducts annual surveys on trends in the embedded and IoT industries. In the fiscal year 2022 survey, questionnaires were distributed to 7,864 organizations, and 1,214 responses were collected. In this survey, functional enhancements and adoption of new technologies were cited as challenges. Quality assurance and information security are also growing in importance. These changes directly lead to an expansion in development scale.

As many as 68% of companies cited “securing and strengthening human resources” as a management challenge. This percentage is significantly higher than other items. Separately, the IPA continuously collects project data from domestic embedded software development companies.

There are three main factors behind the expanding scale.

• Accumulation of added functions: Even after a product is released, functions are added to meet customer requests and respond to competitors.
• Increase in derivative models: Development that derives multiple models from a common base software is spreading.
• Support for communication and IoT: In addition to controlling individual devices, cloud integration and security support have become essential.

When these accumulate, they create complexity that was not anticipated in the initial design. Even if each individual change is small, the cumulative impact on the overall structure is significant. As a result, the risk of a single change affecting unexpected areas also increases. A system that can track change history is also required.

Relationship Between Development Methods and Increasing Test Man-Hours

According to the IPA’s fiscal year 2022 survey, “Waterfall/V-model development” accounted for 34% of development methods. This figure was higher than other development methods, making it the most common result. On the other hand, “Agile development” was limited to 12%, the lowest result. In embedded development, hardware specifications are often finalized first. Therefore, it is considered to be well suited to the V-model, which proceeds through each stage in sequence.

However, the V-model does not handle returning to earlier stages well. The further the process progresses, the greater the cost of backtracking. As the scale expands, the impact of rework also increases (see “Waterfall vs. Agile Development”).

An expansion in scale directly leads to an increase in test man-hours. The IPA has published the “Software Development Analysis Data Book 2022.” In it, the distribution of actual man-hours for new development is analyzed. The man-hours for integration testing were 20.1%, and system testing was 11.9%. Combined, they reached 32.0%, accounting for nearly one-third of the entire development. Although this figure is an aggregate not limited to embedded systems, it serves as a useful reference. The workload of the testing process snowballs as the scale expands.

In embedded development, in addition to this, verification of compatibility with hardware is required. Even if the software works normally on its own, defects may occur when combined with actual equipment or sensors. Similar problems can occur when combined with communication modules. Environmental conditions such as temperature and vibration also affect operation. Lot differences and individual variations in hardware can also cause defects.

As the number of derivative models increases, verification of combinations for each model becomes necessary. Test cases increase multiplicatively rather than through simple addition. According to the same data book, in new development, there is a tendency to conduct 37.63 integration tests and 9.41 system tests per 1,000 lines.

As a countermeasure, narrowing down combinations by organizing testing perspectives is effective. By verifying from high-priority combinations, it becomes easier to detect critical defects even with limited man-hours. In addition, the man-hours for regression tests, which are conducted repeatedly, can be reduced through automation. The strength of automated testing is that it can accurately repeat the same steps any number of times. If you set up a test environment, you can proceed with verification even at a stage where actual equipment is not used. If you assume only manual testing, man-hours will not be able to keep up with the speed of scale expansion.

• View PDF for details

The Essence of Labor Shortages: Both Hardware and Software Knowledge Are Required

For embedded software developers, programming knowledge alone is not enough. An understanding of electrical circuits, mechanisms, and communication protocols is also required (see “Constraints in Embedded Development”). For example, in motor control, an understanding of current and noise behavior is essential. Simply transferring general-purpose software developers is not sufficient. The reality is that they are unlikely to become immediately effective. A certain amount of experience is required to understand hardware constraints. In addition to classroom learning, experience with actual equipment is also indispensable.

In the IPA’s FY2022 survey, companies were asked about their efforts to secure human resources. Fifty-five percent of companies cited "hiring to fill shortages." This was higher than other efforts and was the most common response. On the other hand, "reskilling in-house talent" stood at only 18%, making it the least common effort. Competition for recruitment is intensifying across the entire industry. Relying solely on replenishment through hiring intensifies the competition to acquire talent with hardware knowledge. The reality is that replenishment is not progressing as desired.

A realistic solution is to establish a system for training. By preparing design documents and test Specifications, even inexperienced engineers can proceed with work while referring to them. Pair work, which pairs veterans with younger staff, is also useful for passing down hardware knowledge. Relying solely on OJT (On-the-Job Training) places an uneven burden on instructors.

Furthermore, it is important to establish an environment in which hardware behavior can be simulated in software. This has the advantage of allowing learning and verification to proceed even without the actual equipment. Such a system also leads to a shorter training period.

Risks of Dependency on Specific Individuals and Their Resolution

In embedded development, there is a tendency for code to be understood only by specific personnel. The same thing happens with jigs and work procedures. The greater the reliance on specific individuals, the more burdensome handovers become. If a person in charge is transferred or leaves the company, maintenance and modifications may be delayed. Similar problems arise in testing as well. Much of a veteran’s judgment is based on experience. Such details are difficult for others to see (see “Turning Tester Experience and Tacit Knowledge into Explicit Knowledge”).

To eliminate dependency on specific individuals, it is essential to capture tacit knowledge in documents and checklists. Listing test perspectives and past defect trends makes it easier to maintain quality standards even when the person in charge changes. This is a great help, especially for new personnel. Establishing code reviews as a system is also effective. It serves as a means to transfer knowledge from individuals to the organization. A sharing mechanism forms the foundation for preventing dependency on specific individuals.

For jigs and test environments as well, configurations and operating procedures must be recorded. Especially for jigs, reproducing them becomes difficult if drawings and setting values are not preserved. Eliminating dependency on specific individuals is not a one-time effort. It requires continuously recording and sharing information during daily development.

• View PDF for details

Responding to Rising Quality Demands

Embedded devices are products for which the cost of handling defects after they are released to the market is high. If recalls or on-site support occur, both the cost and man-hours will be significant. It takes a long time to regain trust once it is lost. In fields related to safety, compliance with standards is also required. The importance of ensuring quality is higher than ever before. Quality issues also affect corporate credibility.

To meet quality demands, building quality into upstream processes is essential. Thorough reviews are conducted at the requirements and specifications stages. The basic approach is to reduce rework in downstream processes. Building quality in during the design stage also directly leads to reducing the burden on downstream processes. In addition, trends in defects found during testing are analyzed. A system to reflect these in the design guidelines of the next project is also effective.

Shifting the Focus to Increasing Productivity per Person

There is a common solution to the five challenges we have seen so far. It is not to increase headcount. It is not easy to recruit engineers with hardware knowledge. Training also takes a corresponding amount of time. An ideal system is one that can maintain results even with a limited number of people. A realistic solution is to restructure development practices to increase productivity per person. The concept of raising overall organizational productivity is essential.

Particularly effective is establishing an environment that reduces the time required for testing and verification. Introducing test automation tools can reduce the manpower required for repetitive testing. This is also an area where the return on investment is relatively easy to see. A verification environment that can reproduce hardware behavior is similarly useful. The freed-up time can be allocated to design reviews and work to eliminate dependency on specific individuals. This makes it easier to achieve both quality and efficiency.

Scale expansion, labor shortages, and dependency on specific individuals are not independent problems. Preparing test environments and development processes addresses multiple challenges simultaneously. Accumulating small improvements leads to significant effects. It is also important for management and development teams to have a common understanding. Once they understand the structure correctly, development teams need to prioritize and take action.

• View PDF for details

Summary

Embedded software development continues to increase in scale and complexity due to the addition of functions and an increase in derivative models. Increasing test man-hours, labor shortages, dependency on specific individuals, and rising quality demands are structural challenges that are interrelated. These cannot be resolved by solving just one of them. Rather than relying solely on increasing headcount, a concept of increasing productivity per person through test automation and improving the development environment is required.