Quality Assurance and Health Management for Software Systems

Companies have experienced an increased frequency of failures due to defective software that have cost millions of dollars. Recent high-profile incidents have drawn increased scrutiny to the risks of failed software systems to the public. Yet, aside from the Therac-25 case, very few incidents of software failure causing humans harm have been proven or widely reported. With increased government oversight and the expanded use of social networking for real time reporting of problems, the potential for major injury or death related to software failures is only beginning to be understood. However, debugging defective software can be costly and time consuming. Moreover, undetected bugs could induce great harm to the public when software systems are applied in safety-critical areas, such as consumer products, public infrastructure, transportation systems, etc. Therefore, it is vital that software developers remove these bugs as early and as completely as possible. In this dissertation, a review of existing catastrophic accidents caused by software failures is conducted first. By analyzing the accidents, a more comprehensive understanding towards software quality assurance can be achieved. This understanding includes the nature of software failures as well as lessons learned regarding the implications for future software system development. To ensure the quality of software systems, various software testing methodologies have long been deployed in both academia and industry. Combinatorial Testing (CT) has proven to be a good candidate to both maintain the strength of the test set and reduce the set size at the same time. Therefore, CT is used in this dissertation for the quality assurance of several industrial software systems, including a dashboard software of subway control system, NeoKylin Operating System, and DaMeng Database Management System. The results demonstrate that CT can be successfully applied to various software systems to not only improve the effectiveness of the test sets but also reduces costs in the testing process. However, traditional testing methodologies encounter issues such as low efficiency when dealing with a special type of software systems, namely, Software Product Lines (SPLs). Unlike traditional software systems, a SPL is a group of software-intensive systems that share a common, managed set of features and can be developed from a common set of core assets in a prescribed way. By adopting the concept of SPL, companies can significantly reduce their capital investment and man power in developing software for multiple similar products. What should be emphasized is that these advantages turn out to be meaningless if the quality of these products are relatively low. Therefore, a Genetic Algorithm (GA)-based test generation technique is proposed which integrates fault localization metrics with a focus on how test cases can be generated for SPLs and be applied to test other products of the same family. Moreover, it has been widely acknowledged that it is insufficient to only focus on the last phase — testing and debugging — of software systems in their entire life cycles to ensure their quality. It has been proven that the later in the life cycle that a bug is detected, the more costly it will be to fix the bug. As a result, SAdoc — a comprehensive software health management framework — is also proposed in this dissertation. This framework helps engineers perform quality evaluation of the software under development in the early phase of the software life cycle. It also provides hints on which aspects to improve the structure of their systems. It also provides hints on which aspects of the program need improvement to ensure the quality of the structure of the program’s system.