Abstract
A rule-base coverage analysis method has been developed which provides an assessment of both the rule-base under review and the test set that has been used for evaluation. Lack of coverage can result from either incompleteness of the test data or errors in the rule-base. A series of heuristics have been developed which use coverage information and meta-knowledge about the larger population to suggest additional test cases from the population, in the event that the initial test set is incomplete. This forms the basis of an incremental approach which allows us to both increase completeness of the test suite and improve coverage of the rule-base. Rule-based system testing usually faces the difficult dual problems of incompleteness and errors in both the rule-base and the test data. Performance of a system on a limited suite of test data is never sufficient to predict performance on a larger set of data in routine use without additional assumptions. An important one of these is the assumption of representative coverage of the population for which the system is intended. The heuristic approach to test data selection is demonstrated using information generated by TRUBAC, a tool which implements the coverage analysis methods. We have applied these techniques to analyze a number of prototype systems for diagnosis of rheumatological diseases. In addition, we demonstrate the use of coverage information to identify class dependencies and guide rule-base pruning. We also introduce a complexity metric for rule-bases. Finally, we discuss extensions of the coverage measures for rule-based systems with dynamic computation of certainty factors.