Searching for “automatic coverage maximization” results with ~16 million hits. Alas, this does not reflect 16 million solutions, rather it is an indication of the big interest in this topic, and the many attempts to automate the process of “getting full coverage, faster”. There are several tools available, performing smart coverage analysis and directing the tests to the less covered areas.
For example - with the Incisive Metric Center you can rank cover, and identify which tests contribute most to the coverage, and which tests are redundant.
If you use vManager as your runner, you can furthermore enhance this process, automatic rerun the most effective tests based on accumulated coverage.
Another approach to automate coverage maximization is Specman Constraint Solver’s Coverage Driven Distribution. This feature affects the probability of generating a value, based on the coverage model definition. Assume, for example, this coverage definition:
cover packet_ended using per_unit_instance is {
item size : uint cur_packet.size using
ignore = size > 2048,
ranges = {
range([0..1023], "illegal too short", UNDEF, 1);
range([1024], "legal shortest", UNDEF, 1);
range([1025..2047], "legal ", UNDEF, 1);
range([2048], "legal longest", UNDEF, 1);
};
};
By defining these ranges, we express the coverage goal – each of the buckets (the two buckets containing ranges, and the two buckets each containing just one value) are of same importance, and all of the buckets should be filled. Activating CDD on the code containing this coverage model, will result with packets that are not longer than 2048 (thanks to the ignore option); and, even more important – getting the edge values of 1024 and 2048 will have same probability as generating any value in the range of [1025..2047].
We realize that developing the perfect solution for coverage maximization would take some time. While waiting for the ideal solution that will “solve it all”, you can create your own utilities, targeted at specific use models. If you implement your verification environment in e, you can use the coverage api. The coverage api is a set of methods that enable getting information of the current model. The following code is a small example, demonstrating the possibilities. We implement a method named wait_for_full. This method samples the coverage database every scan_every cycles, and stops when getting indication that the required coverage goal was reached. The information of the current status of the coverage model is gathered by calling scan_cover(), and then extending the end_item() method and reading the item_grade field.
struct my_cover_struct like user_cover_struct {
!reached_goal : bool;
!goal_grade : int;
!req_instance_name : string;
wait_for_full( instance_name : string, struct_to_cover : string,
group_to_cover : string, item_to_cover : string,
goal_grade : int, scan_every : uint) @sys.any is {
me.goal_grade = goal_grade;
me.req_instance_name = instance_name;
var res : int;
while not reached_goal {
wait [scan_every] * cycle;
// scan_cover accepts a string, concatenation of the names of containing struct, group and item
res = scan_cover(append(struct_to_cover, ".",
group_to_cover, ".", item_to_cover));
};
};
end_item() is {
if instance_name == append("e_path==", req_instance_name) {
if item_grade/1000000 >= goal_grade {
reached_goal = TRUE;
};
};
};
};
And this is an example of using this utility. We want to run until the cross cover item cross__sma_state__smb_state (created by crossing the item smb_state and smb_state), in the coverage group named system_cover, in the unit system_monitor, gets the grade 80:
extend my_env {
run() is also {
start run_until();
};
run_until() @sys.any is {
var cover_sm_cross : my_cover_struct = new;
raise_objection(TEST_DONE);
cover_sm_cross.wait_for_full(me.system_monitor.e_path(),
"dummy_dut", "system_cover",
"cross__sma_state__smb_state", 80, 100);
drop_objection(TEST_DONE);
};
};
The covers api is simple to use, and can be used as basic building blocks in implementing a sophisticated Coverage Maximization algorithm. You can enhance it, for example, to print report of coverage progress throughout the test, or even to stop the run if seeing that there was no significant change in the coverage status in the last X cycles.
And while some of you will enjoy creating your own smart utilities, we, in Specman, will continue enhancing the language and the engines, to get us closer and closer to the full Automatic Coverage Maximization we all look for.
Efrat Shneydor,
Team Specman