Specman’s Callback Coverage API

Our customers’ tests have become more complex, longer, and consume more resources than before. This increases the need to optimize the regression while not compromising on coverage.

Some advanced customers of Specman use Machine Learning based solutions to optimize the regressions while some use simpler solutions. Based on a request of an advanced customer, we added a new Coverage API in Specman 19.09 called CoverageCallback. In 20.03, we have further enhanced this API by adding more options. Now there are two Coverage APIs that provide coverage information during the run (the old scan_cover API and this new Callback API). This blog presents these two APIs and compares between them while focusing on the newer one.

Before we get into the specifics of each API, let’s discuss what is common between these APIs and why we need them. Typically, people observe the coverage model after the test ends, and get to know the overall contribution of the test to the coverage. With these two APIs, you can observe the coverage model during the test, and hence, get more insight into the test progress.

Are you wondering about what you can do with this information? Let’s look at some examples.

1. Recognize cases when the test continues to run long after it already reached its coverage goal.
2. View the performance of the coverage curve. If a test is “stuck” at the same grade for a long time, this might indicate that the test is not very good and is just a waste of resource.

These analyses can be performed in the test itself, and then a test can decide to either stop the run, or change something in it configuration, or – post run. You can also present them visually for some analysis, as shown in the blog: Analyze Your Coverage with Python.

scan_cover API (or “Scanning the Coverage Model”)

With this API you can get the current status for any cover group or item you are interested in at any point in time during the test (by calling scan_cover()). It is very simple to use; however it has performance penalty. For getting the coverage grade of any cover group during the test, you should
1. Trigger the scan_cover at any time when you want the coverage model to be scanned.
2. Implement the scan_cover related methods, such as start_item() and end_bucket(). In these methods, you can query the current grade of group/item/bucket.
The blog mentioned earlier: Analyze Your Coverage with Python describes the details and provides an example.

Callback API

The Callback API enables you to get a callback for a desired cover group(s), whenever it is sampled. This API also provides various query methods for getting coverage related information such as what the current sampled value is. So, in essence, it is similar to scan_cover API, but as the phrase says: “same same but different”:

1. Callback API has almost no performance penalty while scan_cover API does.
2. Callback API contains a richer set of query methods that provide a lot of information about the current sampled value (vs just the grade with scan_cover).
3. Using scan_cover API, you decide when you want to query the coverage information (you call scan_cover), while with the Callback API you query the coverage information when the coverage is sampled (from do_callback). So, scan_cover gives you more flexibility, but you do need to find the right timing for this call.

There is no absolute advantage of either of these APIs, this only depends on what you want to do.  

Callback API details

The Callback API is based on a predefined struct called: cover_sampling_callback. In order to use this API, you need to:

1. Define a struct inheriting cover_sampling_callback (cover_cb_save_data below)
   1. Extend the predefined do_callback() method. This method is a hook being called whenever any of the cover groups that are registered to the cover_sampling_callback instance is being sampled.
   2. From do_callback() you can access coverage data by using queries such as: is_currently_per_type(), get_current_group_grade() and get_current_cover_group() (as in the example below) and many more such as: get_relevant_group_layers() and get_simple_cross_sampled_bucket_name().
2. Register the desired cover group(s) to this struct instance using the register() method.

Take a look at the following code:

// Define a coverage callback.
// Its behavior – print to screen the current grade.
struct cover_cb_save_data like cover_sampling_callback {
    do_callback() is only {
       // In this example, we care only about the per_type grade, and not per_instance
       if is_currently_per_type() {           
            var cur_grade : real = get_current_group_grade();
            sys.save_data (get_current_cover_group().get_name(), cur_grade);
        };//if
    };//do_callback()
};// cover_cb_send_data


extend sys {
    !cb : cover_cb_save_data;

   // Instantiate the coverage callback, and register to it two of my coverage groups
    run() is also {
       cb  = new  with {
        var gr1:=rf_manager.get_struct_by_name("packet").get_cover_group("packet_cover");
        .register(gr1);
        var gr2:=rf_manager.get_struct_by_name("sys").get_cover_group("mem_cover");
       .register(gr2);
       };//new  
    };//run()

  save_data(group_name : string, group_grade : real) is {
        //here you either print the values to the screen, update a graph you show or save to a db 
  };// save_data
};//sys

In the blog Analyze Your Coverage with Python mentioned above, we show an example of how you can use the scan_cover API to extract coverage information during the run, and then use the Specman-Python API to display the coverage interactively during the run (using plotting Python library - matplotlib). If you find this usage interesting and you want to use the same example, by the Callback API instead of the scan_cover API, you can download the full example from GIT from here: https://github.com/efratcdn/cover_callback.

Specman Team