In the former blog about Python and Specman: Specman: Python Is here!, we described the technical information around Specman-Python integration. Since Python provides so many easy to use existing libraries in various fields, it is very tempting to leverage these cool Python apps.
Coverage has always been the center of the verification methodology, however in the last few years it gets even more focus as people develop advanced utilities, usually using Machine Learning aids. Anyhow, any attempt to leverage your coverage usually starts with some analysis of the behavior and trends of some typical tests. Visualizing the data makes it easier to understand, analyze, and communicate. Fortunately, Python has many Visualization libraries.
In this blog, we show an example of how you can use the plotting Python library (matplotlib) to easily display coverage information during a run. In this blog, we use the Specman Coverage API to extract coverage data, and a Python module to display coverage grades interactively during a single run and the way to connect both.
Before we look at the example, if you have read the former blog about Specman and Python and were concerned about the fact that python3 is not supported, we are glad to update that in Specman 19.09, Python3 is now supported (in addition to Python2).
The Testcase
Let’s say I have a stable verification environment and I want to make it more efficient. For example: I want to check whether I can make the tests shorter while hardly harming the coverage. I am not sure exactly how to attack this task, so a good place to start is to visually analyze the behavior of the coverage on some typical test I chose. The first thing we need to do is to extract the coverage information of the interesting entities. This can be done using the old Coverage API.
Coverage API
Coverage API is a simple interface to extract coverage information at a certain point. It is implemented through a predefined struct type named user_cover_struct. To use it, you need to do the following:
- Define a child of user_cover_structusing like inheritance (my_cover_struct below).
- Extend its relevant methods (in our example we extend only the end_group() method) and access the relevant members (you can read about the other available methods and members in cdnshelp).
- Create an instance of the user_cover_structchild and call the predefined scan_cover() method whenever you want to query the data (even in every cycle). Calling this method will result in calling the methods you extended in step 2.
The code example below demonstrates these three steps. We chose to extend the end_group() method and we keep the group grade in some local variable. Note that we divide it by 100,000,000 to get a number between 0 to 1 since the grade in this API is an integer from 0 to 100,000,000.
!cur_group_grade:real;
end_group() is also {
}
};
!cover_info : my_cover_struct;
start monitor_cover ();
};
cover_info = new;
while(TRUE) {
wait [10000] * cycles;
compute cover_info.scan_cover("packet.packet_cover");
};//while
};// monitor_cover
};//sys
Pass the Data to a Python Module
After we have extracted the group grade, we need to pass the grade along with the cycle and the coverage group name (assuming there are a few) to a Python module. We will take a look at the Python module itself later. For now, we will first take a look at how to pass the information from the e code to Python. Note that in addition to passing the grade at certain points (addVal method), we need an initialization method (init_plot) with the number of cycles, so that the X axis can be drawn at the beginning, and end_plot() to mark interesting points on the plot at the end. But to begin with, let’s have empty methods on the Python side and make sure we can just call them from the e code.
# plot_i.py
def init_plot(numCycles):
print (numCycles)
def addVal(groupName,cycle,grade):
print (groupName,cycle,grade)
def end_plot():
print ("end_plot")
And add the calls from e code:
addVal(groupName:string, cycle:int,grade:real) is imported;
end_group() is also {
cur_group_grade = group_grade/100000000;
addVal(group_name,sys.time, cur_group_grade);
};//user_cover_struct
init_plot(numCycles:int) is imported;
end_plot() is imported;
start scenario();
};
init_plot(numCycles);
{
//Here you add your logic
cover_info = new;
var num_items:= cover_info.scan_cover("packet.packet_cover");
end_plot();
}//sys
- The green lines define the methods as they are called from the e
- The blue lines are pre-defined annotations that state that the method in the following line is imported from Python and define the Python module and the name of the method in it.
- The red lines are the calls to the Python methods.
Before running this, note that you need to ensure that Specman finds the Python include and lib directories, and Python finds our Python module. To do this, you need to define a few environment variables: SPECMAN_PYTHON_INCLUDE_DIR, SPECMAN_PYTHON_LIB_DIR, and PYTHONPATH.
The Python Module to Draw the Plot
After we extracted the coverage information and ensured that we can pass it to a Python module, we need to display this data in the Python module. There are many code examples out there for drawing a graph with Python, especially with matplotlib. You can either accumulate the data and draw a graph at the end of the run or draw a graph interactively during the run itself- which is very useful especially for long runs.
Below is a code that draws the coverage grade of multiple groups interactively during the run and at the end of the run it prints circles around the maximum point and adds some text to it. I am new to Python so there might be better or simpler ways to do so, but it does the work. The cool thing is that there are so many examples to rely on that you can produce this kind of code very fast.
import matplotlib
import matplotlib.pyplot as plt
plt.ion()
ax = fig.add_subplot(111)
class CGroup:
self.name = name
self.XCycles=[]
self.XCycles.append(cycle)
self.YGrades=[]
self.YGrades.append(grade)
self.line_Object= ax.plot(self.XCycles, self.YGrades,label=name)[-1]
self.firstMaxCycle=cycle
self.firstMaxGrade=grade
self.XCycles.append(cycle)
self.YGrades.append(grade)
if grade>self.firstMaxGrade:
self.firstMaxGrade=grade
self.firstMaxCycle=cycle
self.line_Object.set_xdata(self.XCycles)
self.line_Object.set_ydata(self.YGrades)
fig.canvas.draw()
class CData:
groupsList=[]
found=0
if groupName in group.name:
group.add(cycle,grade)
found=1
break
obj=CGroup(groupName,cycle,grade)
self.groupsList.append(obj)
for group in self.groupsList:
left, right = plt.xlim()
x=group.firstMaxCycle
y=group.firstMaxGrade
#ax.annotate("first\nmaximum\ngrade", xy=(x,y),
#xytext=(right-50, 0.4),arrowprops=dict(facecolor='blue', shrink=0.05),)
plt.scatter(group.firstMaxCycle, group.firstMaxGrade,color=group.line_Object.get_color())
#Add text next to the point
text='cycle:'+str(x)+'\ngrade:'+str(y)
plt.text(x+3, y-0.1, text, fontsize=9, bbox=dict(boxstyle='round4',color=group.line_Object.get_color()))
myData=CData()
def init_plot(numCycles):
plt.xlabel('cycles')
plt.ylabel('grade')
plt.title('Grade over time')
plt.ylim(0,1)
plt.xlim(0,numCycles)
def addVal(groupName,cycle,grade):
myData.add(groupName,cycle,grade)
def end_plot():
plt.ioff();
myData.drawFirstMaxGrade();
#Make sure the plot is being shown
plt.show();
#addVal("xx",1,0)
#addVal("yy",1,0)
#addVal("xx",50,0.3)
#addVal("yy",60,0.4)
#addVal("xx",100,0.8)
#addVal("xx",120,0.8)
#addVal("xx",180,0.8)
#addVal("yy",200,0.9)
#addVal("yy",210,0.9)
#addVal("yy",290,0.9)
#end_plot()
In the example we used, we had two interesting entities: packet and state_machine, thus we had two equivalent coverage groups. When running our example connecting to the Python module, we get the following graph which is displayed interactively during the run.
When analyzing this specific example, we can see two things. First, packet gets to a high coverage quite fast and significant part of the run does not contribute to its coverage. On the other hand, something interesting happens relating to state_machine around cycle 700 which suddenly boosts its coverage. The next step would be to try to dump graphic information relating to other entities and see if something noticeable happens around cycle 700.
To run a complete example, you can download the files from: https://github.com/okirsh/Specman-Python/
Do you feel like analyzing the coverage behavior in your environment? We will be happy to hear about your outcomes and other usages of the Python interface.
Orit Kirshenberg
Specman team