Verification of ML IP and Specman—Our Hackathon Project

If you are lucky enough and your company spends a few working days each year on a Hackathon, you must know that it is usually a lot of fun. The latest 2018 Hackathon in Cadence was all about Machine Learning. We, in Specman R&D, debated a bit around how to approach the topic since Machine Learning means a lot of different things in our industry. Take a look at the following interesting article: Where ML works best in which Anirudh Devgan, president of Cadence, describes few main areas.

When you talk about verification and Machine Learning, usually two main things come in mind:

1. Using Machine Learning in Verification: We encountered multiple teams which use Specman for implementing machine learning techniques to optimize their regression. We have been getting different requests for more notifications and hooks, especially around coverage. And, we already have some of these requests on our roadmap.
2. Verification of Machine Learning IPs: For any team working on machine learning IP or chip, verification is a critical and a challenging task and if done right, it can get you a leading edge. This usually involves a significant large dataset, which means you need to carefully decide which samples to test, pick up the good corner cases, and get to sufficient coverage fast.It is a well-known fact that Specman is the best verification engine- it has the best constraint solver and the easiest aids to define coverage model. Therefore, we decided to challenge our tool and see it in action in a Machine Learning (ML) environment. This blog describes our small Hackathon project that few of us from the Specman team were working on. This project presents Specman strengths in an ML IP verification environment. 

What model are we verifying?

We decided that in favor of the Hackathon project, we are the verification team of a company that produces an IP of a weighted trained Neural Network (NN). We have decided to take a black box verification approach, meaning we are not verifying the NN internals but rather its outputs. Therefore, the exact usage of the NN was not that important for us but since image recognition is so useful and popular, we decided to go with it. In addition, we wanted to take advantage of the knowledge of the Machine Learning experts in Cadence.

These experts provided us with a Convolutional NN (CNN) comprising 1 input-layer, 5 hidden-layers, and 1 fully-connected layer written in Python on top of Tensorflow. In addition, they provided us with a dataset called mnist-fashion containing small images of 10 different types of clothing items (shirt, dress etc.). In our Hackathon project, we aimed that our IP should be able to identify these clothing items. To make it more interesting, we decided that our IP will be used for identifying clothing items in an image constructed of 4 different clothing items (for example Shirt-Dress-Sandals-Trouser). Now, three things needed to happen:

1. The Python SW model should be trained with a set of images. Our Machine Learning experts divided the dataset of images into a training set and a classification set and started training the CNN with the training set.
2. The SW model should be converted into HDL model, defiantly a challenging and interesting task. However, since this is a time-consuming task and our main focus is on verification, we skipped this stage and decided to use some “dummy” HDL.
3. The verification of this HDL model- is the focus of this Hackathon project.

The verification

In our little project, we want to perform black box verification to the HDL model. This means that without knowing its inner implementation and algorithms we just want to verify whether it is doing what it was designed to do- in our case identify clothing items in an image of 4 items. The Python SW model will be used as the reference model. Though in the real world there should be differences between what the HDL model is able to do as compared to the Python SW model, but this only makes the verification effort more important and challenging.  

What kind of images we want to use for the verification?

Now, we have what we previously called the “classification dataset” of images that we wanted to use to verify that our trained dummy HDL Model can identify just as our Python SW model. As decided earlier, we need to identify images composed of 4 different clothing items and also need to verify all 10 possible clothing items. However, there are also other important factors such as the position in the image, and moreover, if there are specific types it is important to verify while they are next to one another? etc. Well, this requires defining a model including a few constraints which is quite natural to do with e. Let’s take the following constraints for example in which ‘items’ is a list containing 4 cloth types:

// each combined image should have 4 different cloth types

keepsoft items.all_different(it.cloth_type);

// have many coats

keepsoft cloth_type == select {

        30 : Coat;

        70 : others;

    };

// never have an image containing both dress and skirt

keep (skirt notin cloth) or (dress not in cloth);

How will we construct this image of 4 different items?

Remember, we have a dataset of single clothing items and we want to use Specman generation to pick up these specific 4 small images and their positions, but “who” will construct this full image? Logically, the right thing to do instead of inventing the wheel is to use some existing Python API. We found the Python algorithm, ComputerVision (thanks again to our ML guys). Now we were left with the question of how to activate it from Specman. Python has a C interface. Specman has two flavors of C interface, a basic one and an advanced one which is called FLI. The basic C interface is easy to use and is sufficient for our project, so we decided to use it. We defined the method that constructs an image out of 4 small images in e as a C function. In the C implementation, we had to write some code to call the Python API. Once we found the Python algorithm, ComputerVision, it took one of our team members few hours to get it working.

How do we activate the reference model?

Our CNN was written in Python using Tensorflow. Once we were already using Python API for constructing the image, we just needed to do the same for calling the Python SW model. Calling Python from e the second time was obviously easier and faster.

How would we know which images and combinations were tested?

Naturally, we want to know whether we have covered all types, what combinations were covered and what we did not cover etc. while there are a few different interesting combinations. The fastest way for us to do so was to use Defined As Macro to generate the different coverage items. Using a coverage cross item would have been a more elegant and robust solution but we assumed that using the macro was the fastest way.

Running the verification

Now that we had all the parts working, we only needed to push the simulation “play” button and all the parts came into life as shown in the following workflow in a loop:

1. Specman picks a combination of 4 items according to the defined constraints.
2. Specman calls the ComputerVision API to construct a new image.
3. Specman calls the Python SW model with this new constructed image and gets a result of 4 items identified.
4. Specman sends this image also to the HDL Model and gets a result (hmmm… not really, as we said at the beginning, we used some dummy HDL model that usually produces the same result as the Python SW model, except in a few odd cases).
5. Specman “checks” the two results and compares them.

When the test ended, we were left with the result of the passed and failed cases and our coverage model. For those of you who were using TensorFlow in a parallel mode, yes, we might have been able to do things more productively in TensorFlow, but remember that this is a short project and we had time constraints…

How did the coverage report look like?

All 10 clothing types were covered.

For the different combinations, you can see that we have never had the combination of: Dress- Sandal- Sneaker-T-shirt. Since we defined 20 as the “at_least” value for each combination, you can see that we got 4 Dress-Sandal-T-shirt-Trouser (second line: 20%).

Summary

First things first, as a team we had a lot of fun- the whole exercise was exciting and very different compared to what we usually do in our daily work. Moreover, it was very fulfilling to see Specman’s strengths in action in a relatively new domain. It is well known that Specman is the best verification engine, but it seems that as a tool it can especially shine in a domain which involves massive amount of data, where you need to pick the most important pieces of data and monitor them. Furthermore, SW engineers may find these capabilities useful for validating their SW and can use Specman as a stand along application without the simulator. More information will come later, stay tuned…

Orit Kirshenberg

Specman team