Last week, a favorite customer of mine called me in a panic, just days from tape-out of a large multimedia SoC. After a minor change in their RTL code their Specman testbench started crashing, even though the e code wasn't changed. Could I help?
Knowing that this customer compiles their e code, and that Specman doesn't tend to crash, the first thing I did was to get them to recompile the e code with the debug switch set to irun: "irun -sncompargs -debug". This turns off some of the compiler optimizations. Significantly in this case it turns on null pointer checks in the user's code, since these checks are normally turned off to get higher performance. With debug enabled, the test was re-run and we quickly saw the point of failure: a call to the vr_ad register model was dereferencing a null pointer. Phew! At least I knew Specman's reputation wasn't about to be blemished by some random crash, but why would a minor RTL change cause such a dramatic effect in the testbench?
Knowing the scale of the testbench, and the proximity of the tape-out, I figured we would track down the null pointer problem another day. The more immediate problem was to identify where the RTL had gone wrong so it could be fixed ASAP, but how to pinpoint that bug on such a big chip, from the other end of the phone without even knowing the design? A big challenge...
First we tried running the simulation using UVM FULL verbosity and comparing the log against the previous iteration of the RTL, but this was slow and not particularly easy. I needed a better solution, and quick.
What I hit upon was to take the binary search approach: split the problem into RTL and testbench, and determine which side of the boundary the problem occurred. To enable such an approach, I wrote a small e file that loaded on top of Specman, using the e language's powerful Reflection API to scan the entire design for any "simple_port" connections to the RTL. By fetching the ports' hdl_path() attributes, the script dynamically created "probe" commands for all the ports. My customer then loaded this e code into her "good" and "bad" RTL versions, saving the ~4000 waveforms into two separate database files.
Next we loaded the databases into SimVision and used the powerful SimCompare feature to locate the differences between the two simulations. We nailed the problem in moments: the RTL change had left a register without a reset, leading to large X propagation problems once the reset signal was de-asserted. 
Figure 1: Results from a SimCompare analysis highlighting the differences between identical signals from two different databases
With hindsight I found myself thinking, could we have debugged this any faster? Perhaps, if we'd known the exact nature of the RTL changes and where in the design those changed modules were instantiated, we could have started from the changed file and worked forward towards the testbench, but that doesn't work if you don't know every last detail about the design. As an e expert, could I have debugged back from a vr_ad call to understand why that pointer was null? Probably, but it was most likely not an easy thing to trace; after all we didn't know if we should be tracing a wrongly null pointer or an errant call that used a legitimately null pointer.
As it stands, I'm happy that we took the most efficient route; the time to write the e code was minimal, and it made the analysis really simple. Best of all, it required no design knowledge and is totally reusable on any Specman testbench, which is quite a result! All thanks to Reflection and SimVision's SimCompare GUI...
For the curious, here's the e code, a mere 83 active lines including a convenience macro and debugging output code.
Steve Hobbs
>// Dumps an ncsim Tcl script for probing all the signals that Specman is connected to.
//
// Usage:
//
// load sn_waveports;
// wave_ports [-db <name>] [-tb <name>] [unit_instance]
// Where -db is the ncsim SHM database name to generate.
// -tb specifies the VHDL testbench name to be stripped off.
// <unit_instance> is something like sys.tb_env.my_agent.smp
//
// Limitations:
//
// - Macro options are parsed in a basic fashion and must be in the order shown.
// - Testing has been against mixed Verilog+VHDL designs,
// Verilog-only names may be mangled.
// - No support yet for dynamic use in interactve simulation,
// this is primarily meant for batch-mode use.
//
// Plans:
//
// - Add support for sending commands direct to ncsim / simvision.
// - Add grouping to SimVision waveforms, based on unit tree.
// - Use default "ncsim" SHM if -db not given.
// - Allow multiple invocations to share the same SHM file.
<'
struct waveports_util {
!all_signals : list of string;
!user_tb_name : string;
dump_ports( db : string, dump_unit : any_unit = sys ) is {
var version : string = "0.1";
var nctcl : file;
var log : file;
var logName : string = append(db,".log");
var nctclName : string = append(db,".tcl");
nctcl = files.open(nctclName, "w", "Tcl script");
log = files.open(logName, "w", "Log file");
all_signals.clear();
files.write(nctcl, "# Auto-generated by sn_waveports utility");
files.write(nctcl, append("database -open ",db," -shm"));
files.write(log, append("# sn_waveports version ", version));
files.write(log, append("# db : ", db));
files.write(log, append("# unit : ", dump_unit.e_path()));
files.write(log, append("# tb : ", user_tb_name));
files.write(log, append("# tcl : ", nctclName));
for each (u) in rf_manager.get_all_unit_instances(dump_unit) {
if u is not a message_logger {
var subtype: rf_struct = rf_manager.get_exact_subtype_of_instance(u);
var signals : list of string;
for each (port_field) in subtype.get_fields().all(it.is_port_instance()) {
if port_field.get_type() is a rf_simple_port {
var port : any_port = port_field.get_value_unsafe(u).unsafe();
var full_hdl_path : string = port.full_hdl_path();
files.write(log,append("e_path : ", port.e_path()));
files.write(log,append("hdl_path : ", full_hdl_path));
files.write(log,append("connected : ", port.is_connected()));
files.write(log,append("agent : ", port.agent()));
full_hdl_path = str_replace(full_hdl_path, "/~\//", ""); // remove ~/ prefix
full_hdl_path = str_replace(full_hdl_path, "/[./]/", ":"); // replace . and / with :
full_hdl_path = str_replace(full_hdl_path, "/::+/", ":"); // remove multiple colons
// ignore ports which are not connected or are already probed
if port.is_connected() and not all_signals.has(it==full_hdl_path) {
all_signals.add(full_hdl_path);
if user_tb_name != NULL {
// search and replace explicit VHDL-TB name if given
full_hdl_path = str_replace(full_hdl_path, appendf("/^%s/",user_tb_name), "");
};
signals.add(full_hdl_path);
files.write(log,append("hdl_path' : ", full_hdl_path));
};
};
};
if 0 != signals.size() {
files.write(nctcl, appendf("# Unit %s\n", u.e_path()) );
files.write(nctcl, appendf("probe -create -database %s %s\n",
db, str_join(signals, " ") ) );
};
};
};
if 0 == all_signals.size() {
out(
"*** Error: No e ports were found",
" Try again after invoking 'gen' to construct the unit tree."
);
};
files.close(nctcl);
files.close(log);
out("Wrote ", nctclName, " and ", svcfName,".\nSourcing ",nctclName,"...\n");
simulator_command(append("source ",nctclName));
};
};
extend global {
waveports : waveports_util;
};
define <wave_ports'command> "wave_ports[ -db <db'file>][ -tb <tb'any>][ <du'struct_member>]" as computed
{
if NULL == global.waveports {
global.waveports = new;
};
if <tb'any> != NULL {
out("Setting '",<tb'any>,"' as the VHDL testbench name.");
global.waveports.user_tb_name = <tb'any>;
};
var db : string = "sn_waveports";
if <db'file> != NULL { db = <db'file>; };
var du : string = "sys";
if <du'struct_member> != NULL { du = <du'struct_member>; };
return appendf("global.waveports.dump_ports(\"%s\", %s);", db, du);
};
'>