Complex testbench design strategy

[Eli Bendersky]

Hello,

I'm now considering the design of a rather complex testbench, and want
to consult the dwellers of this list on the best strategy to adopt.

The testbench is for a complex multi-functional FPGA. This FPGA has
several almost unrelated functions, which it makes sense to check
separately. So I'm in doubt as to how to structure my testbench. As I
see it, there are two options:

1) One big testbench for everything, and some way to sequence the
tests of the various functions
2) Separate testbenches, one for each function

Each approach has its pros and cons. Option (2) sounds the most
logical, at least at first, but when I begin pursuing it, some
problems emerge. For example, too much duplication between testbenches
- in all of them I have to instantiate the large top-level entity of
the FPGA, write the compilation scripts for all the files, and so on.
In option (1) this would have to be done only once. On the other hand,
option (2) has the clear advantage over (1) of a clean separation of
tested functionality.

I'm sure this issue haunts every verification engineer at one time or
another. What approach do you use - 1, 2 or something else, and why ?

Thanks in advance

 

[Nicolas Matringe]

Eli Bendersky a écrit :

Hello,

I'm now considering the design of a rather complex testbench, and want
to consult the dwellers of this list on the best strategy to adopt.

The testbench is for a complex multi-functional FPGA. This FPGA has
several almost unrelated functions, which it makes sense to check
separately. So I'm in doubt as to how to structure my testbench. As I
see it, there are two options:

1) One big testbench for everything, and some way to sequence the
tests of the various functions
2) Separate testbenches, one for each function

Hello
I once had the same kind of task. I ended up with one big testbench with
a very long list of generic parameters and conditional component
instantiations.
Compilation is done once and for all, you just have to maintain the
simulation scripts that set the generic parameters for each partial
simulation.

Nicolas

 

[Jonathan Bromley]

On Sun, 16 Nov 2008 03:30:34 -0800 (PST), Eli Bendersky wrote:

Nice description of a problem that, as you say, is common to
almost any decent-sized verification project.

Almost any advice I could offer would be more-or-less an
echo of the sensible, practical suggestions in Janick
Bergeron's well-known "Writing Testbenches" book.
I think he misses a few tricks about the best way for
the upper levels of a TB to interact with its
bus-functional models (BFMs), but otherwise it's
pretty much on the money.

The first edition of that book covers both VHDL and Verilog.
The second edition adds extensive discussion of Vera and
'e'. Note that there is yet a third version of the book
which is SystemVerilog-focused; I'm guessing you don't
want that

The testbench is for a complex multi-functional FPGA. This FPGA has
several almost unrelated functions, which it makes sense to check
separately.

Yes, but most of the really "interesting" bugs are likely to
be related to the interactions between those functions...

1) One big testbench for everything, and some way to sequence the
tests of the various functions

I'd certainly go for that. Better still, build a "test harness" -
the structural guts of the TB, containing your DUT instance and
all its life-support systems such as clock and reset generators,
external component models, BFMs for testbench stimulus and
monitoring, and so on. Take very, very great care to give that
test harness the cleanest, simplest possible interface on its
ports. Ideally, there should be one BFM for each major
interface of the DUT, and each BFM should expose only two
record ports on the TB side - one "request" port and one
"response" port. You don't need handshakes at that level;
BFMs and TB can both use the 'transaction attribute on those
record signals to detect each new request or response.

The top level of this test harness is likely to be
horrible, with all the DUT-connection signals, DUT
instance and a pile of other big things instantiated too.
But you write it exactly once for the whole project.

You can now instantiate this test harness into a top-level
testbench containing only a few processes that sequence the
activity of the various BFMs. With luck, this top-level
TB will have only a few hundred lines of code and will be
fairly easy to modify to create alternative test sequences.
If you want to exercise your blocks one at a time, you can
easily write a top-level TB that leaves most of the BFMs
idle and merely exercises just one of them. Your top level
TB can easily randomize some of its behavior if you wish;
both the contents and the timing of the transactions (record
values) that you send to the BFMs can be adjusted by
creative use of ieee.math_real.uniform().

The devil is in the detail, of course, but something like
this works pretty well for me and it's what we recommend in
our advanced VHDL verification classes.

I think you can find examples of similar ideas in Jim Lewis's
training materials too.
--
Jonathan Bromley, Consultant

DOULOS - Developing Design Know-how
VHDL * Verilog * SystemC * e * Perl * Tcl/Tk * Project Services

Doulos Ltd., 22 Market Place, Ringwood, BH24 1AW, UK
jonathan.bromley@MYCOMPANY.com
http://www.MYCOMPANY.com

The contents of this message may contain personal views which
are not the views of Doulos Ltd., unless specifically stated.

 

On Nov 16, 7:50 am, Jonathan Bromley <jonathan.brom...@MYCOMPANY.com>
wrote:

On Sun, 16 Nov 2008 03:30:34 -0800 (PST), Eli Bendersky wrote:
I'm now considering the design of a rather complex testbench [...]

Nice description of a problem that, as you say, is common to
almost any decent-sized verification project.

Almost any advice I could offer would be more-or-less an
echo of the sensible, practical suggestions in Janick
Bergeron's well-known "Writing Testbenches" book.  
I think he misses a few tricks about the best way for
the upper levels of a TB to interact with its
bus-functional models (BFMs), but otherwise it's
pretty much on the money.

The first edition of that book covers both VHDL and Verilog.
The second edition adds extensive discussion of Vera and
'e'.  Note that there is yet a third version of the book
which is SystemVerilog-focused; I'm guessing you don't
want that

The testbench is for a complex multi-functional FPGA. This FPGA has
several almost unrelated functions, which it makes sense to check
separately.

Yes, but most of the really "interesting" bugs are likely to
be related to the interactions between those functions...

1) One big testbench for everything, and some way to sequence the
tests of the various functions

I'd certainly go for that.  Better still, build a "test harness" -
the structural guts of the TB, containing your DUT instance and
all its life-support systems such as clock and reset generators,
external component models, BFMs for testbench stimulus and
monitoring, and so on.  Take very, very great care to give that
test harness the cleanest, simplest possible interface on its
ports.  Ideally, there should be one BFM for each major
interface of the DUT, and each BFM should expose only two
record ports on the TB side - one "request" port and one
"response" port.  You don't need handshakes at that level;
BFMs and TB can both use the 'transaction attribute on those
record signals to detect each new request or response.  

The top level of this test harness is likely to be
horrible, with all the DUT-connection signals, DUT
instance and a pile of other big things instantiated too.
But you write it exactly once for the whole project.

You can now instantiate this test harness into a top-level
testbench containing only a few processes that sequence the
activity of the various BFMs.  With luck, this top-level
TB will have only a few hundred lines of code and will be
fairly easy to modify to create alternative test sequences.
If you want to exercise your blocks one at a time, you can
easily write a top-level TB that leaves most of the BFMs
idle and merely exercises just one of them.  Your top level
TB can easily randomize some of its behavior if you wish;
both the contents and the timing of the transactions (record
values) that you send to the BFMs can be adjusted by
creative use of ieee.math_real.uniform().

The devil is in the detail, of course, but something like
this works pretty well for me and it's what we recommend in
our advanced VHDL verification classes.

I think you can find examples of similar ideas in Jim Lewis's
training materials too.
--
Jonathan Bromley, Consultant

DOULOS - Developing Design Know-how
VHDL * Verilog * SystemC * e * Perl * Tcl/Tk * Project Services

Doulos Ltd., 22 Market Place, Ringwood, BH24 1AW, UK
jonathan.brom...@MYCOMPANY.comhttp://www.MYCOMPANY.com

The contents of this message may contain personal views which
are not the views of Doulos Ltd., unless specifically stated.

I also recommend a structure similar to what Jonathan describes above.
If you haven't already used the toggling transaction trick that
Jonathan mentions, I'd recommend getting a copy of Bergeron's book and
reading it through (at least the first parts and the VHDL parts). It's
well worth it. I can count on one hand those books that I sat down
with, read through from front to back, and found that they made the
light bulb go off, creating a lasting impression, and have guided me
in the years since. This was one of those books for me.

Other benefits of a single large test harness over a bunch of
individual testbenches for submodules is that it's also easy to plug
in the gate level code for final verification of the whole FPGA if you
like. I'm not sure if by "separate testbenches, one for each function"
you were planning to expose the top-level FPGA interface or just the
sub-function interface. It would be a shame to lose the ability to
verify the whole FPGA for want of a decent test harness.

One potential drawback of the mother-of-a-test-harness-monster-code,
though, can be efficiency. Lighting up all sorts of PLL/DCM and high-
speed SERDES components in simulation can be pretty slow if you get to
gate level sims, especially if you have to light up every single one
in a large FPGA just to test one of the ten subfunctions. At the RTL
level, some very creative use of abstract high level models for, say
SERDES, can speed up runtime drastically while still letting you
validate the other 99% of the logic inside the guts of the chip.

Good luck!

- Kenn

 

[Eli Bendersky]

I once had the same kind of task. I ended up with one big testbench with
a very long list of generic parameters and conditional component
instantiations.
Compilation is done once and for all, you just have to maintain the
simulation scripts that set the generic parameters for each partial
simulation.

Can you elaborate how you setup generics in runtime for partial
simulations ? I thought generics can only be set at compile/load time
and instantiations are static at runtime. This is my biggest problem
with the monolithic testbench approach.

Eli

 

[Eli Bendersky]

On Nov 16, 2:50 pm, Jonathan Bromley <jonathan.brom...@MYCOMPANY.com>
wrote:

On Sun, 16 Nov 2008 03:30:34 -0800 (PST), Eli Bendersky wrote:
I'm now considering the design of a rather complex testbench [...]

Nice description of a problem that, as you say, is common to
almost any decent-sized verification project.

Almost any advice I could offer would be more-or-less an
echo of the sensible, practical suggestions in Janick
Bergeron's well-known "Writing Testbenches" book.  
I think he misses a few tricks about the best way for
the upper levels of a TB to interact with its
bus-functional models (BFMs), but otherwise it's
pretty much on the money.

Thanks - I'll definitely order it. Any other good books you can
recommend on writing testbenches ?

<snip>

Yes, but most of the really "interesting" bugs are likely to
be related to the interactions between those functions...

Yes, this is why I planned each function-testbench to instantiate the
whole top-level DUT, so as not to lose those valuable internal
interfaces. When writing TBs, we always attempt to treat the DUTs as
black boxes, without getting inside. It really helps to test the whole
path from the top level of the FPGA to internal functions.

1) One big testbench for everything, and some way to sequence the
tests of the various functions

I'd certainly go for that.  Better still, build a "test harness" -
the structural guts of the TB, containing your DUT instance and
all its life-support systems such as clock and reset generators,
external component models, BFMs for testbench stimulus and
monitoring, and so on.  Take very, very great care to give that
test harness the cleanest, simplest possible interface on its
ports.  Ideally, there should be one BFM for each major
interface of the DUT, and each BFM should expose only two
record ports on the TB side - one "request" port and one
"response" port.  You don't need handshakes at that level;
BFMs and TB can both use the 'transaction attribute on those
record signals to detect each new request or response.  

The top level of this test harness is likely to be
horrible, with all the DUT-connection signals, DUT
instance and a pile of other big things instantiated too.
But you write it exactly once for the whole project.

You can now instantiate this test harness into a top-level
testbench containing only a few processes that sequence the
activity of the various BFMs.  With luck, this top-level
TB will have only a few hundred lines of code and will be
fairly easy to modify to create alternative test sequences.
If you want to exercise your blocks one at a time, you can
easily write a top-level TB that leaves most of the BFMs
idle and merely exercises just one of them.  Your top level
TB can easily randomize some of its behavior if you wish;
both the contents and the timing of the transactions (record
values) that you send to the BFMs can be adjusted by
creative use of ieee.math_real.uniform().

snip

A very interesting idea! While I mused about something similar, I
couldn't think it through to a full solution on my own. Let me see if
I get you straight. You propose to 'dress' my DUT in a harness
convenient for simulation, with simple transaction-like record ports
instead of complex signal interfaces. So far I understand.
But then, for each tested function, I suppose I should have a separate
TB that instantiates this harness, right ?

This makes it more similar to method (2) in my original message, with
some of the same problems. For instance, for each of the testbenches I
will have to compile all the DUT files separately, this giving some
duplication in compile scripts. However, much of the duplication of
instantiating the DUT with its 400 pins over and over again is saved,
because the harness presents a very simple interface to the outside -
one that can be only partially instantiated.

Did I understand you correctly ?

Thanks for the advice!
Eli

 

[Mike Treseler]

Eli Bendersky wrote:

I'm now considering the design of a rather complex testbench, and want
to consult the dwellers of this list on the best strategy to adopt.

Interesting question.

This FPGA has
several almost unrelated functions, which it makes sense to check
separately. So I'm in doubt as to how to structure my testbench. As I
see it, there are two options:

1) One big testbench for everything, and some way to sequence the
tests of the various functions

I certainly have to be able to wiggle all the inputs
and watch all the outputs, so I don't see any way
to get around this one. And as long as I have
to do this anyway, it is probably a good idea
to make top testbench drill as deep as my schedule allows.

2) Separate testbenches, one for each function

I don't have a final answer for this one.
Design time often alternates between top down
and bottom up, and an entity/module is the
only synthesizable object that a testbench can test.
I design some modules, I reuse or buy others.
One module may be a simple synchronizer.
Another might be an entire front end.

I'm sure this issue haunts every verification engineer at one time or
another. What approach do you use - 1, 2 or something else, and why ?

In the something else category, i might add

1. Assertion testing of packaged functions.
If I can extract a vhdl function from a complex expression,
I can verify this much out of time by testing the package.
I also have the option on reusing these pretested functions.
I might even have some useful assertions to make about constants.

2. Avoid text files, custom languages, BFLs etc.
VHDL has functions and procedures that cover this ground.
VHDL does run out of gas at the higher level verification
abstractions, but I prefer to make due with vanilla vhdl.

3. I like the discussion in these papers.
http://enpub.fulton.asu.edu/cse517/bfm.pdf
http://www.syncad.com/paper_coding_techniques_icu_sep_2003.htm

-- Mike Treseler

 

[Jonathan Bromley]

On Sun, 16 Nov 2008 08:19:00 -0800 (PST), Eli Bendersky wrote:

[...]

Bergeron's well-known "Writing Testbenches" book.  

Thanks - I'll definitely order it. Any other good books you can
recommend on writing testbenches ?

Sadly, there's nothing I know of that really hits the
spot in quite the same way. Of course, there surely
are other good books I'm unaware of.
[...]

[test harness instantiated into a behavioral testbench]

A very interesting idea! While I mused about something similar, I
couldn't think it through to a full solution on my own. Let me see if
I get you straight. You propose to 'dress' my DUT in a harness
convenient for simulation, with simple transaction-like record ports
instead of complex signal interfaces. So far I understand.
But then, for each tested function, I suppose I should have a separate
TB that instantiates this harness, right ?

Yes; or other broadly similar possibilities. For example,
you could make a somewhat general-purpose testbench entity
with ports matching (complementary to) the harness's ports,
and instantiate both the TB and the harness in a separate
top level that simply hooks them together. Then you can
create as many different architectures for the TB module
as you wish, and use VHDL configurations to decide which
to load; the enclosing top-level remains invariant across
all tests. Functionally equivalent, but quite possibly
rather easier to maintain, than the "harness inside TB"
architecture.

Whichever top-level structure you adopt, the nifty thing
about it is that you can push transaction record objects
to and from the test harness in zero time, which means that
it is often possible to stimulate multiple interfaces on the
DUT from a single process in the top-level TB. That makes
it much, much easier to coordinate stimulus across multiple
interfaces, giving you a better chance to mimic the action of
software more accurately and to explore some of the
interesting interactions more fully. Expect to build a rich
set of procedures to manage this sort of thing in your TB;
put them in a package so that they can easily be re-used
by many different versions of the TB.

This makes it more similar to method (2) in my original message, with
some of the same problems. For instance, for each of the testbenches I
will have to compile all the DUT files separately,

Yes and no; you could compile the DUT, test harness and
supporting models into a library, then instantiate the
ready-compiled harness in each of several TBs that you
compile independently. Or you could write a test-harness
compilation script, and invoke it from your TB compile script
using the Tcl "source" command.

However, much of the duplication of
instantiating the DUT with its 400 pins over and over again is saved,
because the harness presents a very simple interface to the outside -

Absolutely. Hide all that dreary once-per-project kruft behind
your elegantly engineered transaction-passing interface.

one that can be only partially instantiated.

I'm not quite sure what you mean by this - the test harness
can be a perfectly ordinary entity, instantiated into the
testbench in the ordinary way. uh...second thoughts... do you
mean that you can leave many of its ports disconnected? Yes,
I guess that's so, since the values on those ports will be
ignored by the TH functional blocks unless they see 'transaction.
You would need appropriate defaults on all the input ports,
of course.

Did I understand you correctly ?

It certainly looks like it. And, as I said, the exact details
will surely be somewhat up to you. Verilog provides the
marvellous facility of procedural call into the test harness
module, but you can't do that in VHDL and the record ports
and 'transaction trick is the only vanilla-VHDL way I know
to do it.

While I'm rambling about the test harness: don't forget to
add modules within the test harness to check for good behaviour
on all the DUT's major interfaces, especially the ones that
are supposed to conform to standardized protocols. Minor
misbehaviour on interfaces is often overlooked when eyeballing
waveforms, but is a valuable early-warning sign for more
serious errors deep in the DUT as well as being a good sanity
check on the correctness of your bus functional models.
I guess you knew that already and, of course, if you are
using PSL or OVL assertions to do that, then so much the better.
--
Jonathan Bromley, Consultant

DOULOS - Developing Design Know-how
VHDL * Verilog * SystemC * e * Perl * Tcl/Tk * Project Services

Doulos Ltd., 22 Market Place, Ringwood, BH24 1AW, UK
jonathan.bromley@MYCOMPANY.com
http://www.MYCOMPANY.com

The contents of this message may contain personal views which
are not the views of Doulos Ltd., unless specifically stated.

 

[Mark McDougall]

Eli Bendersky wrote:

1) One big testbench for everything, and some way to sequence the
tests of the various functions

In my experience, that's the way to go. You always end up with less
duplication, and it's easier to maintain changes which might otherwise
require synchronisation between the various separate testbenches.

Another mechanism I employed - for both synthesis and simulation - is to
define a set of constants in a global package, and assign them in the
package body - that enable/disable various (unrelated) sections of the
design (using if-generate for example). You can then selectively build
various combinations of the design to simplify/speed up simulation and/or
synthesis.

In this way you could, for example, use the exact same testbench framework
but have several simulation project files with unique "global package
bodies" that build different parts of the design for each
functionally-unrelated testbench script.

Regards,

--
Mark McDougall, Engineer
Virtual Logic Pty Ltd, <http://www.vl.com.au>
21-25 King St, Rockdale, 2216
Ph: +612-9599-3255 Fax: +612-9599-3266

 

[Aiken]

On Nov 16, 12:46 pm, Jonathan Bromley <jonathan.brom...@MYCOMPANY.com>
wrote:

On Sun, 16 Nov 2008 08:19:00 -0800 (PST), Eli Bendersky wrote:

[...]

Bergeron's well-known "Writing Testbenches" book.  
Thanks - I'll definitely order it. Any other good books you can
recommend on writing testbenches ?

Sadly, there's nothing I know of that really hits the
spot in quite the same way.  Of course, there surely
are other good books I'm unaware of.
[...]

[test harness instantiated into a behavioral testbench]

A very interesting idea! While I mused about something similar, I
couldn't think it through to a full solution on my own. Let me see if
I get you straight. You propose to 'dress' my DUT in a harness
convenient for simulation, with simple transaction-like record ports
instead of complex signal interfaces. So far I understand.
But then, for each tested function, I suppose I should have a separate
TB that instantiates this harness, right ?

Yes; or other broadly similar possibilities.  For example,
you could make a somewhat general-purpose testbench entity
with ports matching (complementary to) the harness's ports,
and instantiate both the TB and the harness in a separate
top level that simply hooks them together.  Then you can
create as many different architectures for the TB module
as you wish, and use VHDL configurations to decide which
to load; the enclosing top-level remains invariant across
all tests.  Functionally equivalent, but quite possibly
rather easier to maintain, than the "harness inside TB"
architecture.

I am a new guy in FPGA design.
How can I use VHDL configurations to decide which one load into
simulation?

 

[Eli Bendersky]

On Nov 16, 7:46 pm, Jonathan Bromley <jonathan.brom...@MYCOMPANY.com>
wrote:

On Sun, 16 Nov 2008 08:19:00 -0800 (PST), Eli Bendersky wrote:

[...]

Bergeron's well-known "Writing Testbenches" book.  
Thanks - I'll definitely order it. Any other good books you can
recommend on writing testbenches ?

Sadly, there's nothing I know of that really hits the
spot in quite the same way.  Of course, there surely
are other good books I'm unaware of.
[...]

[test harness instantiated into a behavioral testbench]

snip

Thanks Jonathan, you've been very helpful. I'll try the harness
approach and see how it goes. In the meantime I'll order the book and
see what else I can glean from it about TB design.

Eli

 

[Eli Bendersky]

3. I like the discussion in these papers.http://enpub.fulton.asu.edu/cse517/bfm.pdfhttp://www.syncad.com/paper_coding_techniques_icu_sep_2003.htm

I'll look at these, thanks.

 

[Eli Bendersky]

On Nov 17, 2:42 am, Mark McDougall <ma...@vl.com.au> wrote:

Eli Bendersky wrote:
1) One big testbench for everything, and some way to sequence the
tests of the various functions

In my experience, that's the way to go. You always end up with less
duplication, and it's easier to maintain changes which might otherwise
require synchronisation between the various separate testbenches.

Another mechanism I employed - for both synthesis and simulation - is to
define a set of constants in a global package, and assign them in the
package body - that enable/disable various (unrelated) sections of the
design (using if-generate for example). You can then selectively build
various combinations of the design to simplify/speed up simulation and/or
synthesis.

In this way you could, for example, use the exact same testbench framework
but have several simulation project files with unique "global package
bodies" that build different parts of the design for each
functionally-unrelated testbench script.

Sounds like an interesting approach that can save simulation/synthesis
time. One problem I see with it is that the DUT is not always under
our control. As far as I'm concerned, it's sometimes just a black box
I can assume nothing about except its spec.

Eli

 

[Martin Thompson]

kennheinrich@sympatico.ca writes:

One potential drawback of the mother-of-a-test-harness-monster-code,
though, can be efficiency. Lighting up all sorts of PLL/DCM and high-
speed SERDES components in simulation can be pretty slow if you get to
gate level sims, especially if you have to light up every single one
in a large FPGA just to test one of the ten subfunctions. At the RTL
level, some very creative use of abstract high level models for, say
SERDES, can speed up runtime drastically while still letting you
validate the other 99% of the logic inside the guts of the chip.

Indeed.

I have several "layers" of TB, so I can run a restricted set of
regression tests on (say) the algorithmic part of my design, which
presents a record for its configuration to the encompassing element.
I can then sidestep the tedious setup of all the registers (which is
through SPI) and just concentrate on validating the algorithm bit.

The "right-answers" for the algorithm testbench are generated from the
Matlab environment in which the algorithm is developed. So new
algorithmic regression tests are very easily added as new situations
are discovered in "algorithm-land".

The SPI regs have their own TB to check their operation in isolation.

Large lumps of the algorithm TB are then instantiated in the higher level
testbenches, which checks that I've wired them up right etc. This can
be recycled for the post-synth and post-PAR sims which occasionally
(very occasionally in the case of the post-PAR, for power anaylsis data!)

At the top I have a python script which finds all my regression tests
and feeds them to vsim to make sure they all pass. I run this
periodically, as well as before release, but most of the time I am
working down the hierarchy, so my TBs concentrate on testing just the
lower functionality.

Cheers,
Martin

--
martin.j.thompson@trw.com
TRW Conekt - Consultancy in Engineering, Knowledge and Technology
http://www.conekt.net/electronics.html

 

[KJ]

On Nov 16, 6:30 am, Eli Bendersky <eli...@gmail.com> wrote:

Hello,

The testbench is for a complex multi-functional FPGA. This FPGA has
several almost unrelated functions, which it makes sense to check
separately. So I'm in doubt as to how to structure my testbench. As I
see it, there are two options:

1) One big testbench for everything, and some way to sequence the
tests of the various functions
2) Separate testbenches, one for each function

Each approach has its pros and cons. Option (2) sounds the most
logical, at least at first, but when I begin pursuing it, some
problems emerge. For example, too much duplication between testbenches
- in all of them I have to instantiate the large top-level entity of
the FPGA, write the compilation scripts for all the files, and so on.

I would have a separate testbench for each of these individual
functions, however they would not be instantiating the top level FPGA
at all. Each function presumably has it's own 'top' level that
implements that function, the testbench for that function should be
putting that entity through the wringer making sure that it works
properly. Having a single instantiation of the top level of the
entire design to test some individual sub-function tends to
dramatically slow down simulation which then produces the following
rather severe consequences:

- Given an aribtrary period of wall clock time, less testing of a
particular function can be performed.
- Less testing implies less coverage of oddball conditions.
- When you have to make changes to the function to fix a problem that
didn't happen to show up until the higher level testing was performed,
regression testing will also be hampered by the above two points when
trying to verify that in fixing one problem you didn't create another.

As a basic rule, a testbench should be testing new design content that
occurs roughly at that level not design content that is buried way
down in the design. At the top level of the FPGA design, the only new
design content is the interconnect to the top level functions and the
instantiation of all of the proper components.

Also, I wouldn't necessarily stop at the FPGA top level either. The
FPGA exists on a PCBA with interconnect to other parts (all of which
can be modelled), and the PCBA exists in some final system that simply
needs power and some user input/output (which again can be modelled).
All of this may seem to be somewhat of a side issue but it's not.
Take for example, the design of a DDR controller as a sample design.
That controller should have it's own testbench that rather extensively
tests all of the operational modes. At the top level of the FPGA
though you would simply instantiate it. Exhaustively testing again at
that level would be pretty much wasted time. A better test would be
to model the PCBA (which would instantiate a memory model from the
vendor) and walk a one across the address and data bus since that
effectively verifies the interconnect which at the FPGA top level and
the PCBA level is all the new design content that exists in regards to
the DDR controller.

As another example, let's say your system processes images and
produces JPEG output and you're writing all the code yourself. You
would probably want some extensive low level testing of some of the
basic low level sub-functions like...
- 1d DCT transform
- 2d DCT transform
- Huffman encoding
- Quantizer

Another level of the design would tie these pieces (and all of the
other pieces necessary to make a full JPEG encoder) and test that the
whole thing compresses images properly. At that level, you'd probably
also be extensively varying the image input, the Q tables, the Huffman
tables, the flow control in and out and run lots of images through to
convince yourself that everything is working properly. But you would
be wasting time (and wouldn't really be able to) vary all of the
parameters to those lower level DCT functions since they would likely
be parameterized for things like input/output data width and size but
in the JPEG encoder it doesn't matter if there is a bug that could
only affect an 11x11 element 2d DCT since you're using it in an 8x8
environment. That lower level testbench is the only thing that could
uncover the bug in the 11x11 case but if you limit yourself to
testbenches that can only operate at some higher level you will be
completely blind and think that your DCT is just fine until you try to
use it with some customer who wants some 11x11 DCT for whatever
reason.

Similarly, re-testing those same things that you do vary at the 'image
compression' level at the next higher level would then be mostly
wasted time that could've been better spent somewhere else. Often
times integration does produce conditions that were not considered at
the lower functional testbench level but to catch that what you need
then is to have something that predicts the correct response given the
testbench input and assert on any differences.

Testing needs to occur on various levels, not one (even if that one
level is able to effectively disable all the other functions but the
one is currently interested in). Ideally this testing occurs at every
level where significant new design content is being created. At the
level where 'insignificant' (but still important) new content is
created (like the FPGA top level which simply instantiates and
interconnects), you can generally get more bang for the buck by going
up yet another level (to the PCBA).

Kevin Jennings

 

[Eli Bendersky]

On Nov 17, 4:44 pm, KJ <kkjenni...@sbcglobal.net> wrote:

On Nov 16, 6:30 am, Eli Bendersky <eli...@gmail.com> wrote:



Hello,

The testbench is for a complex multi-functional FPGA. This FPGA has
several almost unrelated functions, which it makes sense to check
separately. So I'm in doubt as to how to structure my testbench. As I
see it, there are two options:

1) One big testbench for everything, and some way to sequence the
tests of the various functions
2) Separate testbenches, one for each function

Each approach has its pros and cons. Option (2) sounds the most
logical, at least at first, but when I begin pursuing it, some
problems emerge. For example, too much duplication between testbenches
- in all of them I have to instantiate the large top-level entity of
the FPGA, write the compilation scripts for all the files, and so on.

I would have a separate testbench for each of these individual
functions, however they would not be instantiating the top level FPGA
at all.  Each function presumably has it's own 'top' level that
implements that function, the testbench for that function should be
putting that entity through the wringer making sure that it works
properly.  Having a single instantiation of the top level of the
entire design to test some individual sub-function tends to
dramatically slow down simulation which then produces the following
rather severe consequences:

- Given an aribtrary period of wall clock time, less testing of a
particular function can be performed.
- Less testing implies less coverage of oddball conditions.
- When you have to make changes to the function to fix a problem that
didn't happen to show up until the higher level testing was performed,
regression testing will also be hampered by the above two points when
trying to verify that in fixing one problem you didn't create another.

As a basic rule, a testbench should be testing new design content that
occurs roughly at that level not design content that is buried way
down in the design.  At the top level of the FPGA design, the only new
design content is the interconnect to the top level functions and the
instantiation of all of the proper components.

While I agree that this approach is the most suitable, and your JPEG
example is right to the point, it must be stressed that there are some
problems with it.

First of all, the "details devil" is sometimes in the interface
between modules and not inside the modules themselves. Besides, you
can't always know where a module's boundary ends, especially if it's
buried deep in the design.

Second, the guys doing the verification are often not the ones who
wrote the synthesizable code, and hence have little idea of the
modules inside and their interfaces.

Third, the modules inside can change, or their interfaces can change
due to redesign or refactoring, so the testbench is likely to change
more often. On the top level, system requirements are usually more
static.

Eli

 

[KJ]

On Nov 17, 11:45 am, Eli Bendersky <eli...@gmail.com> wrote:

On Nov 17, 4:44 pm, KJ <kkjenni...@sbcglobal.net> wrote:

First of all, the "details devil" is sometimes in the interface
between modules and not inside the modules themselves.

I disagree, the only 'between' modules construct is on the port map to
the modules. The only types of errors that can occur here are:
- Forgot to connect something
- Connected multiple outputs together
- Connect signal 'xyz' to port 'abc' but it should've been connected
to port 'def'

While all of the above three do occur, they are easily found in and
fixed early on. Almost any testbench and/or synthesis result will
flag it.

If instead by "details devils" 'between' modules you mean connecting
incompatible interfaces that you thought were compatible (for example,
port 'abc' is an active high output but it is used by some other
module as an active low input) then I would suggest that you're not
viewing the interface connection logic in the proper light. That
connection logic is itself it's own module and logic set. While the
logic inversion example is of course trivial and likely not a good
candidate for it's own entity/architecture, a more complicated example
requiring more logic (say an 8 bit to 32 bit conversion as a slightly
more involved example) is more ammo that if there is any interface
conversion logic required it should be seen as it's own design and
tested as such before integration into the bigger design.

While I wouldn't bother with a testbench (or even a stand alone
design) for an interface conversion that consisted of simply a logic
inversion, I would for something like a bus size converter.

Besides, you
can't always know where a module's boundary ends, especially if it's
buried deep in the design.

Nor does it matter where a module boundary ends. Either a design
performs the required function given a particular stimulus or it does
not. When it doesn't, the most likely design change required is
something that changes the logic but finding just what line of code
needs changing requires debug regardless of methodology. Except for
the port mapping errors mentioned above which are almost always fixed
really early on, any other changes would be in 'some' module's rtl
code.

Second, the guys doing the verification are often not the ones who
wrote the synthesizable code, and hence have little idea of the
modules inside and their interfaces.

The line between modules that are to be tested by some separate
verification guys and modules that are not, is an arbitrary line that
you yourself define. Part of what goes into the decision about where
you draw the line between which interfaces get rigorous testing and
which do not is time (=money). The higher you go up before testing,
the less control you get in testing and verification.

Also, it is not unreasonable for the verification guys to be adding
non-synthesizable test/verification code right into the design itself.

Third, the modules inside can change, or their interfaces can change
due to redesign or refactoring, so the testbench is likely to change
more often.

The testbenches would be similarly refactored. Assuming a working
design/testbench pair before refactoring then there should also be a
working set of design/testbench pairs after. A designer may very well
create a small module that can be tested at some higher level without
creating undo risk to the overall design but that should tend to be
more of the exception than the rule.

I understand the likely concern regarding cost/effort in maintaining
and migrating testbenches along with the designs but moving all
testing and verification up to the top level because the interfaces up
there don't change much as a methodology is likely to severly limit
the quality of the design verification. It will also tend to inhibit
your ability to adequately regression test the effects of design
changes that address problems that have been found to verify that
something else didn't break.

Those lower level testbenches are somewhat analogous to incoming
receiving/inspection at a manufacturing plant. No manufacturer would
start building product without performing their own inspection and
verification on incoming parts to see that they are within spec before
building them into their product. Similarly, an adequate testbench(s)
should exist for modules before they are integrated into a top level
design. If those testbences are not adequate, beef them up. If they
are then there is not too much point in going hog wild testing the
same things at a higher level.

Kevin Jennings

 

[Mike Treseler]

KJ wrote:

The line between modules that are to be tested by some separate
verification guys and modules that are not, is an arbitrary line that
you yourself define.

Indeed. There are significant FPGA based projects
where the total number of design and verification
engineers is one. Collecting and maintaining all module tests for
automated regression can be an efficient and effective strategy.

Also, it is not unreasonable for the verification guys to be adding
non-synthesizable test/verification code right into the design itself.

I agree. If source code is untouchable, there is a problem
with the version control process.

Those lower level testbenches are somewhat analogous to incoming
receiving/inspection at a manufacturing plant. No manufacturer would
start building product without performing their own inspection and
verification on incoming parts to see that they are within spec before
building them into their product. Similarly, an adequate testbench(s)
should exist for modules before they are integrated into a top level
design. If those testbences are not adequate, beef them up. If they
are then there is not too much point in going hog wild testing the
same things at a higher level.

Good point.
I wouldn't use anyone's IP core without
a testbench, including my own.

-- Mike Treseler

 

[Mike Treseler]

Andy wrote:

I agree that it is a good thing to be able to test submodules
individually. However, in some designs the "system" can be greatly
simplified, and the same system level interface (re)used to test the
individual modules, which can reduce the effort required to create the
module level test benches in the first place.

I wasn't going to rock the BFM boat, but I also use generics
and along with shell scripts to to make a regression.
I list command options at the top of the testbench
for interactive use something like this:

-- force an error 0:
-- vsim -Gwire_g=stuck_hi -c test_uart -do "run -all; exit"
-- change template:
-- vsim -Gtemplate_g=s_rst -c test_uart -do "run -all; exit"
-- slow baud:
-- vsim -Gtb_tics_g=42 -c test_uart -do "run -all; exit"
-- verify strobe calibration:
-- vsim -Gtb_tics_g=42 test_uart
-- then "do uart.do"

... and later code up a script to play with coverage.

-- Mike Treseler

 

[Andy]

On Nov 17, 2:41 pm, Mike Treseler <mtrese...@gmail.com> wrote:

KJ wrote:
The line between modules that are to be tested by some separate
verification guys and modules that are not, is an arbitrary line that
you yourself define.  

Indeed.  There are significant FPGA based projects
where the total number of design and verification
engineers is one. Collecting and maintaining all module tests for
automated regression can be an efficient and effective strategy.

Also, it is not unreasonable for the verification guys to be adding
non-synthesizable test/verification code right into the design itself.

I agree. If source code is untouchable, there is a problem
with the version control process.

Those lower level testbenches are somewhat analogous to incoming
receiving/inspection at a manufacturing plant.  No manufacturer would
start building product without performing their own inspection and
verification on incoming parts to see that they are within spec before
building them into their product.  Similarly, an adequate testbench(s)
should exist for modules before they are integrated into a top level
design.  If those testbences are not adequate, beef them up.  If they
are then there is not too much point in going hog wild testing the
same things at a higher level.

Good point.
I wouldn't use anyone's IP core without
a testbench, including my own.

       -- Mike Treseler

I agree that it is a good thing to be able to test submodules
individually. However, in some designs the "system" can be greatly
simplified, and the same system level interface (re)used to test the
individual modules, which can reduce the effort required to create the
module level test benches in the first place.

If the RTL includes generics that can "disable" individual modules,
and those generics are passed up to the top of the DUT (defaulted to
enable everything), then a testbench can instantiate the DUT with
generics that disable all but the module under test, yet re-use the
same system level interface routines to do so, while avoiding the wall-
clock time penalties of simulating a far more complex system than what
you are interested in.

Of course, every project is different, and some include too much
overhead to incorporate the system level interface to make this
practical.

Andy