slow execution - skill profiler: a lot time spent on gc() c

[Marcel Preda]

Hi,

I have a problem with some script which is traversing a schematic
hierarchy.
The problem is that when I run it for the first time (just opent the
schematic top),
it takes from 7000 to 8000 seconds.

If I run it aging (to traverse hierachy), on the same schematic it
takes only 1500 ~ 2000 seconds.

I have run the profiler and I saw that ~5000 seconds are spent only
inside of garbage collector "gc()" calls.
It's true that the allocated memory is very high, it goes up to 3GB .
There is a huge number of lists build inside of the script.

Is there a way to tell to SKILL interpreter: pls, don not call (very
often) the gc() ?

Best Regards,
Marcel

 

[Andrew Beckett]

Marcel Preda wrote, on 05/27/09 08:52:

Hi,

I have a problem with some script which is traversing a schematic
hierarchy.
The problem is that when I run it for the first time (just opent the
schematic top),
it takes from 7000 to 8000 seconds.

If I run it aging (to traverse hierachy), on the same schematic it
takes only 1500 ~ 2000 seconds.

I have run the profiler and I saw that ~5000 seconds are spent only
inside of garbage collector "gc()" calls.
It's true that the allocated memory is very high, it goes up to 3GB .
There is a huge number of lists build inside of the script.

Is there a way to tell to SKILL interpreter: pls, don not call (very
often) the gc() ?

Best Regards,
Marcel

The best way is to pre-allocate memory using needNCells() . You need to know
what type of cells are needed - gcsummary() can tell you that.

Essentially each time it runs out of memory, it does a gc() first before
allocating more memory in order to try to use the garbage first.

You can use the variable gcdisable (this is documented) but this should be used
with EXTREME caution, because if it gets accidentally left off it can completely
mess up your session (memory will never get recovered).

Also, think carefully about your list creation - does it really need to create
as many lists - is it creating lists and then throwing them away?

Regards,

Andrew.

 

[Marcel Preda]

On May 27, 12:54 pm, Andrew Beckett <andr...@DcEaLdEeTnEcTe.HcIoSm>
wrote:

Marcel Preda wrote, on 05/27/09 08:52:



Hi,

I have a problem with some script which is traversing a schematic
hierarchy.
The problem is that when I run it for the first time (just opent the
schematic top),
it takes from 7000 to 8000 seconds.

If I run it aging (to traverse hierachy), on the same schematic it
takes only 1500 ~ 2000 seconds.

I have run the profiler and I saw that ~5000 seconds are spent only
inside of garbage collector "gc()" calls.
It's true that the allocated memory is very high, it goes up to 3GB .
There is a huge number of lists build inside of the script.

Is there a way to tell to SKILL interpreter: pls, don not call (very
often) the gc() ?

Best Regards,
Marcel

The best way is to pre-allocate memory using needNCells() . You need to know
what type of cells are needed - gcsummary() can tell you that.

Essentially each time it runs out of memory, it does a gc() first before
allocating more memory in order to try to use the garbage first.

You can use the variable gcdisable (this is documented) but this should be used
with EXTREME caution, because if it gets accidentally left off it can completely
mess up your session (memory will never get recovered).

Also, think carefully about your list creation - does it really need to create
as many lists - is it creating lists and then throwing them away?

Regards,

Andrew.

Hi Andrew,

thank you for the reply.
For sure I have some lists created, used and after that I throw them
away or I remove some elements from the lists.

In the meantime I've found what could be the real problem:
I netlist the entire hierarchy: instances plus some properties of the
instances..
Because I want the instances to be sorted (+ other reasons) I keep all
of them into a hash.
Each key is an inst name, and each value of the hash is also a hash
(with 2 ~ 5 keys - depends by the inst/cell type).
On this 2nd hash each key is a property, and the keys are the prop
values .

Beacus ethis design is huge ~400 000 insts, I'll have:
- a hughe hash, with about 400 000 keys
- about 400 000 hashes with 2 ~ 5 keys
- plus other data..

Looks like the hash structure was not the best idea,
until now the number of instances was not so big,
so the code was fast enough.

For now I've changed a little the data structure, the 2nd hash
(prop=>value) has become a string.
And right now I've started the script, I have to wait to see the
results.
If the time will not decrease as much as I want,
next thing that I'll do is to dump the instances unsorted into a
temporary file.
And to call an external script to do the sort of the instances.
In this way I do not need to keep in memory the huge hash,
which is eating more than 1GB.


BR,
Marcel