How to load efficiently load a file into a variable (memory)

[jmss]

Hi.

How can I rean an entire large file into memory in an efficient way?
The file contents are lists (of lists (of lists...))

I tried 'load' but it gives a value stack overflow. The file around 8
MB.

Thanks and best regards,
Joăo M. S. Silva

 

[Andrew Beckett]

On Wed, 26 Sep 2007 08:12:59 -0700, jmss <joao.m.santos.silva@gmail.com> wrote:

Hi.

How can I rean an entire large file into memory in an efficient way?
The file contents are lists (of lists (of lists...))

I tried 'load' but it gives a value stack overflow. The file around 8
MB.

Thanks and best regards,
Joăo M. S. Silva

I just read a file which contained a list (of lists of lists of lists) which was
somewhat bigger than 8Mbyte ) by using lineread() to read the data in the file.

That said, by having an assignment in the file, I could also load it
using load() - so not sure what your problem is.

Perhaps you should report it to Cadence customer support?

Regards,

Andrew.

--
Andrew Beckett
Senior Solution Architect
Cadence Design Systems, UK.

 

[Nicolas]

You may also try parsing the file. In other words, strip out the
skill "list()" calls around your data and replace with something like
tabs. Don't load the file, instead, open() it, and read it line by
line. I believe SKILL limits the size of a skill file it can load by
the number of lines or size, can't recall the exact limitation. If
you parse and build your lists manually, you can circumvent that
mechanism and only be left to deal with the memory size available for
your lists.

Nicolas

 

[jmss]

Hi,

I'm managing to solve the problem by changing the format of the file
to be loaded to one list per line, so that I can do:

(while (gets line port)
buffer = (tconc buffer (readstring line))
)
buffer = car(buffer)

The problem is that I have lines larger than the limit of gets(). How
can I change the gets() macro to accept larger string sizes?

Thanks.

 

[Thomas F. Burdick]

On Sep 26, 5:12 pm, jmss <joao.m.santos.si...@gmail.com> wrote:

Hi.

How can I rean an entire large file into memory in an efficient way?
The file contents are lists (of lists (of lists...))

I tried 'load' but it gives a value stack overflow. The file around 8
MB.

Thanks and best regards,
Joăo M. S. Silva

A value stack overflow generally means that you passed too many
arguments to a function. I'm guessing this means that your file
looked like:

SomeGlobalVar = list(1 2 3 4 ... 800000)

Rather than construct a new list from static data, you could just dump
the literal list to a file and read but not evaluate that expression.
So you'd have a file called something like "foo.sexp" which contains a
single structured expression, which you would "load" in without using
the function load:

in = infile("foo.sexp")
SomeGlobalVar = (car (lineread in))
close(in)

Now SomeGlobalVar should contain your very large list without hitting
any object or stack size limitations.