What are the limitations of cStringIO.StringIO() ?

Discussion:

(too old to reply)

Benjamin Kaplan

2009-07-02 00:27:33 UTC

Hi,
I think I'm up against a limitation in cStringIO.StringIO(), but could not
find any clues on the web, especially not in
http://docs.python.org/library/stringio.html.
        import types
        f = cStringIO.StringIO()
            log_stream = LogStream(filename)
                    string_ = log_stream.input_file.read()
                        sys.stderr.write("AttributeError: "+str(e)+"\n")
"+str(log_stream)+"\n")
                return(None)
            f.write(string_)
        return (f)
And, when the list of files - in filename_array - is pointing to long/large
File "svm_ts_tool_in_progress.py", line 244, in OnSelectCell
    self.InspectorViewController(row,col)
File "svm_ts_tool_in_progress.py", line 1216, in InspectorViewController
    self.InspectorePaneGetRecords(self.req_table, rec)
File "svm_ts_tool_in_progress.py", line 1315, in InspectorePaneGetRecords
    log_stream =
ConcatenatedLogStream(self.events_dict[req_table]["db_dict"].keys())
File "c:\Documents and
Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 31, in
__init__
    self.input_file = self.concatenate_files(filename_array)
File "c:\Documents and
Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 47, in
concatenate_files
    string_ = log_stream.input_file.read()
MemoryError
Anyone knows whet's the limitation on cStringIO.StringIO() objects ?
Could you suggest a better way to create a string that contains the
concatenation of several big files ?

MemoryErrors are caused because, for whatever reason, the OS won't
allocate any more memory for the process. For a 32-bit Windows
process, that maximum is 2 GB. No matter what programming language or
library you use, you can't bypass that limit. There is a way to up the
limit to 3GB but I don't know how to do it. The link below has all the
information about Windows memory limits. If you need lots of really
big strings in memory simultaneously, you'll have to get a 64-bit
system with a ton of RAM.

http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx

Thanks,
Ron.
--
http://mail.python.org/mailman/listinfo/python-list

ryles

2009-07-03 02:23:54 UTC

Permalink

This reminds me. Would anyone object to adding a sentence to
http://docs.python.org/library/stringio.html#module-cStringIO just to
mention that attributes cannot be added to a cStringIO, like such:

% import cStringIO, StringIO
% StringIO.StringIO().value = 1
% cStringIO.StringIO().value = 1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'cStringIO.StringO' object has no attribute 'value'

Benjamin Peterson

2009-07-03 02:54:31 UTC

Permalink

Post by ryles
This reminds me. Would anyone object to adding a sentence to
http://docs.python.org/library/stringio.html#module-cStringIO just to

That's true of almost all types that are implemented in C.

Simon Forman

2009-07-03 02:52:10 UTC

Permalink

MemoryError means you're out of memory. Try using the operating
system commands for file concatenation, i.e. "cat" on *nix, dunno how
on windows. Or just do whatever it is you're doing in a different way
that doesn't require having all the data in memory at once.

Also, this:

not isinstance(log_stream, types.NoneType)

should just be this:

log_stream is not None

and return is not a function, leave out the ()'s around the expression
you're returning, and if you're returning None, you can leave out the
None as it is the default return value anyway.

Generally speaking don't do module imports in functions.