[Twisted-Python] re: Congrats / a little newbie confusion

[Bob Ippolito]

On Friday, Apr 4, 2003, at 14:08 America/New_York, Peter Hansen wrote:

> Jeffrey Mathews wrote:
>>
>> As for Peter's question about the need for a CGI.FieldStorage type 
>> api, it's
>> really only an issue when you're doing huge multipart mime posts 
>> (e.g., file
>> uploads).  The idea of passing around those results (potentially multi
>> megabyte) as strings is kind of frightening.  That said, I have no 
>> grasp on
>> what issues are raised when handling these things asynchronously (the 
>> cgi
>> module is all synchronous).
>
> What is it about passing around multi-megabyte data as strings that 
> scares
> you?  The data is only needed temporarily, I would think, until it is 
> written
> to a file.  Servers have gobs of memory, and massive file uploads are
> rare... or if they're not, surely the server can have adequate capacity
> to handle the load (memory-wise).
>
> Maybe I'm too naive, but unless I was expecting, say, greater than 10MB
> or 20MB files on a regular basis, I wouldn't think twice about just 
> using
> strings.  Much simpler than trying to juggle temporary files and 
> references
> to them and removing them when appropriate etc.
>
> Is your concern a case of premature optimization?

It's not premature optimization, it's proper design for an async 
framework.  If for no other reason, large strings take non-trivial 
amounts of time to process.  For example, on my laptop (which has a gig 
of ram): ('a' * 2**20).replace('a', 'b') takes about 0.35 seconds.. 
Twisted is not threaded, so that's a bad thing to do.  Processing 
chunks with file-like interfaces is scalable, large strings are 
definitely not.

-bob