[Twisted-Python] Large Transfers

[Uwe C. Schroeder]

Hi again,

in  another thread we had the "pb size limitation" issue. However it still 
doesn't make too much sense to me, so probably someone of the core developers 
can enlighten me.
Assume I have a remotecall using pb. According to the size limitation the 
parameters given to a specific function can not exceed 640kb. What if I hand 
over an object that exceeds the 640k ? Ok, the obvious happens and cBanana 
throws an error "security precaution ....."
Is this really a good thing to do ? Shouldn't pb see that the arguments are 
larger than 640k and start paging ?
What I'm doing is to hand down XML data which is database-generated on the 
server side. Whenever a user requests a too large resultset the network layer 
fails. On the other hand the resultset already is in memory, so why not jut 
transfer it ?
I realize that his is probably bad design, but it's the easiest way to 
transfer this information. Sure I can write the stuff to a temporary file and 
page it over, however this defies the purpose, since then the original call 
results in a message to go get the file. This means I need at least 4 
callbacks for any given call (the original ok callback, then another one for 
the possible paging as well as 2 error callbacks, one for each call)
I can extend the problem by compressing the parameters with zlib ( which I'm 
doing anyways), but at some point I will hit the limit.
The other problem this creates is a timing issue. Since I have to make several 
calls in order to transfer the resultset, I have to delay database calls 
until the whole resultset is valid.
To put it more technically:

self.perspective.transfer_large_result(small_int,label,large_result_array)

will fail if large_result_array exceeds 640k. However small_int and label can 
be transferred. The only way to do this is

if numberofbytes(large_result_set) >= 640k:
	
self.perspective.transfer_first_part(small_int,label).callback(self.smallpart_ok)

and in smallpart_ok

self.page_the_rest(large_result_array).callback(self.whole_stuff_transfered)

I can continue the flow of the application in whole_stuff_transfered, however 
I have to temporarily store the values of small_int and label, since I can't 
create a database record without the rest.

This is an enormous overhead.

What even strikes me more is that this size limit not even prevents large 
memory consumption - since the object is already there and in cBanana the 
object is already stored in the buffer.

So what makes is "more secure" to limit the size ??

I think I'll just remove all size limits and go thru the (unwanted) way to 
create a own package.


	UC

--
Open Source Solutions 4U, LLC	2570 Fleetwood Drive
Phone:  +1 650 872 2425		San Bruno, CA 94066
Cell:   +1 650 302 2405		United States
Fax:    +1 650 872 2417