[Twisted-Python] Large Transfers

[Glyph Lefkowitz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday, May 10, 2003, at 10:33 AM, Uwe C. Schroeder wrote:

>> That's *way* to DWIMy, IMHO.
>
> but it would be more convenient and transparent for the programmer not 
> to have take care of paging :-)

And surprising in corner cases, and very difficult to debug.  This is 
basically the same case for using large numbers of threads rather than 
an event-driven API.  This is not how Twisted works.

In fact, we have taken great care to describe PB as a "translucent" 
remote access API, never "transparent".  Transparency is explicitly not 
a goal.

>> Use StringPager. It's in memory. *Always* use StringPager, even if 
>> you're below the security limit.
>
> The busines logic behind it simply gets way to complicated if I have 
> to separate calls into "small" and "large" ones.

Your response implies you have inverted Moshe's meaning.  I think what 
Moshe was saying, in your terminology, is: 'Do not separate calls into 
"small" and "large" ones.  Treat all calls as "large".  In the lowere 
levels of Twisted, this will do a reasonably efficient thing even if 
the data is actually small, and it will not complicate your code.'

> if I have to split the stuff and put some logic in there that catches 
> the large calls and pages them it will add another 5k lines of code.

Given the expansion that we have repeatedly given using 
twisted.spread.util (this may turn 1 line into 3 lines, although in 
most cases it just makes a slightly longer line) this implies that you 
have at least 1600 separate lines where you are doing a callRemote that 
is directly retrieving a potentially large XML payload.  It seems like 
this is something which you could get quite a bit of benefit out of 
abstracting into a thinner interface.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (Darwin)

iD8DBQE+vR7svVGR4uSOE2wRAgweAJwLXwdczFVevEBloRK+rDB0bfEH3gCfcOWx
7MbwYxG/2/LctNJ5EPJYfFk=
=Ehit
-----END PGP SIGNATURE-----