[Twisted-Python] Sending large amounts of data over Pb

[Jean-Paul Calderone]

On Tue, 25 Apr 2006 11:32:21 -0600, Gabe Rudy <rudy at goldenhelix.com> wrote:
>Hey guys,
>
>In a few specific cases (not as a general case) I'm sending a significant list
>of items over pb in a pb.RemoteCache/pb.Cacheable object pair. To give you an
>idea, it's a list of ~4000 tuples where each tuples has about ~10 strings of
>~10 length.
>
>Right now twisted works fine, but takes about 10 seconds to
>serialize/send/unserialize the data, or I guess in twisted terms,
>banana/send/recv/banana the data.
>
>Since my list is all primitives I thought maybe pickling it before sending
>would speed it up as banana would have to do less work, but I get a
>
>twisted.spread.banana.BananaError: string is too long to send (706378)
>
>Is that an arbitrary length limit or technically significant?

Arbitrary.

>
>Is there other methods of speeding this up?
>
>Or alternatively, is it possible (within pb) to receive the state of  a
>RemoteCache or RemoteObject in definable chunks (say 100 items from the list
>at a time), allowing me to start displaying the data in my UI and adding more
>to it as it comes async?

I'm not sure how you are using RemoteCache, but the general technique here is to use a pager.  Take a look at twisted.spread.util.Pager, and the two examples/tools that come with it, StringPager and FilePager.

Jean-Paul