[Twisted-Python] Strange behavior when transferring large strings

[Gabriel Rossetti]

Gabriel Rossetti wrote:
> Hello,
>
> I have been trying to use the producer/consumer paradigm to transfer 
> large strings. I've been having trouble because I have strange things 
> going on, but I now suspect it isn't the P/C that's not working, 
> because it works fine with smaller strings than the ones I'm having 
> trouble with, so I suspect the problem comes from elsewhere. What 
> happens is it starts off fine, but then the destination at some point 
> only gets part of the string but the source has already sent 
> everything, closed it's transport and called stopProducing(). Could 
> there be some internal buffering problem, like I'm sending too much 
> data at too high of a throughput, or maybe something like some sort of 
> max time a socket can stay open or something? If I tell it to split 
> the string up into smaller parts, it doesn't change anything, if I 
> make the string smaller, whatever the size of each chunk, it transfers 
> fine.
>
> Thanks,
> Gabriel
>
I removed the P/C code and used the previous version and the problem 
persists, if I send a XML msg (very small XML part) with a data payload 
containing a string generated like this :

    initialMsg = "".join([str(x) for x in range(5000)])


which is 10374 bytes of data (including the XML), this makes me suspect 
that there is maybe some sort of buffer or something that is full and 
doesn't have the time to empty and twisted stops working or something. 
I'm using the 2.5 version since Ubuntu hasn't upgraded (and apparently 
won't upgrade until the lenny) their version yet and I can't get it to 
compile from source (yes, I installed python-dev and build-essentials), 
so maybe it's a problem only to this version. Does anyone have an Idea 
of what the problem is? I'm using the following code to send messages, I 
don't think that's the problem but you never know :

    def sendMessage(address, port, message, needAnswer=False):

        d = defer.Deferred() if needAnswer else None
       
        class MessageSender(Protocol):

            def sendMessage(self, msg):
                if domish.IElement.providedBy(msg):
                    msg = msg.toXml()
        
                if isinstance(msg, unicode):
                    msg = msg.encode('utf-8')
                   
                self.transport.write(msg)
               
            def dataReceived(self, data):
                d.callback(data)
           
        def gotProtocol(proto):
            proto.sendMessage(message)
            if(not needAnswer):
                proto.transport.loseConnection()

        c = ClientCreator(reactor, MessageSender)
        c.connectTCP(address, port).addCallback(gotProtocol)
        return d

   
Gabriel