[Twisted-Python] dataReveived() buffer best practice?

[Phil Mayers]

On 10/06/2011 05:22 PM, Fabian Rothfuchs wrote:

> I also ran a tcpdump to confirm – The opposite server is obviously
> pushing content to the socket in arbitrary frequencies, ending up in my
> dataReceived() method to get called arbitrarily as well.

Yes, this is normal. TCP stacks are free to segment the data as they see 
fit. Machine-readable protocols have framing for this reason. Sadly, 
text streams run over telnet/SSH/CLI interfaces are usually badly 
framed, if at all.

>
> My question: is there any best practice for buffer techniques here?

Twisted comes with a bunch of higher level protocol sub-classes that 
buffer the data for you, then deliver chunks of the buffer to methods on 
your class.

For example, lineReceiver delivers whole lines.

There may well be a conch helper class that does the same.

The generic do-it-yourself approach is to append the data to a buffer, 
then look for complete protocol messages in the buffer and process them 
in order, possibly with a state machine. e.g.

def dataReceived(self, data):
   self.buffer += data

   if self.state=='waiting-for-userid':
     if 'USERID:' in self.buffer:
       # send userid
       self.transport.write('0004\r')
       self.state = 'waiting-for-password'
       # empty the buffer
       self.buffer = ''

   elif self.state=='waiting-for-password':
     if 'PASSWORD:' in self.buffer:
       self.transport.write(password+'\r')
       self.state = 'logged-in'
       self.buffer = ''

...and so on. This is boring to write by hand, so see if there is 
something in Twisted you can use already.