[Twisted-Python] Is there pb documentation somewhere?

[Glyph Lefkowitz]

On Aug 4, 2014, at 9:47 PM, Daniel Sank <sank.daniel at gmail.com> wrote:

> glyph,
> 
> >> 2. Is there a specification for the pb dialect of banana?
> >>
> > Beyond the code, no.
> 
> Ok.
> 
> > I would be happy to answer questions, but obviously I'm not super responsive :).
> > Let me know what you need.
> 
> For two personal projects, I would like to have a reasonable remote objects library in python. I need something which can announce state changes to clients, and receive state change requests from clients. My solution:
> 
> 1. Make server side class which can spawn Cacheables when it wants to tell clients of its existence.
> 2. Give RemoteCaches to clients and use observe_* methods as appropriate.
> 3. Stuff a Viewable into the RemoteCaches so that clients can request state changes.
> 
> Question #1: Is this a reasonable use of pb?

Yes.

> This all worked great until I ran into a bug. In trying to fix the bug, I found that
> 
> 1. pb code is really hard to understand

Sorry about that.

> 2. exarkun's thinks pb is bad and that I should implement what I need in AMP.

I really wish we would stop calling things "bad" and "good".  This isn't a helpful classification.  PB is adequate for a particular set of requirements.  Those requirements are somewhat unusual, and AMP is better for a lot of use-cases.

It sounds to me like you are a lot more interested in 

> 3. exarkun thinks banana and jelly are reasonable.

Again, what does "reasonable" mean in this context?

Let me explain my own opinion about this.

Banana is a perfectly serviceable low-level marshaling format.  It's pretty efficient when compared to something like JSON, and has compression mechanisms which can make it even more efficient (the "dialect" support you referred to).  The only thing about it that isn't very general is that its implementation (although not the protocol specification) hard-codes the PB abbreviated-string dialect.

Jelly is higher level, but more language-specific. Its specification implicitly encodes numerous Python implementation details, like the distinction between "tuple" and "list".  It also couples very tightly to your program's structure.  This can be a real benefit to getting a protocol up and running quickly, but it still allows you to create protocols where you don't really know what the wire format is, where you develop hidden dependencies.  In more complex protocols (where the "ease of getting up and running quickly" thing really starts to shine) this attribute of Jelly can cause real difficulty in any kind of cross-system communication: communicating with a peer from a different language, or even in Python without access to all the protocol class definitions from the original system, is hard because it requires reverse-engineering.  This is where it becomes "bad".  Still, it isn't as big of a disaster security- and maintenance-wise as Pickle.  The information you need is recorded in the code, it's just spread out, you don't need to work backwards from protocol dumps.  If I were going to spend some time maintaining PB, this is where I'd focus: if the schemas were a bit more explicit, could be collected into one place more easily, and were all validated in advance (before passing deserialized objects to the application code, or serializing them across the wire), then these problems could be addressed without changing the API too much.

PB basically just inherits all of the benefits and caveats of Jelly.  It's a trivial serialization of remote references to objects.

> Question #2: Would you recommend implementing a simplified replacement for pb on top of banana/jelly, or starting over from AMP? I favor the banana/jelly route because the protocol seems intrinsically flexible, but I read your blog explaining why protocols like banana are bad, so I'm confused about what I "should" do.

First of all, don't take my development advice as gospel.  When I write an article and publish it, I'm just trying to make people aware of issues they may not have considered; make your own decisions about how to write your own code.

(Unless your decision is to write it yourself in PHP, of course, in which case you are a danger to yourself and others and should be remanded to compulsory treatment.)

It seems like PB fits your style, and the problems with it are all tractable and fixable.  I am sad that you're not getting the development support you need to maintain it (most of all I'm sad you're not getting it from me!) but let's see if we can fix that.  I'll start by replying to your other email.

One thing that might speed things along is if you can help out with some code reviews.  We've got a _really_ long queue right now and that's making it hard for me to spend any focused effort in one particular area.  I'm happy to trade 2-for-1 - if you do two code reviews, I will regard it as an immediate obligation for me to review a ticket you direct me to ;).

It might also help to write more small, simple patches for PB.  Especially adding docstrings to make the nature of your other, more complex changes easier for reviewers to understand.  These would also be easier to land, and a couple of decades in open source has taught me that nothing motivates development activity like successful development activity ;).

Good luck,

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20140805/c5b9de65/attachment-0002.html>