[Twisted-Python] doWrite on twisted.internet.tcp.Port

[chris at cmsconstruct.com]

Hey guys,

 

Last year I hit a condition discussed in this ticket:
https://twistedmatrix.com/trac/ticket/4759 for doWrite called on a
twisted.internet.tcp.Port.  

 

I ignored it at the time since it was just on Linux, and my main platform
was Windows.  Now I'm coming back to it.  I'll add context on the problem
below, but first I want to ask a high-level, design-type question with
multiprocessing and Twisted:

 

Referencing Jean-Paul's comment at the end of ticket 4759, I read you
shouldn't fork a process (multiprocessing module) that already has a Twisted
reactor.  Understood.  But what about a parent process (not doing anything
Twisted) forking child processes, where each child process starts their own
Twisted reactor?  Is that intended to work from the Twisted perspective?

 

 

Context:

I only hit this problem on Linux, not Windows.

 

The software project (github.com/opencontentplatform/ocp) has been a lot of
fun, especially with walking the tight rope in using multi-processing,
multi-threading, and Twisted reactors.  The main controller process kicks
off about 10 child processes, each doing different types of work.  In
general though, the child processes individually start a Twisted reactor,
connect to Kafka, connect to a database, use a shared REST API, and some
listen for connecting clients to accomplish work.

 

I test on Linux about once a year, so too many changes to rollback and
figure out that way.  It was working on Linux 2 years ago, but last year's
testing and current testing, receive the doWrite error.  It continues
running fine on Windows.  I've gone back about 2 years of versions with
Python3, Twisted, and dependent libs. on both Windows and Linux.  Every
version change yields the same result - continues to work on Windows and
continues to hit the error on Linux.  So something I added has caused Linux
to throw that error.  

 

I'm not explicitly sharing much between the main process and the sub
processes.  They are spun up with (1) a shared multiprocessing.Event() - to
have the main process shut the children down, (2) their own unique
multiprocessing.Event() - to have the child processes notify back to the
parent, and (3) a deep copy of a dictionary (containing a bunch of settings
that remain constant).  The main process uses twisted.logger, but for
testing I strip that out to remove any twisted imports in the main process.
So I'm not importing anything Twisted in the main process, and I don't
believe I'm explicitly sharing something I shouldn't.  Seems like something
is implicitly being exposed/shared across Linux child processes, that aren't
on Windows.

 

The tracebacks come through on Linux (sometimes randomly), on the console of
the parent controller process.  No need to paste here, since it's the same
as the ticket shows.  I can't reliably reproduce the problem, but I know if
I stop/start client connections
(ServerFactory/twisted.internet.interfaces.IReactorTCP and
twisted.internet.protocol.ReconnectingClientFactory) then it will eventually
happen.  I need to devote time at whittling down the code and attempting to
create a reliable test case. if even possible.

 

The error is slightly different when running HTTP vs HTTPS, but the story is
the same.  It cripples whatever child process that hits it, from doing much
of anything thereafter.  Not much luck with troubleshooting.  The tracebacks
do not include a calling function from my code, to tell me where to start.
And it happens across different child process types, so not the same one
each time.  When I throw debuggers on the child processes, the problem seems
to mask itself.  Well, at least I didn't hit the problem over the last 3
days using pudb and stepping through code at breakpoints.

 

I'm absolutely open to suggestions for troubleshooting, but first wanted to
take a HUGE step back and ask a design question regarding Twisted and
multiprocessing.

 

Thanks!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20200911/2f9e1396/attachment.htm>