[Twisted-Python] epoll reactor problems

[Alec Matusis]

> That's old (debian stable ? :)). I don't say that'll solve your
> problem, but you
> could try with 2.4.4 (warning, not 2.4.3).

It's SuSE stable ;-) Our stuff on that machine is pretty convoluted now, so
we will probably have a chance to test with 2.4.4 only in a week, when we
add a brand new server with 2.4.4.

> {EPOLLIN|EPOLLPRI|EPOLLOUT|EPOLLRDNORM|EPOLLRDBAND|EPOLLWRNORM|EPOLLWRB
> AND|E
> > POLLMSG|EPOLLERR|EPOLLHUP|0x7820, {u32=5529648, u64=5529648}},
> > {EPOLLIN|EPOLLPRI|EPOLLRDNORM|EPOLLRDBAND|EPOLLMSG|0x1000, {u32=0,
> > u64=22827751178240}}, {0, {u32=0, u64=0}},
> > {EPOLLOUT|EPOLLWRNORM|EPOLLERR|EPOLLONESHOT|EPOLLET|0x3fffa820,
> {u32=32767,
> > u64=18097643565645823}}}, 1434, 30) = 5
> 
> This part is highly suspect. The flags on the fd should not reach these
> values.

I noticed a difference between this from the 99.9% CPU server:

epoll_wait(4, {{EPOLLERR|EPOLLHUP, {u32=423, u64=12304606485815493031}},
{EPOLLIN|EPOLLPRI|EPOLLOUT|EPOLLRDNORM|EPOLLRDBAND|EPOLLWRNORM|EPOLLWRBAND|E
POLLMSG|EPOLLERR|EPOLLHUP|0x7820, {u32=5529648, u64=5529648}},
{EPOLLIN|EPOLLPRI|EPOLLRDNORM|EPOLLRDBAND|EPOLLMSG|0x1000, {u32=0,
u64=22827751178240}}, {0, {u32=0, u64=0}},
{EPOLLOUT|EPOLLERR|EPOLLONESHOT|EPOLLET|0x3fffa820, {u32=32767,
u64=18097643565645823}}}, 1432, 68) = 5

and this from a normal server running at 5% CPU:

epoll_wait(4, {{EPOLLIN, {u32=1769, u64=12304606485815494377}}, {0,
{u32=4294944684, u64=140737488332716}}}, 1728, 17) = 2

What does this mean?

> What's the global state of the process? Memory, number of opened fd ?

We immediately reverted to poll, so I do not have it in front of me. The RSS
size was 45MB, and the number of open fd I do not know: it should have been
about 1500, but I did not check.

I can do another test run with epoll in about 20hrs, since I do not want to
upset users too much. If you have some specific data I should get from the
test run, please let me know now.

Thanks
Alec
 


> -----Original Message-----
> From: twisted-python-bounces at twistedmatrix.com [mailto:twisted-python-
> bounces at twistedmatrix.com] On Behalf Of Thomas Hervé
> Sent: Wednesday, April 11, 2007 2:43 AM
> To: 'Twisted general discussion'
> Subject: RE: [Twisted-Python] epoll reactor problems
> 
> Quoting Alec Matusis <matusis at matusis.com>:
> 
> >> We would probably need more information. What's your version of
> python?
> >
> > We are using Python 2.4.1
> 
> That's old (debian stable ? :)). I don't say that'll solve your
> problem, but you
> could try with 2.4.4 (warning, not 2.4.3).
> 
> >> Can you
> >> provide a reproductible example?
> >
> > It's hard to provide a reproducible example: we observe this problem
> only on
> > the live servers. So I do not know how to simplify the code (it has
> 40000+
> > lines) such that the problem still remains, since we would have to
> try it on
> > real users...
> 
> OK...
> 
> > Did you try to do a strace on your
> >> running
> >> server to see what's going on?
> >
> > I did run strace- it made the server unresponsive, so it had to be
> > restarted. Here is the output from the problematic server at 99% CPU:
> 
> > epoll_wait(4, {{EPOLLERR|EPOLLHUP, {u32=1023,
> u64=12304606485815493631}},
> >
> {EPOLLIN|EPOLLPRI|EPOLLOUT|EPOLLRDNORM|EPOLLRDBAND|EPOLLWRNORM|EPOLLWRB
> AND|E
> > POLLMSG|EPOLLERR|EPOLLHUP|0x7820, {u32=5529648, u64=5529648}},
> > {EPOLLIN|EPOLLPRI|EPOLLRDNORM|EPOLLRDBAND|EPOLLMSG|0x1000, {u32=0,
> > u64=22827751178240}}, {0, {u32=0, u64=0}},
> > {EPOLLOUT|EPOLLWRNORM|EPOLLERR|EPOLLONESHOT|EPOLLET|0x3fffa820,
> {u32=32767,
> > u64=18097643565645823}}}, 1434, 30) = 5
> 
> This part is highly suspect. The flags on the fd should not reach these
> values.
> 
> What's the global state of the process? Memory, number of opened fd ?
> 
> --
> Thomas
> 
> 
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python