[Twisted-Python] doPoll timeout problems
Wim Lewis
wiml at hhhh.org
Tue Oct 16 20:28:11 MDT 2018
Scott, Barry wrote:
> We are seeing a problem with poll being called very often and think that the
> problem is in the timeout calculation.
I think you're right and that changing that line to use ceil() is a good
fix. It does mean that some timeouts might happen up to a millisecond
later than they would otherwise, but I think that's better than having the
reactor spin-wait to consume time.
This probably only happens with the poll() reactor, because all the other
APIs seem to have higher resolution timeouts --- select, epoll, kqueue,
ppoll all have microsecond or even nanosecond timeout parameters.
If there is code that actually needs sub-millisecond timeout resolution it
might be possible for Twisted to implement that using setitimer() or
similar, but that seems like a lot of work to support an exotic use case
that could be handled more efficiently by switching to epoll etc.
I checked the history of that line of code and it dates back all the way to
the first poll()-based event loop implementation in 2001. Perhaps computers
were slow enough back then that a call to poll() would take most of a
millisecond :) but in any case the rounding-down behavior doesn't seem to have
been explicitly chosen.
> These _updateLogDateTime call seems to be a lot of complexity for no
> benefit. After all time.time() is very fast on Linux, why cache the log
> time strings?
Perhaps it is not the time() call but the cost of converting a time value
to a string that is being avoided here? Sometimes the calendar/date
calculations are expensive.
It seems to me that HTTPFactory could be implemented more efficiently by
only caching the string on demand, and then setting a timer to clear the
cache (reset _logDateTime to None) at the next 1-second mark. On a
heavily-loaded server, that would have the same properties as the current
code; but on a lightly-loaded server it would avoid running that callback
unneccessarily. And overall I don't think it's any more complicated than
the current implementation.
Wim.
More information about the Twisted-Python
mailing list