[Twisted-Python] Client/Server to stream text logfiles?

[Victor Hooi]

heya,

Thanks to everybody for their replies =).

@Jason Rennie: Thanks for the links - I suspect there's going to be a bit of
a curve to me getting on board with Twisted =).

Still not sure if Twisted will satisfy all of my requirements, but I'll need
to do some more investigation.

However, what did you mean by it may be "nontrivial to use twisted with
non-twisted friendly libraries"? What would make a library non-Twisted
friendly?

Not sure what libraries we'll be depending on, but watchdog (
http://packages.python.org/watchdog/installation.html) might be just one
example.

@Glyph Lefkowitz: Aha, thanks for the headsup about Solaris. I probably
should have checked that myself...lol.

In terms of the buildbot, I'm not sure I'm the most suitable person. The
primary reason I use Solaris is because my workplace is partly a Solaris
shop - and even then, I have minimal contact with most sysadmin aspects of
it.

And my experience with OpenCSW and the awesome team there shows that some of
the quirks on the Solaris platform (particularly the older versions) can be
a pain to work around, and getting stuff to build cleanly on it can often be
non-trivial, at least if you're not experienced with the platform.

@Johan Rydberg: Reliable delivery is definitely an issue - one of the
production management guys today said "whatever you're hacking up, we want
to make sure we don't miss any lines."

The Scribe logging server you (https://github.com/facebook/scribe/wiki)
mentioned actually looks very interesting.

Another avenue I'm looking at the moment is RabbitMQ - the idea being I hook
up a pseudo "tail -f" in Python, and use that to publish
to a RabbitMQ instance. (I have little experience with messaging buses
before this - it's only something I'm learning about now, to be honest).

Then, I just get each application that needs to monitor log events to just
subscribe to the appropriate queue. The thing I like about this approach is
that it decouples the producers/consumers somewhat - we can have multiple
different applications spread across several monitoring boxes subscribing to
the same events. That, and I suspect it might scale a little bit better than
any slipshod job with Twisted I'm able to hack together in the time I have.
Thoughts?

Cheers,
Victor

On Tue, Sep 27, 2011 at 14:09, Victor Hooi <victorhooi at yahoo.com> wrote:

> Hi,
>
> We have several Solaris 10 running an application that generates large text
> logfiles
>
> We need to stream these logfiles in realtime to a central monitoring
> server.
>
> Unfortunately (for both technical and non-technical reasons), we can't use
> an NFS-mount. At the moment, we're using SSH with tail -f to stream the
> files from the servers to the monitoring box.
>
> However, I was going to look at hacking a client/server up in Python to
> monitor for filesystem events, and then stream any updates.
>
> In terms of filesystem monitoring, this is Solaris 10, so there's not
> really that much available it seems. Gamin seems to be the best bet,
> otherwise I might have to fallback to polling.
>
> On the Twisted side, is the above something that's fairly easy to
> accomplish? I haven't used Twisted before, so I thought this might be a good
> excuse to learn it. Any thoughts or caveats I should be aware of? Any
> existing projects I might be able to leverage off? Good starting docs for
> this sort of thing?
>
> Thanks,
> Victor
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </pipermail/twisted-python/attachments/20110930/dd5c1297/attachment-0001.html>