Re: anybody know how to health check a twisted xmlrpc server?

On Thu, Dec 24, 2009 at 12:05:54AM -0800, David Birdsong wrote:
> On Wed, Dec 23, 2009 at 11:27 PM, Willy Tarreau <w#1wt.eu> wrote:
> > On Wed, Dec 23, 2009 at 10:52:52PM -0800, David Birdsong wrote:
> >> this can be dropped for now.  it turns out that twisted doesn't
> >> actually return error codes when you instruct it to.  it puts the
> >> error codes in the xml response for code to deal with but slaps a http
> >> 200 in the headers.
> >
> > so we really need a working regex-based health check :-/
> >
> > Willy
>
> i've had coworkers complain about health checks only looking at http
> response codes.
>
> i'm undecided. i think it's reasonable for proxies/load balancers to
> work at the protocol level, but i've definitely been hamstrung by it
> when working with libraries that don't always provide access to the
> protocol levels without combing through and learning the entire
> library.

Other people need to ensure that a specific tag is present (or is not present) in a given page. For instance, it can make sense to kill a server which reports SQL errors in the HTML code.

> how hard would that be to add?

It's not that hard and it has already been done once, but the code contained various issues and bugs which need to be fixed. After some review, it appeared that it would be easier to re-implement it (possibly keeping some parts of the patch) than auditing it completely and fixing the remaining issues. If you want to work on something like that, please first check with Krzysztof (CCed), because he regularly works on stats and checks and I would not be surprized if he already had some pieces of code in good shape.

> i guess there's not much overhead
> involved seeing any sane admin runs health checks at most a handful of
> times in a 3-5 second period.

You could even do that dozens of times a second (provided you don't have thousands of servers).

Regards,
Willy Received on 2009/12/24 10:12