High Availability Web Services Using HAProxy

I was recently tasked with increasing the up time of my employer's main Web site. The site uses a content management system that lives on two Windows/IIS servers. (I know, the system was purchased before I was hired.) One server is for making changes to content (design-time server) and the other is the public web site (run-time server). The design-time server has a complete copy of the site which is replicated to the run-time server. Unfortunately the run-time server has a habit of refusing to serve pages at the most inopportune times, usually when I'm on vacation or somewhere without a computer.

I decided to place a load balancer in front of the two servers. Not really for load balancing but for failover. This way when the run-time server decides to act up connections will automatically be redirected to the design-time server until the run-time servers tantrum is finished (which is usually after a a spanking [read reboot]). I looked at a number of different tools for handling the load balancing/failover process: pen, pound, crossroads, LVS and HAProxy. All are very capable. Ultimately I chose to use HAProxy.

I configured a Linux machine and installed Apache, HAProxy, and SEC (simple event correlator). I configured Apache to listen on a high number port and respond only to connections from localhost. I also set Apache up to redirect all 403 and 404 errors to a single page explaining that a system outage has occurred. I configured HAProxy to listen on port 80 and direct requests to the run-time server, then fail over to the design time server, then fail over to the local Apache server. In the event both web servers are unable to serve requests users will receive an informative message instead of connection or routing errors.

Unfortunately, HAProxy does not provide a notification mechanism. So I configured SEC to watch the HAProxy log and send an e-mail to the appropriate network administration group whenever a web server goes down or comes up.

HAProxy like most good proxies supports the X-Forwarded-For HTTP header. However, IIS does not support it out of the box. The problem, of course, is that when requests come through HAProxy the web servers will record the IP address of the balancer instead of the requesters address. I found IIS X-Forward-For ISAPI Filter which solves this problem very nicely. Once installed it checks all requests for the X-Forwarded-For header and, if present, replaces the value of c-ip with the value of X-Forwarded-For.

We aren't doing any load balancing yet but if we want to it's as simple as putting more web servers on our network and adding them to the HAProxy configuration. HAProxy supports round robin and weighted round robin algorithms. Apparently there's a least connections patch out there as well.

The end result is an automated solution which fails over to secondary servers and keeps administrators notified of the health of the servers. Maybe this will mean I can take a vacation without restarting a web server.

Comments

Hi!
This article helps a lot. I was looking for exact same solution.
I already managed to install and configure HAProxy. But I didn't know about SEC and IIS X-Forward-For ISAPI Filter.
Thanks for the information. But somehow I cannot get IIS X-Forward-For ISAPI Filter. It is not in that link you provided. Can you help me? Can you also give me some examples to configure SEC?
Thank you.

You'll have to register at DevCentral and go to the CodeShare section. Quoting the DevCentral page, "You can download it in the CodeShare section of DevCentral. If you find a way to optimize this filter, please let me know and I'll update the sources here."

Here's an edited version of my SEC conf file:


# SEC - Simple Event Correlator Configuration File
#
# Author: Steve Moitozo <god <> zilla <> us>
# Created: 20070304
# Description:
# SEC is being used for real-time notification of
# web server failure events. When HAProxy recognizes
# a server failure it writes information about the
# failure to its log file. SEC monitors the log file
# looking for certain events and sends notification
# e-mails to administrators
#
# match on a line like this:
# Server http_proxy/www0 is DOWN. 0 active and 2 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue.
# Take the name of the instance (http_proxy/www0) and put it in $1
# Take the server status (DOWN or UP) and put it in $2
# Take the rest of the line and put it in $3
#
type=Single
ptype=RegExp
pattern=Server\s+(\S+)\s+\S+\s+(\S+)(.*)
desc=$0
action=pipe '%t server $1 went $2 $3' /bin/mail -s 'HAProxy: $1 went $2' me@example.com

As a developer i've been focusing a lot recently on CMS and installed a few to test out on my local server. I found that joomla is the most feature rich sytem to use and have also started using ready made joomla sites for my clients.

Ok, I understand you want your free advertising. But isn't this a stretch?