Hypermail

From: Jordi Espasa Clofent <jespasac#minibofh.org>
Date: Wed, 22 Oct 2008 11:04:58 +0200

Hi all,

I've several spamassassin balanced with simple robin-round DNS; so, the last goal is balance them with HAProxy.

# GENERAL SCHEMA
192.168.4.4 Virtual Shared IP

192.168.4.9				192.168.4.10
Load-Balancer 1	                   Load-Balancer 2

192.168.1.101
192.168.1.102			Spamassassin servers

192.168.1.103
....

# GENERAL INFO
Load-Balacers are XEN domU under Debian 4.0 and XEN 3.2.1 Spamassassin are also XEN domU Debian 4.0 and XEN 3.2.1

Load-Balancers and spamassasin domUs are domUs inside the same dom0 (so, they "share" the same physical machine).

# CONFIGURATION INFO IN LOAD-BALANCER 1 (CALLED XEN-LBSPAM01)
xen-lbspam01:~# ifconfig

eth1      Link encap:Ethernet  HWaddr 00:16:3e:6d:89:6e
           inet addr:192.168.4.9  Bcast:192.168.255.255  Mask:255.255.0.0
           inet6 addr: fe80::216:3eff:fe6d:896e/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:10572 errors:0 dropped:0 overruns:0 frame:0
           TX packets:2588 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:3705886 (3.5 MiB)  TX bytes:206272 (201.4 KiB)

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:8 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:1340 (1.3 KiB)  TX bytes:1340 (1.3 KiB)

xen-lbspam01:~# cat /etc/haproxy/haproxy.cfg
# this config needs haproxy-1.1.28 or haproxy-1.2.1

global
# log 127.0.0.1 local0
# log 127.0.0.1 local1 notice

         log 127.0.0.1  local3
         maxconn 4096
         user haproxy
         group haproxy
         #daemon
         debug
         #quiet

defaults
         log     global
         mode    http
         option  httplog
         option  dontlognull
         retries 3
         option redispatch
         maxconn 2000
         contimeout      30000
         clitimeout      50000
         srvtimeout      50000

listen spamassassin_proxy 192.168.4.4:783
        maxconn 500
        mode tcp
        stats enable
        stats uri /stats
        log global
        balance roundrobin
        server xen-spam01 192.168.1.101:783 check inter 30000 fall 4 rise 1
        server xen-spam02 192.168.1.102:783 check inter 30000 fall 4 rise 1
        server xen-spam03 192.168.1.103:783 check inter 30000 fall 4 rise 1
        server xen-spam07 192.168.1.216:783 check inter 30000 fall 4 rise 1
        server xen-spam08 192.168.1.217:783 check inter 30000 fall 4 rise 1
        server xen-spam09 192.168.1.219:783 check inter 30000 fall 4 rise 1
        server xen-spam10 192.168.1.220:783 check inter 30000 fall 4 rise 1
        server xen-spam11 192.168.1.233:783 check inter 30000 fall 4 rise 1

xen-lbspam01:~# cat /etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {           # Requires keepalived-1.1.13
         script "killall -0 haproxy"     # cheaper than pidof
         interval 2                      # check every 2 seconds
         weight 2                        # add 2 points of prio if OK

}

vrrp_instance VI_1 {

         interface eth1
         state MASTER
         virtual_router_id 51
         priority 101                    # 101 on master, 100 on backup
         virtual_ipaddress {
            192.168.4.4
         }
         track_script {
             chk_haproxy
         }

}

The keepalived it's ok, because of:
xen-lbspam01:~# ip addr sh eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

     link/ether 00:16:3e:6d:89:6e brd ff:ff:ff:ff:ff:ff
     inet 192.168.4.9/16 brd 192.168.255.255 scope global eth1
     inet 192.168.4.4/32 scope global eth1
     inet6 fe80::216:3eff:fe6d:896e/64 scope link
        valid_lft forever preferred_lft forever

# THE PROBLEM
When a email calls the virtual shared IP (192.168.4.4 in this case) I see the petition on HAProxy's debug output, but the mail is not processed by spamassassin servers.

The debug output is:

Starting haproxy: haproxyAvailable polling systems :

      sepoll : pref=400,  test result OK
       epoll : pref=300,  test result OK
        poll : pref=200,  test result OK
      select : pref=150,  test result OK

Total: 4 (4 usable), will use sepoll.
Using sepoll() as the polling mechanism.

00000000:spamassassin_proxy.accept(0004)=0006 from [192.168.1.41:59842]
00000000:spamassassin_proxy.srvcls[0006:0007]
00000000:spamassassin_proxy.clicls[0006:0007]
00000000:spamassassin_proxy.srvcls[0006:0007]
00000000:spamassassin_proxy.closed[0006:0007]

As you can see, a petition from 192.168.4.41 (one of emails server which use the spamassasins pool)
arrivedm but when I checked the processed email in mailbox, the headers shows that the mails _hasn't been_ processed by spamassassin.

Moreover, I can't access to stats using http://192.168.4.4/stats

¿?¿?¿

PD: I know that I'm using several netmask ranges. Don't worry about it, because of the all machines are configured with 255.255.0.0 netmask. All oh them view all others without problems.

-- 
Thanks,
Jordi Espasa Clofent

Received on 2008/10/22 11:04

Balancing Spamassassin