Re: Load balancing PostFix Mail Out

On Thu, Dec 03, 2009 at 06:55:19AM +0100, boris17480#gmail.com wrote:
>
>
> Le 03/12/2009 06:40, Willy Tarreau a écrit :
> >On Thu, Dec 03, 2009 at 06:09:06AM +0100, boris17480#gmail.com wrote:
> >>We are "handle" arround 500'000 mails 3 times by week, but we have to
> >>load balance because we used dkimproxy for signing emails, and we have
> >>to use 3 IPs to don't be "bad" for mail services receiver.
> >
> >But your first postfix or whatever relay should be able to do that by
> >itself if you specify multiple outgoing gateways, am I wrong ?
>
> Not at all, haproxy is just easier to manipulate adding/stopping server.
> If my "first" relay keep mail into his queue, he really going to load. I
> prefer to unload it as fast as I can

No, your relay must be configured to limit the number of concurrent outgoing mails, that's all. That's precisely its job. It manages a queue and sends a few mails at a time to ensure it will never overload. There is nothing wrong with having 1 million mails in queue (though you need a decent filesystem).

> >>Currently my stats of sending are like this with HAProxy:
> >>ara:/var/log# time smtp-source -s 40 -l 10120 -m 100097 -c -f
> >>root#mydomain -4 -t root#another_domain_of_me localhost:10025
> >>100097
> >>
> >>real 6m39.309s
> >>user 0m5.864s
> >>sys 0m13.913s
> >
> >6 minutes for how many mails ? Also something looks wrong here. Your
> >tool took 19 seconds of CPU and waited for something else for 6m20.
> >Most likely this is the remote end but I find it disturbing that the
> >remote end needs about 19* more time than the local sender to deliver
> >a mail, looks like some DNS resolver issues.
>
> Hum, no I just want to "send" email "faster" to remote MTA. It's not
> really "sent" in 6 minutes, but my "sender source" have send all this
> emails and now it "unloads".

Sorry but I don't get it. How many mails was that ?

> >haproxy will not queue your mails anyway. This is a job for your
> >local postfix server. By the way, it's very common to use a local
> >MTA from web applications to send mails out. That way you use
> >local queuing which is very cheap and offers excellent response
> >times. And it's easy to recover from network issues. At worst,
> >you just have to wipe out the queue in case of problem.
>
> That's what I'm saying to the PHP dev, but he said it's my fault and he
> wants to keep the "smtp configuration" in his soft

They can keep the SMTP config in their soft and use the local postfix at the same time. It's just a matter of IP:port.

> >I have really no idea about what your php script does. But your mention
> >of "fgets" seems strange to me because in the libc and in other languages,
> >"fgets" is what you use to read one line from a file/pipe/socket. So I
> >don't see the relation between that and emitting a mail :-/
>
> He uses this kind of script to send mail :
> http://support.webecs.com/KB/a390/php-mail-script-with-smtp-authentication.aspx

OK, fgets here is used as in an y other language, just to read one status line at a time. fputs is used to send one line or part of the message.

> I just see an "fgets changelog" into haproxy.

This is completely unrelated. haproxy does not know what function your PHP script uses. The fgets you see in the changelog concerns the log parser, "halog", which uses its own fgets implementation.

> I'm trying it on our next "emailing campaign".

Once again, I really think you're talking the wrong approach in your architecture. But it's your choice, not mine :-)

> >>I just begin to improve haproxy performance for load balancing mail, and
> >>my email to the mailing list is just here to have tips and mistakes I've
> >>done ;)
> >
> >Quite honnestly, it has little added value for SMTP because the standard
> >tools already do that very well. I really think that you're using it to
> >try to solve a wrong problem. If your mails take a long time to be
> >forwarded
> >from the local postfix to the remote ones, you have another problem you
> >need
> >to solve. Haproxy might only slightly hide the problem, which does not mean
> >it will be solved.
> >
>
> Completly all right with it. I just want to now if there is "better" way
> to do it. For now HAProxy seems to do it good works, like MTA.

The problem you'll have when using haproxy instead of a local MTA is that every time a remote server will slow down for any reason, your php script will suffer from that because haproxy will not spool anything. This is why it is very important to have a local spool on the machine.

Regards,
Willy Received on 2009/12/03 07:10