Hi Alex,
On Tue, Oct 04, 2011 at 01:38:40AM +0100, Alex Davies wrote:
> Hi,
>
> I am running haproxy 1.4.18 on RHEL6 (64 bit). I am trying to run a
> large number of connections (I have long pooling backends), ideally
> several hundred k. My servers have 24G RAM and are doing nothing else.
> I have two problems.
>
> Firstly, if I increase teh "nofile" hard limit in
> /etc/security/limits.conf to anything over 1048576 (1 Megabyte,
> although I think this is defined as a # of files) I can't SSH to the
> box or start haproxy - new SSH connections are immediately closed
> without entries in the logs. The kernel limit (/proc/sys/fs/file-max)
> is set to >200k without problem. Haproxy balks as follows:
>
> Starting haproxy: [WARNING] 276/012759 (2924) :
> [/usr/sbin/haproxy.main()] Cannot raise FD limit to 2000046.
> [WARNING] 276/012759 (2924) : [/usr/sbin/haproxy.main()] FD limit
> (1048576) too low for maxconn=1000000/maxsock=2000046. Please raise
> 'ulimit-n' to 2000046 or more to avoid any trouble.
There is a default limit of 1048576 fds per process, which you can change with nr_open. But be careful when increasing all those values! Some programs might not like it at all and some scripts may become slow as hell when they close every possible fd before a fork(). So you don't want to have everything running unlimited. Ideally, keep the default ulimit at 1024 and let haproxy change it without impacting other processes.
Note: the highest test that was done on haproxy was with 1M fds, nothing above (about 480k conns).
> Secondly, I notice that as my number of connections rises I get "Out
> of socket memory" errors in the kernel log. Google led me to believe
> that this would be a problem caused by excessive orphan sockets, but
> this seems not to be the case:
I don't believe in orphans too for this case, because normally you see a message "too many of orphans" long before running out of memory, since orphans are often limited to a small number.
> [root#frontend2 log]# cat /proc/net/sockstat | grep orphan
> TCP: inuse 134376 orphan 0 tw 87 alloc 150249 mem 100217
> (output taken during a time of problem, where haproxy was mostly
> inaccessible and most connections were refused)
>
> I also seem to have plenty of spare sockets:
>
> [root#frontend2 log]# cat /proc/sys/net/ipv4/tcp_mem
> 2303712 3071616 4607424
> [root#frontend2 log]# cat /proc/net/sockstat
> sockets: used 150458
> TCP: inuse 134376 orphan 1 tw 69 alloc 150250 mem 100217
> UDP: inuse 12 mem 3
> UDPLITE: inuse 0
> RAW: inuse 0
> FRAG: inuse 0 memory 0
>
> (100217 < 4607424 by some margin).
The usual thing is when some socket buffers are too large. For instance, the default 87k read and 64k write buffers imply that you consume 150k by default for each socket, until the kernel starts to reclaim socket memory. All the people I know who use very large amounts of sockets use small buffers, both in haproxy and in the kernel, because most often it's for chat/websocket or such things which transport few data. You could very well set your default kernel TCP buffers to 8k and let them grow larger if needed :
tcp_rmem 4096 8192 65536
tcp_wmem 4096 8192 65536
You should never forget that a full connection through haproxy uses :
So if you set haproxy to 8kB and your kernel to 8kB, you will need 16kB for haproxy and 32kB for the system = 48kB per connection. That's already too much for 1M connections at 24 GB. The kernel will never go below 4kB, and at this point you'll have 32kB total = 32 GB of RAM per million connection.
> Ideally i'd like to set the number of open files and sockets to as
> close to infinite as possible - and allow them to use all the system
> RAM.
You will use all the system's RAM if socket buffers are properly tuned.
I'm used to say that you need 1/3 of the RAM for TCP buffers, another 1/3
for haproxy and the last 1/3 for the rest of the system (caches, ...).
With 24 GB, you can even assume close to 12+12 and almost nothing for
the system. Your TCP settings already allow you to use up to 18 GB RAM
for sockets. Haproxy will allocate what it needs for its maxconn, so in
practice you can max out memory. It is even a bit dangerous, you should
ensure that the system will never overcommit memory (set the param to 2)
so that it fails on malloc() when the limit is reached instead of
pretending the memory exists and killing the process when it tries to
use it. But that's for a second point.
Regards,
Willy
Received on 2011/10/04 07:49
This archive was generated by hypermail 2.2.0 : 2011/10/04 08:00 CEST