Re: Low performance in solaris 9

Hello,

On Wed, May 07, 2008 at 04:03:16PM -0700, manuelsspace-listas#yahoo.com wrote:
> Hello + thanks,
>
> I have changed some items
>
> 1) rebuild haproxy. 1.2.17 now
> 2) compile pcre too, it was omitted before
> 3) haproxy's config reduced as
>
> global
> daemon
> maxconn 1000 # warning: this has to be 3 times the expected value!
> log localhost local0
>
> defaults
> mode http
> balance roundrobin
> option dontlognull
> option httpclose
> retries 1
> redispatch
> maxconn 2000
> contimeout 5000
> clitimeout 50000
> srvtimeout 50000
> stats enable
>
>
> listen backend1
> bind :8001
> log global
> option httplog
> capture request header X-Forwarded-For len 15
> option httpchk /check.html
> cookie PROXY_SERVERID
> # always down
> server apache1 10.27.40.81:1024 maxconn 100 check inter 2000 fall 3 cookie server1
> # small pc, some times down
> server apache2 10.21.19.27:80 maxconn 2 check inter 2000 fall 3 cookie server2
> #always running
> server apache3 10.27.40.81:80 maxconn 200 check inter 2000 fall 3 cookie server14
>
>
>
> 4) 1K object test, 10 threads during 200secs
>
> $ prstat -T -v 1 -u msoto 10
>
>
> PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
> 4255 msoto 4.9 26 - - - - 0.0 - 4K 2K 75K 0 haproxy/1
> ...
>
> $ vmstat 10
> kthr memory page disk faults cpu
> r b w swap free re mf pi po fr de sr dd s0 -- -- in sy cs us sy id
> 0 0 0 1206800 184952 5 40 1 0 0 0 0 0 0 0 0 496 269 132 1 1 98
> 0 0 0 1186000 161104 0 1 0 0 0 0 0 0 0 0 0 3803 11096 1800 7 42 51
> 0 0 0 1186000 161104 0 0 0 0 0 0 0 0 0 0 0 3722 10801 1755 9 38 53
> 0 0 0 1186000 161104 0 0 0 0 0 0 0 0 0 0 0 3782 10982 1794 7 38 56
> 0 0 0 1186000 161104 0 0 0 0 0 0 0 0 0 0 0 3774 10926 1781 7 39 54
> 0 0 0 1186000 161104 0 0 0 0 0 0 0 0 0 0 0 3802 11133 1817 7 41 52
> 0 0 0 1186000 161104 0 0 0 0 0 0 0 0 0 0 0 3736 10912 1791 6 40 54
> ...
>
>
> [################100%##################] 200s/200s
>
> Requests: 32835
> Errors: 117
> Avg Response Time: 0.053
> Avg Throughput: 163.79
> Current Throughput: 169
> Bytes Received: 336373758

Huh? the CPU spends 40% of its time in system at only 164 hits/s ??? What CPU is this ? My $100 ALIX board with a 500 MHz Geode on it consuming only 5W max achieves 2000 hits/s (under linux), so by this standard, you would have a 40 MHz CPU.

Hmmm... could you check the tcp_strong_iss value using ndd /dev/tcp ? I suspect it is set to 2, which means that solaris will create truly random and secure initial sequence numbers for each connection, but will consume a lot of CPU for this. You can lower it to 1.

> $ netstat|grep 10.21.19.27|awk '{print $7}'|sort | uniq -c
> 15 ESTABLISHED
> 3831 TIME_WAIT
> #// this is a hi value observed, all snapshot was near thees numbers. This may be the issue

No, TIME_WAIT is a final state, it consumes really nothing (just a few tens of bytes of RAM per session). There's no data anymore, no buffer, and of course no file descriptor. You can have millions of them without a problem.

> HAProxy does not reports any error. Connections rejected at OS layer

so it could very well be that the OS takes too much time creating new sequence numbers.

> At the backend, this is what I saw.
>
> $ netstat|awk '/sg5ts14:http/{print $6}'|sort | uniq -c
> 1 FIN_WAIT1
> 8 FIN_WAIT2
> 1 SYN_RECV
> 2941 TIME_WAIT
> ...
> $ netstat|awk '/sg5ts14:http/{print $6}'|sort | uniq -c
> 5 FIN_WAIT2
> 4425 TIME_WAIT
> ...
>
> When the test is executed directly from the client to the backend, no more that 97 TIME_WAIT was observed in a 10 simultaneous connection test. keepalive observer in server status.

If you use keep-alive, you have very few sessions, explaining why you have very few TIME_WAIT.

> Do I have to increase rlim_fd_cur & rlim_fs_max variables to include ESTABLISHED + TIME_WAIT?

No, rlim_fd_* will just be used to increase ESTABLISHED, but since you already have very poor performance with 10 concurrent sessions, it's not the problem here. As I explained above, don't worry about TIME_WAIT.

> or is there an argument to reduce TIME_WAIT timeout?

I believe in solaris 9/10, it was merged with the parameter tcp_close_wait_timeout you see with ndd /dev/tcp, but I'm not absolutely sure, I don't use solaris very often. By default, it is set to 240 seconds (240000 ms). It's very large but should not cause any problem since haproxy does a setsockopt with SO_REUSE_ADDR when connecting.

regards,
Willy Received on 2008/05/08 08:28