HAProxy TCP session count stops at 400

I am trying HAProxy for TCP load balancing. Connections come in to port X on a single IP, and the HAProxy then balances these connections to a back-end using the "leastconn" balancing method to keep the number of connections even. This is on Ubuntu 10.04 x64.

I've turned up file-max in the kernel config to 700,000. I've turned up the per-process ulimit to about 400,000. I've turned up maxconn in the haproxy config to 200,000. It reports seeing this maxconn fine:

show info
Name: HAProxy
Version: 1.3.22
Release_date: 2009/10/14
Nbproc: 1
Process_num: 1
Pid: 1355
Uptime: 0d 4h38m46s
Uptime_sec: 16726
Memmax_MB: 0
Ulimit-n: 400013
Maxsock: 400013
Maxconn: 200000
Maxpipes: 0
CurrConns: 1113
PipesUsed: 0
PipesFree: 0
Tasks: 1113
Run_queue: 1
node: XXXXX

This front-end load-balances across 5 back-end systems. However, when it gets to 400 sessions per back-end, it just plain stops balancing, and is just deferring additional connections. I can see this with the "smax" stat. You will note that the maximum number of sessions on each is 400, and the total maximum number of sessions is 2000:

show stat
#
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,
protoa,FRONTEND,,,958,2000,2000,12624,6230219,6445523,0,0,0,,,,,OPEN,,,,,,,,,1,1,0,,,,0,0,0,406,
protoa,XXX1672,0,0,191,400,,3222,1249403,1286659,,0,,221,0,664,0,no check,1,1,0,,,,,,1,1,1,,2559,,2,0,,198,
protoa,XXX1674,0,0,192,400,,3106,1242103,1289247,,0,,178,0,535,0,no check,1,1,0,,,,,,1,1,2,,2572,,2,0,,171,
protoa,XXX1707,0,0,193,400,,3043,1266305,1305311,,0,,164,0,492,0,no check,1,1,0,,,,,开发者_Python百科,1,1,3,,2551,,2,0,,161,
protoa,XXX1782,0,0,189,400,,3046,1236790,1282690,,0,,204,0,619,0,no check,1,1,0,,,,,,1,1,4,,2429,,2,0,,190,
protoa,XXX1851,0,0,193,400,,3060,1235618,1281616,,0,,189,0,570,0,no check,1,1,0,,,,,,1,1,5,,2490,,2,0,,180,
protoa,BACKEND,0,0,958,2000,2000,12624,6230219,6445523,0,0,,956,0,2880,0,UP,5,5,0,,0,17645,0,,1,1,0,,12601,,1,0,,406,
protob,FRONTEND,,,4,6,2000,28,15204,15726,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,2,
protob,XXX1672,0,0,2,2,,5,2313,2322,,0,,0,0,0,0,no check,1,1,0,,,,,,1,2,1,,5,,2,0,,1,
protob,XXX1674,0,0,0,2,,5,3520,3803,,0,,0,0,0,0,no check,1,1,0,,,,,,1,2,2,,5,,2,0,,1,
protob,XXX1707,0,0,0,2,,8,3303,3214,,0,,0,0,0,0,no check,1,1,0,,,,,,1,2,3,,8,,2,0,,1,
protob,XXX1782,0,0,1,2,,5,3529,3745,,0,,0,0,0,0,no check,1,1,0,,,,,,1,2,4,,5,,2,0,,1,
protob,XXX1851,0,0,1,1,,5,2539,2642,,0,,0,0,0,0,no check,1,1,0,,,,,,1,2,5,,5,,2,0,,1,
protob,BACKEND,0,0,4,6,2000,28,15204,15726,0,0,,0,0,0,0,UP,5,5,0,,0,17645,0,,1,2,0,,28,,1,0,,2,

Where is this limitation coming from? I really want to cramp hundreds of thousands of connections into this haproxy instance. (The machine has network, CPU and RAM to keep up)

So, reading the source for version 1.3.x, I find out that: There are two maxima. One is the global max # connections, set with -n on the command line and maxconn in global config. The other is the per-proxy max # connections, set with -N on the command line, or configured per proxy in the config. Specifically, you cannot configure the default max # connections per proxy other than from the command line! The default is ... wait for it ... 2000! So adding a "maxconn 200000" to each of the "listen" sections in my /etc/haproxy/haproxy.cfg file solves this problem.

Note that, despite the documentation being pretty good, it did not do a good job of explaining this. Specifically, from reading the documentation, I thought that the global maxconn would be applied to each proxy as well, but not so. The global maxconn is enforced for the total number of connections, but the local max number of connections for a given proxy front end has to be explicitly specified.