boost::thread_resource_error when more than 32705 threads

I'm implementing a message passing algorithm. The messages propagate through the nodes of the graph, blocking until they have have received enough information (from other neighbours) to send a message.

The algorithm is easy to write if I put each message in its own thread and use a boost::condition to pause the thread until all the required information is available. I create many thousands of threads, but mostly only a few are active at any time. This seems to work pretty well.

My problem is, when unit testing I find that if I create more than about 32705 threads, I get

unknown location(0): fatal error in "Tree_test": std::exception: boost::thread_resource_error

and I don't know what causes this, or how to fix it.

There seems to be pleanty of memory available (Each thread only holds two pointers - the objects that the message passes between).

From this question: Maximum number of threads per process in Linux? I think the following information is relevent (although I don't really know what any of it means...)

~> cat /proc/sys/kernel/threads-max
1000000

(I increased this from 60120 - do I need to restart?)

 ~>ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I tried fiddling with the pending signals (my limit is very close to 2* that number) and stack size with ulimit -S -i 8191 - (I couldn't increase it) but these changes seemed to make no effect at all)

I'm on a 64 bit Ubunt开发者_开发问答u-10-10 if that helps...

I think with 32K threads on the system, you should look at potential solutions other than how to have more threads. For example, a thread pool (Boost has some things for this).

Anyway, on your system, aren't PIDs limited to 32768 or some such value? You're going to run out sooner or later, may as well design the system to allow processing more items than the max number of threads, I'd think.

That said, look at /proc/sys/kernel/pid_max to see your max PID--and try increasing it. This may get you beyond 32K (but may also cause unexpected behavior with programs not designed for unusually large PIDs, so be cautious).

And then you may be limited by the stack space (as opposed to virtual memory space). You could try creating threads with smaller stacks if you like.

Okay, to answer the question: you need to increase

/proc/sys/vm/max_map_count

As discussed here:

https://listman.redhat.com/archives/phil-list/2003-August/msg00025.html

and here:

http://www.kegel.com/c10k.html#limits.threads

HOWEVER: FOR BETTER WAYS TO DO THIS LOOK AT THE FOLLOW UP QUESTION:

Non-threaded alternative to waiting on a condition. (Edit: Proactor pattern with boost.asio?)

It really depends on how big your stacks are, but you're going to run out of address-space (32-bit) or virtual memory (64-bit) if you create a lot of threads.

In Linux pthreads the default stack size was 10Mb last time I checked; this means that 32k threads uses 320G of address space (note it will probably be lazily initialised, so it won't use that much virtual memory); this is probably too much.

Even if you make the stack quite small and don't exhaust the memory this way, 32k threads is going to use a lot of virtual memory for stacks. Consider using a different approach.

ulimit only affects the stack size of the initial thread (which is dynamic normally under Linux); other threads' stack size is fixed and set at thread creation time by the pthread library.