How do you conserve memory when receiving messages from POSIX queues?

How do you conserve memory when receiving messages from POSIX queues?

It seems that when using POSIX queues in a multiprocess / multithreaded environment, there is no thread safe way to dequeue a message into a buffer that is anything smaller than the max_msgsize.

Are there any standard solutions to this problem? Or is it even a problem?

I am well aware that there are other really great libraries to do this, but I just wanted to include a completely standard solution for users 开发者_如何转开发if they don't want to deal with dependencies.

FYI, I am trying to queue up potentially hundreds of megabytes per message and have a pool of processes with multiple threads each dequeuing the messages for processing.

Thanks, Chenz

The POSIX queue interface, as you note, does not allow you to query the size of a message.

In effect, therefore, all messages may be at the maximum size as configured by the queue definition, and you have to assume that a simplistic implementation might well make use of that for ease of record-keeping.

Given that you are dealing with multi-megabyte messages as yo say, putting those messages into the queue is unlikely to be a good solution (unfortunately).

If your message rate is low (to some definition of low) and you actually do have a reasonable upper bound, then just go ahead and try it out.

Barring that, your next best bet would be to use the queue as a work-order queue, and not as a work-item queue. Your work items would have to be stored differently, in files perhaps. Then in the queue you have a nice short filename, pointing to the location of the work-item to take care of.

Good Luck

I think there is no really good way to do this. Here is an idea, but I think you will find it performs badly due to lock contention:

Have one static buffer that is equal the size of the maximum message. Because there is a shared buffer, now your dequeue process must look like this:

• lock sempaphore
• dequeue into static buffer
• figure out the real size of the message
• copy from static buffer to a thread local buffer that is the actual size of the message
• unlock semaphore

There is the overhead of having the static buffer, but depenidng on the distribution of the sizes of your messages, you are still likely to see a reduction in total memory usage. However, now you have to deal with the contention for the static buffer, which is likely to be great especially when a few large message arrive in a row. If very large messages are rare, then this might not be a terrible solution.