Are there performance issues with CSocket::Send?

UPDATE 14 June 2011

A quick update... Most respondents have focused on the dodgy method for handling the queue of messages to be logged however while there is certainly a lack of optimisation there it's certainly not the root of the problem. We switched the Yield over to a short sleep (yes, the Yield did result in 100% CPU once the system went quiet) however the system still can't keep up with the logging even when it's going nowhere near that sleep. From what I can see the Send is just not very efficient. One respondent commented that we should block up the Send() together in to one send and that would seem like the most appropriate solution to the larger underlying issue and that's why I have marked this as the answer to the original question. I certainly agree the queue model is very flawed though, so thanks for feedback on that and I have up-voted all answers that have contributed to the discussion.

However, this exercise has got us to review why we're using the external logging over a socket like we are, and while it may well have made sense previously when the logging server did lots of processing over the log entries... it no longer does any of that and therefore we have opted to remote that entire module and go for a direct-to-file approach via some pre-existing logging framework, this should eliminate the problem entirely as well as remove unnecessary complexity in the system.

Thanks again for all the feedback.

ORIGINAL QUESTION

In our system we have two components important to this problem - one is developed in Visual C++ and the other is Java (don't ask, historic reasons).

The C++ component is the main service and generates log entries. These log entries are sent via a CSocket::Send out to a Java logging service.

The problem

Performance of sending data seems very low. If we queue on the C++ side then the queue gets backed up progressively on busier systems.

If I hit the Java Logging Server with a simp开发者_如何转开发le C# application then I can hammer it way faster then I will ever need to from the C++ tool and it keeps up beautifully.

In the C++ world, the function that adds messages to the queue is:

void MyLogger::Log(const CString& buffer)
{
    struct _timeb timebuffer;
    _ftime64_s( &timebuffer );

    CString message;
    message.Format("%d%03d,%04d,%s\r\n", (int)timebuffer.time, (int)timebuffer.millitm, GetCurrentThreadId(), (LPCTSTR)buffer);

    CString* queuedMessage = new CString(message);
    sendMessageQueue.push(queuedMessage);
}

The function run in a separate thread that sends to the socket is:

void MyLogger::ProcessQueue()
{
    CString* queuedMessage = NULL;
    while(!sendMessageQueue.try_pop(queuedMessage))
    {
        if (!running)
        {
            break;
        }
        Concurrency::Context::Yield();
    }

    if (queuedMessage == NULL)
    {
        return;
    }
    else
    {
        socket.Send((LPCTSTR)*queuedMessage, queuedMessage->GetLength());
        delete queuedMessage;
    }
}

Note that ProcessQueue is run repeatedly by the outer loop thread itself, which excluding a bunch of nonsense preamble:

while(parent->running)
{
    try
    {
        logger->ProcessQueue();
    }
    catch(...)
    {
    }
}

The queue is:

Concurrency::concurrent_queue<CString*> sendMessageQueue;

So the effect we're seeing is that the queue is just getting bigger and bigger, log entries are being sent out to the socket but at a much lower rate than they're going in.

Is this a limitation of CSocket::Send that makes it less than useful for us? A mis-use of it? Or an entire red-herring and the problem lies elsewhere?

Your advice is much appreciated.

Kind Regards

Matt Peddlesden

Well, you could start by using a blocking producer-consumer queue and to get rid of the 'Yield'. I'm not surprised that messages get blocked up - when one is posted, the logger thread is typically, on a busy system, ready but not running. This will introduce a lot of avoidable latency before any message on the queue can be processed. The background thread than has a quantum to try an get rid of all the messages that have accumulated on the queue. If there are a lot of ready threads on a busy system, it could well be that the thread just does not sufficient time to handle the messages. especially if a lot have built up and the socket.send blocks.

Also, almost competely wasting one CPU core on queue polling cannot be good for overall performance.

Rgds, Martin

In my opinion, you're definitely not looking at the most efficient solution. You should definitely call Send() once. For all messages. Concatenate all the messages in the queue on the user side, send them all at once with Send(), then yield.

In addition, this really isn't how you're meant to do it. The PPL contains constructs explicitly intended for asynchronous callbacks- like the call object. You should use that instead of hand-rolling your own.

Things here that might be slowing you up:

• The queue you are using. I think this is a classic example of premature optimization. There's no reason here to use the Concurrency::concurrent_queue class and not a regular message queue with a blocking pop() method. If I understand correctly, the Concurrency classes use non-blocking algorithms when in this case you do want to block while the queue is empty and release the CPU for other threads to use.
• The use of new and delete for each message and the inner allocations of the CString class. You should try and see if recycling the messages and strings (using a pool) will help performance for two reasons: 1. The allocation and deallocation of the messages and string objects. 2. The allocations and deallocations done inside the strings can maybe be avoided if the string class will internally recycle its buffers.

Have you tried profiling to see where your application is having trouble? Is it only when logging that there are issues with the sender? Is it CPU bound or blocking?

The only thing I can see is that you don't protect the message queue with any sort of locking so the container state could get weird causing all sorts of unexpected behavior.