we are currently performing some benchmarks for an open source academic project, Logbus-ng. It basically implements Syslog protocol over UDP (RFC 5426) and TLS (RFC 5425). We know that the advantage of TLS is reliability (ie. we won't lose messages) but with the drawback of performance.

We have a benchmarking client and also a special forged Apache installation that send messages at high rates. Our goal is reduce the loss of UDP packets to the minimum. Apache 1.3.41 has been instrumented in order to send special log messages via UDP (not in Syslog format but in a special short syntax we parse on the server-side), and such instrumentation makes it send over 2000 messages when httpd start, and we want it to happen :)

More, I can tell you that during the ramp-up ph开发者_如何学Goase of Apache this small amount of messages (compared to other workloads we submitted to the log server) is sent at an extremely high rate, possibly flooding UDP.

Now, the log server is located on a different machine than the HTTP server, and both have barely decent hardware (not even a dual core CPU but a Pentium 4 with HyperThread). The log server code is in C#. The following method is run by 4 threads in AboveNormal priority

    UdpClient _client;
    IQueue<T>[] _byteQueues; //not really IQueue, but a special FIFO queue class that reduces overhead to the minimum

    private void ListenerLoop()
    {
        IPEndPoint remoteEndpoint = new IPEndPoint(IPAddress.Any, 0);
        while (_listen)
        {
            try
            {
                byte[] payload = _client.Receive(ref remoteEndpoint);

                _byteQueues[
                    (((Interlocked.Increment(ref _currentQueue))%WORKER_THREADS) + WORKER_THREADS)%WORKER_THREADS].
                    Enqueue(payload);
            }
            catch (SocketException)
            {
            }
            catch (Exception)
            {
            } //Really do nothing? Shouldn't we stop the service?
        }
    }

In order to reduce the time the thread spends outside the Receive method, we don't parse the message once received but store it inside one of 4 special queues that will be read by other worker threads. As far as I know, .NET scheduler is greedy, so no matter how long threads are waiting, higher priority threads will be scheduled earlier and potentially cause starvation, so this is why we currently don't care about the increasing number of threads in the application (they are globally around 20).

Not only we increase thread priority, but we try to increase the UDP buffer size to 1MB. Here is a fragment of the initialization code

try
{
    Socket clientSock = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp)
                            {
#if !MONO
                                //Related to Mono bug 643475
                                ExclusiveAddressUse = true,
#endif
                            };

    if (ReceiveBufferSize >= 0) clientSock.ReceiveBufferSize = ReceiveBufferSize;
    clientSock.Bind(localEp);
    _client = new UdpClient {Client = clientSock};
}
catch (SocketException ex)
{
    throw new LogbusException("Cannot start UDP listener", ex);
}

ReceiveBufferSize is configured at runtime...

Each log message sent by Apache is very short, I think no more than 50 bytes. We run a gigabit Ethernet in our lab.

During the last experiment with such a configuration, the log server received only 700+ of the more than 2900 generated. Wireshark reported more than 2900 messages on the UDP socket, but the log trace of Logbus (which stores all received messages into a file) reports only these 700/800. Doing cat /proc/net/udp and tricking with lsof to find the correct row reports lots of dropped packets. The logs were definitely sent at a very high rate. If we modify Apache core to sleep for a short time (little less than a millisecond) after each log call, we would reduce loss to zero, but performance would be also reduced to almost-zero too. We will do such a test, but we must prove the effectiveness of Logbus-ng in real-life scenarios :(

My straight questions are

1. Does UdpClient.ReceiveBufferSize help preventing packet loss? What else can I do in C#?
2. It is obviously supposed to work in Mono too, but do you know about possible bugs with that property? I mean, did anyone ever reported a bug? (Mono 2.8)
3. Do you know if sending packets to localhost first may reduce packet loss? (I would run a special instance of the log server on the web server machine, then forward logs via TLS, which doesn't lose, to the real log server)
4. What would you suggest me to decrease the loss rate

We must currently perform a special test with Apache, and we can only use UDP to deliver messages. We can't choose TLS because we have only C# APIs for that.

Thank you in advance for any help. I hope to have been clear. You can find the source code of the UDP receiver on SVN if it helps

The ReceiveBufferSize definately affects UDP sockets (ie. UdpClient), if the packet loss is due to buffer overflow then yes increasing the ReceiveBufferSize will help.

Keep in mind that if the data rate is so high that you simple cannot read from the buffer quick enough for long enough then it is inevitable that you will overflow even the largest of buffers.

I have used UdpClient.Client.ReceiveBufferSize effectively on Mono 2.6.7 running on Ubuntu, so I believe the Mono implementation is fine, of course I have not used this with Mono 2.8 yet.

From my experience, sending UDP packets to localhost at extremely high rates, some packet loss is possible, though I have never experienced this packet loss in a realworld application. So you might have some success with this approach.

You also need to look at were the packet loss is occuring, it might be that the packet loss is due to network infrastructure, packet collisions, switch might be dropping the packets because of some limit on the switch.

Simply put, you need to be ready to handle and expect packet loss when using UDP.

If you are using udp, you must expect packet loss. There are a lot of reason for packet loss. For you, most probably its because of packet overflooding anywhere on the channel. could be your switch or could be your receiver. The reason for overflooding is because UDP doesnt have any kind of congestion control. Tcp does have congestion control (slow start) thats why with tcp, its never(theoretically under perfect environment) overflooding.

Promising way of preventing overflooding for udp transmission is by employing tcp slow start congestion control strategy manually.

To answer your straight questions 1. No. See tcp slow start algorithm. 2. Most probably its not a bug. Udp is designed that way. Theres a reason for that and theres a need for that. 3. No. Proxy does not help. 4. Simplest implementation to combat packet loss due to overflooding is to wait for acknowledge from receiver(noting that receiver has successfully received the packet) before sending more packets. Of course, this does not help in combating packet loss due to other reason.

It is obviously supposed to work in Mono too, but do you know about possible bugs with that property? I mean, did anyone ever reported a bug? (Mono 2.8)

Apparently, setting the socket receive buffer size was badly broken in mono up to version 3.2.7: Instead of the specified size, it would get set to random values, so trying to increase the buffer size could actually make performance worse :-(

https://github.com/mono/mono/commit/bbd4ee4181787189fbb1f8ba6364afdd982ae706