C# Async TCP sockets: Handling buffer size and huge transfers

When using a blocking TCP socket, I don't have to specify a buffer size. For example:

using (var client = new TcpClient())
{
    client.Connect(ServerIp, ServerPort);

    using (reader = new BinaryReader(client.GetStream()))
    using (writer = new BinaryWriter(client.GetStream()))
    {
        var byteCount = reader.ReadInt32();
        reader.ReadBytes(byteCount);
    }
}

Notice how the remote host could have sent any number of bytes.

However, when using async TCP sockets, I need to create a buffer and thus hardcode a maximum size:

 var buffer = new byte[BufferSize];
 socket.BeginReceive(buffer, 0, buffer.Length, SocketFlags.None, callback, null);

I could simply set the 开发者_开发问答buffer size to, say, 1024 bytes. That'll work if I only need to receive small chunks of data. But what if I need to receive a 10 MB serialized object? I could set the buffer size to 10*1024*1024... but that would waste a constant 10 MB of RAM for as long as the application is running. This is silly.

So, my question is: How can I efficiently receive big chunks of data using async TCP sockets?

Two examples are not equivalent - your blocking code assumes the remote end sends the 32-bit length of the data to follow. If the same protocol is valid for the async - just read that length (blocking or not) and then allocate the buffer and initiate the asynchronous IO.

Edit 0:

Let me also add that allocating buffers of user-entered, and especially of network-input, size is a receipt for disaster. An obvious problem is a denial-of-service attack when client requests a huge buffer and holds on to it - say sends data very slowly - and prevents other allocations and/or slows the whole system.

Common wisdom here is accepting a fixed amount of data at a time and parsing as you go. That of course affects your application-level protocol design.

EDITED

The best approach for this problem found by me, after a long analysis was the following:

• First, you need to set the buffer size in order to receive data from the server/client.

• Second, you need to find the upload/download speed for that connection.

• Third, you need to calculate how many seconds should the connection timeout last in accordance with the size of package to be sent or received.

Set the buffer size

The buffer size can be set in two ways, arbitrary or objectively. If the information to be received is text based, it is not large and it does not require character comparison, than an arbitrary pre-set buffer size is optimal. If the information to be received needs to be processed character by character, and/or large, an objective buffer size is optimal choice

// In this example I used a Socket wrapped inside a NetworkStream for simplicity
// stability, and asynchronous operability purposes.

// This can be done by doing this:
//
//  For server:
//
//  Socket server= new Socket();
//  server.ReceiveBufferSize = 18000;
//  IPEndPoint iPEndPoint = new IPEndPoint(IPAddress.Any, port);
//  server.Bind(iPEndPoint);
//  server.Listen(3000);
// 
// 
//  NetworkStream ns = new NetworkStream(server);

//  For client:
//  
//  Socket client= new Socket();
//  client.Connect("127.0.0.1", 80);
//  
//  NetworkStream ns = new NetworkStream(client);

// In order to set an objective buffer size based on a file's size in order not to
// receive null characters as extra characters because the buffer is bigger than
// the file's size, or a corrupted file because the buffer is smaller than
// the file's size.

//   The TCP protocol follows the Sys, Ack and Syn-Ack paradigm,  
//   so within a TCP connection if the client or server began the 
//   connection by sending a message, the next message within its
//   connection must be read, and if the client or server began 
//   the connection by receiving a message, the next message must
//   be sent.

// [SENDER]

byte[] file = new byte[18032];

byte[] file_length = Encoding.UTF8.GetBytes(file.Length.ToString());

await Sender.WriteAsync(file_length, 0, file_length.Length);

byte[] receiver_response = new byte[1800];

await Sender.ReadAsync(receiver_response, 0, receiver_response.Length);

await Sender.WriteAsync(file, 0, file.Length);

// [SENDER]

// [RECEIVER]

byte[] file_length = new byte[1800];

await Receiver.ReadAsync(file_length, 0, file_length.Length);

byte[] encoded_response = Encoding.UTF8.GetBytes("OK");

await Receiver.WriteAsync(encoded_response, 0, encoded_response.Length);

byte[] file = new byte[Convert.ToInt32(Encoding.UTF8.GetString(file_length))];

await Receiver.ReadAsync(file, 0, file.Length);

// [RECEIVER]

The buffers that are used to receive the payload length are using an arbitrary buffer size. The length of the payload to be sent is converted to string and then the string is converted in a UTF-8 encoded byte array. The received length of the payload is then converted back into a string format and then converted to an integer in order to set the length of the buffer that will receive the payload. The length is converted to string, then to int and then to byte[], in order to avoid data corruption due to the fact that the information related to the payload length will not be sent into a buffer that has the same size as the information. When the receiver will convert the byte[] content to a string and then to an int, the extra characters will be removed and the information will remain the same.

Get the upload/download speed of the connection and calculate the Socket receive and send buffer size

• First, Make a class that is responsible for calculating the buffer size for each connection.

// In this example I used a Socket wrapped inside a NetworkStream for simplicity
// stability, and asynchronous operability purposes.

// This can be done by doing this:
//
//  For server:
//
//  Socket server= new Socket();
//  server.ReceiveBufferSize = 18000;
//  IPEndPoint iPEndPoint = new IPEndPoint(IPAddress.Any, port);
//  server.Bind(iPEndPoint);
//  server.Listen(3000);
// 
//  NetworkStream ns = new NetworkStream(server);

//  For client:
//  
//  Socket client= new Socket();
//  client.Connect("127.0.0.1", 80);
//
//  NetworkStream ns = new NetworkStream(client);

    class Internet_Speed_Checker
    {
        public async Task<bool>> Optimum_Buffer_Size(System.Net.Sockets.NetworkStream socket)
        {
             System.Diagnostics.Stopwatch latency_counter = new System.Diagnostics.Stopwatch();

             byte[] test_payload = new byte[2048];

             //   The TCP protocol follows the Sys, Ack and Syn-Ack paradigm,  
             //   so within a TCP connection if the client or server began the 
             //   connection by sending a message, the next message within its
             //   connection must be read, and if the client or server began 
             //   the connection by receiving a message, the next message must
             //   be sent.
             //
             //   In order to test the connection, the client and server must 
             //   send and receive a package of the same size. If the client 
             //   or server began the connection by sending a message, the 
             //   client or server must do the this connection test by 
             //   initiating a write-read sequence, else it must do this
             //   connection test initiating a read-write sequence.

             latency_counter .Start();

             await client_secure_network_stream.ReadAsync(test_payload, 0, test_payload.Length);

             
             
             await client_secure_network_stream.WriteAsync(test_payload, 0, test_payload.Length);

             latency_counter .Stop();
             
             int bytes_per_second = (int)(test_payload.Length * (1000 / latency_time_counter.Elapsed.TotalMilliseconds));

             int optimal_connection_timeout = (Convert.ToInt32(payload_length) / download_bytes_per_second) * 1000 + 1000;

             double optimal_buffer_size_double = ((download_bytes_per_second / 125000) * (latency_time_counter.Elapsed.TotalMilliseconds / 1000)) * 1048576;

             int optimal_buffer_size = (int)download_optimal_buffer_size_double + 1024;
             
             // If you want to upload data to the client/server -->  client.SendBufferSize = optimal_buffer_size;
             // client.SendTimeout = optimal_connection_timeout;
             
             // If you want to download data from the client/server -->  client.ReceiveBufferSize = optimal_buffer_size;
             // client.ReceiveTimeout = optimal_connection_timeout;
        }
    }

The aforementioned method is ensuring that the data transmitted between the client buffer and server buffer uses an appropriate socket buffer size and socket connection timeout in order to avoid data corruption and fragmentation. When the data is sent through a socket with an async Read/Write operation, the length of the information to be sent will be segmented in packets. The packet size has a default value but it does not cover the fact that the upload/download speed of the connection is varying. In order to avoid data corruption and an optimal download/upload speed of the connection, the packet size must be set in accordance with the speed of the connection. In the aforementioned example I also showcased also the how to calculate the timeout in relation with the connection speed. The packet size for upload/download can be set by using the socket.ReceiveBufferSize = ... / socket.SendBufferSize = ... respectively.

For more information related to the equations and principles used check:

https://www.baeldung.com/cs/calculate-internet-speed-ping

https://docs.oracle.com/cd/E36784_01/html/E37476/gnkor.html#:~:text=You%20can%20calculate%20the%20correct,value%20of%20the%20connection%20latency.