C# Networking : Server hangs after receiving more than 65535 bytes

UPDATE: Due to problems with the admins here on Stackoverflow, I have posted a very trimmed-down version of the same problem on MSDN forum. This text below used MyNetworking.dll, but that is not the problem. Here is a very slimmed Client-Server thing and the problem is exactly the same. Feel free to try it out =) http://social.msdn.microsoft.com/Forums/en-US/netfxnetcom/thread/d3d33eb9-7dce-4313-929e-a8a63d0f1e03 /UPDATE

So, I have a strange error.

Normally, we have a DLL that handles our networking. Lets call that MyNetworking.dll. We use it everywhere in our servers and clients and have done so for 5 years. I haven't had a problem with it, until now.

I have an "XMLPoller", that reads XML from a MySQL database, serializes that into a byte[] array and sends it over the network. These particular XML messages is 627 bytes in serialized form.

The XMLPoller connects to a port on a "remote server" (that happens to be localhost) and sends the packets, one at a time. At exactly packet nbr 105 the connection is closed. 104 packets are sent from XMLPoller and received by the Server. 104 x 627 = 65208 bytes. But packet 105, when the total number of bytes sent would be 65835 the connection is closed with this error:

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
       at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
       at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

This is the error on the server. However, I have stepped through the XMLPoller (client), and I see when the last 627 bytes are sent (thus sending up til 65835 bytes) and I see no errors on the client, it passes sending without problems.

UPDATE 20:15 SWEDISH TIME

I also get this error in the Client when I debug a little more:

Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.

I think I have confirmed that it is in the Client the error exists. I am stepping through the code and before any Exceptions are caught on the server, I get an exception on the Client as stated above.

/ENDUPDATE

It seems to me that the Server never receives it, getting the error above. The server gets the connection closed because of something happening on the Client. However, the error on the client is in TCPInput; the stream reading data is dead for some reason?

I am not buffering anything in MyNetworking.dll.

When I get a new connection on a Socket (on the Server), I do this code:

public void setConnected(Socket thisClient)
{
    NetworkStream stream = new NetworkStream(thisClient);
 socket = thisClient;
 output = new TCPOutput(stream, outputHandler,this);
 remoteIP = this.socket.RemoteEndPoint.ToString();
 changeState(State.Connected);
    try
    {
        stream.BeginRead(inputBuffer, 0, 5000, new AsyncCallback(OnDataReceived), null);
    }
    catch (Exception e)
    {
        this.disconnect();
    }
}

and then, the OnDataReceived method (where the data is actually received):

public void OnDataReceived(IAsyncResult asyn)
        {
            int nbrRead = 0;
            byte[] tmp = null;
            try
            {
                nbrRead = stream.EndRead(asyn);
                tmp = new byte[nbrRead];
            }
            catch (Exception e)
            {
                // *** HERE IS WHERE THE EXCEPTION IS CAUGHT ***
                System.Diagnostics.Debugger.Log(0, "Bla1", e.ToString());
                this.disconnect();
            }
            if (nbrRead > 0)
            {
                try
                {
                    Array.Copy(inputBuffer, 0, tmp, 0, nbrRead);
                }
                catch(Exception e)
                {
                    System.Diagnostics.Debugger.Log(0, "Bla2", e.ToString());
                    this.disconnect();
                }
                preProcessMessage(tmp);
                try
                {
                    stream.BeginRead(inputBuffer, 0, 5000, new AsyncCallback(OnDataReceived), new object());
                }
                catch(Exception e)
                {
                    System.Diagnostics.Debugger.Log(0, "Bla3", e.ToString());
                    this.disconnect();
                }
            }
            else
                this.disconnect();

        }

Right now Im sort of clueless as to what is going on... Any ideas?

UPDATE 1:

Client code for sending data:

public bool sendData(byte[] data)
        {
            if(this.state == State.Connected)
            {
                if (data != null && data.Length > 0)
                {
                    try
                    {
                        //data = Crypto.Encrypt("a1s2d3", data);
                        outputStream.Write(data, 0, data.Length);
                    }
                    catch(Exception e)
                    {
                        System.Diagnostics.Debug.WriteLine("ClientHandler.sendData> " + e.ToString());
                    }
                    //parent.outDataLog(data.Length);
                }
            }
            return true;
        }

Update 2

I tried to do a Flush on the outgoing stream from the client - no effect:

public bool sendData(byte[] data)
{
    if(this.state == State.Connected)
    {
        if (data != null && data.Length > 0)
        {
            try
            {
                //data = Crypto.Encrypt("a1s2d3", data);
                outputStream.Write(data, 0, data.Length);
                outputStream.Flush();
            }
            catch(Exception e)
            {
                System.Diagnostics.Debug.WriteLine("ClientHandler.sendData> " + e.ToString());
            }
            //parent.outDataLog(data.Length);
        }
    }
    return true;
}

UPDATE 3: Posting more code as per request

This code is old and not the pretties in the world, I know. But it has been working very well for 5 years so =)

ClientHandler.cs (what the actual Client is using for sending etc)

using System;
using System.Net.Sockets;
using System.Net;
using System.Threading;

namespace tWorks.tNetworking.tNetworkingCF
{
    /// <summary>
    /// Summary description for connectionHandler.
    /// </summary>
    public class ClientHandler
    {
        #region Fields (17) 

        string address;
        Connector connector;
        DataHandler dataHandler;
        int id;
        TCPInput input;
        int interval;
        string localAddress;
        IPEndPoint localPoint;
        int localPort;
        NetworkStream outputStream;
        public TTCPClientInterface parent;
        int port;
        tWorks.tNetworking.Protocol.Protocol protocol;
        bool reconnect;
        string remoteIP;
        Socket socket;
        public State state;

        #endregion Fields 

        #region Enums (1) 

        public enum State {Disconnected,Connecting,Connected}

        #endregion Enums 

        #region Constructors (4) 

        public ClientHandler(int id, TTCPClientInterface parent, Socket socket, tWorks.tNetworking.Protocol.Protocol protocol)
        {
            this.id=id;
            this.parent = parent;
            this.protocol = protocol;
            dataHandler = new DataHandler(protocol, this);
            setConnected(socket);
        }

        public ClientHandler(int id, TTCPClientInterface parent, Protocol.Protocol protocol)
        {
            this.id=id;
            this.parent = parent;
            this.protocol = protocol;
            dataHandler = new DataHandler(protocol, this);
            state = State.Disconnected;
        }

        public ClientHandler(int id, TTCPClientInterface parent, Socket socket)
        {
            this.id=id;
            this.parent = parent;
            setConnected(socket);
        }

        public ClientHandler(int id, TTCPClientInterface parent)
        {
            this.id=id;
            this.parent = parent;
            this.protocol = null;
            changeState(State.Disconnected);
        }

        #endregion Constructors 

        #region Delegates and Events (4) 

        // Delegates (2) 

        public delegate void ConnectionLostDelegate(string message);
        public delegate void exceptionDelegate(Exception ex);
        // Events (2) 

        public event exceptionDelegate ConnectionFailed;

        public event ConnectionLostDelegate ConnectionLostEvent;

        #endregion Delegates and Events 

        #region Methods (17) 

        // Public Methods (16) 

        public void connect(string address, int port, int retryInterval, bool reestablish)
        {
            System.Random rand = new Random();
            localPort = rand.Next(40000, 60000);
            IPAddress localIP = Dns.GetHostEntry(Dns.GetHostName()).AddressList[0]; // new IPAddress(Dns.GetHostByName(Dns.GetHostName()).AddressList[0].Address);
            connect(address, port, retryInterval, reestablish, localIP.ToString(), localPort);            
        }

        /// <summary>
        /// Will connect to the address and port specified. If connection failed a new attempt will be made according to the Interval parameter. 
        /// If connection is lost attempts to reastablish it will be made if Reestablish is set to true.
        /// </summary>
        /// <param name="address"></param>
        /// <param name="port"></param>
        /// <param name="retryInterval"></param>
        /// <param name="reestablish"></param>
        public void connect(string address, int port, int retryInterval, bool reestablish, string localAddress, int localPort)
        {
            this.reconnect = reestablish;
            this.address = address;
            this.port = port;
            this.interval = retryInterval;
            this.localAddress = localAddress;
            this.localPort = localPort;
            changeState(State.Connecting);
            connector = new Connector(address, port, this, interval, localPoint, reestablish);
            connector.Connect();
        }

        public void disconnect()
        {
            reconnect = false;
            if (connector != null)
            {
                connector.stopConnecting();
            }
            setDisconnected();
        }

        public void dispose()
        {

        }

        public void failedConnect(Exception e)
        {
            if (ConnectionFailed != null)
                ConnectionFailed(e);
        }

        public int getID()
        {
            return this.id;
        }

        public string getIP()
        {
            return remoteIP;
        }

        public bool isConnected()
        {
            return this.state == State.Connected;
        }

        public void outDataLog(int nbrBytes)
        {
            parent.outDataLog(nbrBytes, id);
        }

        public void preProcessMessage(byte[] data)
        {
            //data = Crypto.Decrypt("a1s2d3", data);
            if(protocol != null)
                dataHandler.addData(data);
            else
                processMessage(data);
        }

        public void processMessage(byte[] data)
        {

            parent.processMessage(data,this);
        }

        public bool sendData(byte[] data)
        {
            if(this.state == State.Connected)
            {
                if (data != null && data.Length > 0)
                {
                    try
                    {
                        //data = Crypto.Encrypt("a1s2d3", data);
                        outputStream.Write(data, 0, data.Length);
                        outputStream.Flush();
                    }
                    catch(Exception e)
                    {
                        System.Diagnostics.Debug.WriteLine("ClientHandler.sendData> " + e.ToString());
                    }
                    //parent.outDataLog(data.Length);
                }
            }
            return true;
        }

        public void setConnected(Socket thisClient)
        {
            socket = thisClient;
            outputStream = new NetworkStream(thisClient);
            input = new TCPInput(outputStream, this);
            remoteIP = this.socket.RemoteEndPoint.ToString();
            changeState(State.Connected);
        }

        public void setDisconnected()
        {
            try
            {
                if (this.state == State.Connected)
                {
                    changeState(State.Disconnected);
                    //socket.Shutdown(SocketShutdown.Both);
                    socket.Close();
                }
            }
            catch { }
            if (reconnect)
                this.connect(address, port, interval, true, localAddress, localPort);
        }

        public void stopConnect()
        {
            connector.stopConnecting();
            changeState(State.Disconnected);
        }

        public override string ToString()
        {
            string returnString = "(D)";
            if(this.state == State.Connected)
                returnString = this.getIP();
            return returnString;
        }
        // Private Methods (1) 

        private void changeState(State state)
        {
            if (this.state == State.Connected && state == State.Disconnected)
            {
                if (ConnectionLostEvent != null)
                    ConnectionLostEvent("Uppkoppling bröts.");
            }
            this.state = state;
            parent.connStateChange(this);
        }

        #endregion Methods 
    }
}

This is TCPInput.cs that is listening on incoming data and forwarding that to the ClientHandler (seen above):

using System;
using System.Net.Sockets;
using System.Net;
using System.Threading;

namespace tWorks.tNetworking.tNetworkingCF
开发者_如何转开发{
    public class TCPInput
    {
        NetworkStream stream;
        ClientHandler client;

        public TCPInput(NetworkStream nS, ClientHandler client)
        {
            stream = nS;
            this.client = client;
            Thread t = new Thread(new ThreadStart(run));
            t.IsBackground = true;
            t.Name = "TCPInput";
            t.Start();
        }

        public void run()
        {
            bool continueRead = true;
            byte[] readBuffer = new byte[32768];
            byte[] receivedBuffer = null;

            int nbrBytesRead = 0;
            int receivedBufferPos = 0;
            while(continueRead)
            {
                try
                {
                    nbrBytesRead = 0;
                    nbrBytesRead = stream.Read(readBuffer, 0, 10000);
                    receivedBuffer = new byte[nbrBytesRead];
                }
                catch (Exception e)
                {
                    System.Diagnostics.Debug.WriteLine("TCPInput> Exception when stream.Read: " + e.ToString());
                    continueRead = false;
                }
                if(nbrBytesRead > 0)
                {
                    try
                    {
                        Array.Copy(readBuffer, 0, receivedBuffer, receivedBufferPos, nbrBytesRead);
                    }
                    catch (Exception e)
                    {
                        System.Diagnostics.Debug.WriteLine("TCPInput> Exception when Array.Copy: " + e.ToString());
                        continueRead = false; 
                    }
                    client.preProcessMessage(receivedBuffer);
                }
                else
                {
                                // *** I can break here, the nbrOfBytes read is 0 when this whole thing explodes =)
                    System.Diagnostics.Debug.WriteLine("TCPInput> Number of bytes read == 0! Setting continueRead = false");
                    continueRead = false;
                }

            }
            client.setDisconnected();
        }
    }
}

The problem is in your other code, the 'client'. It closes the connection after sending all the 'packets'. You must wait until the server has received all of them. A simple approach, beyond negotiating it explicitly, is to wait for the server to close the connection.

That number ("thus sending up til 65835 bytes") is magically close to 2^16-1 (65535) -- looks like just one packet over!

(I'm assuming it's just the larger size that made things go kaboom! -- this can be tested reliably.)

I suspect there is an unsigned 16-bit variable used (in the library) where you need something with more range. Perhaps you can "empty" the internals of the library periodically or perform the operation in multiple connection? (Okay, just trying to throw out some 'quick hack' ideas :-)

So, after much testing and discussing with my partner-in-crime we found out that instead of using port 21 and taking for example port 22 - the problem goes away.

I have no idea why it behaves like this, but it does...

You post raises questions for me. Like why are you choosing well known ports for this service? I don't believe in coincidences and suspect your use of the term "partner-in-crime" may have more truth then I would care to be associated with.

Then also I am wondering why you assume a Windows bug and not one in the MyNetowrking.dll. Sure, you have been using this for five years. But it still hasn't had the level of vetting that Microsoft gives their code.