Distributing an application server

I have an application server. At a high level, this application server has users and groups. Users are part of one or more groups, and the server keeps all users aware of the state of their groups and other users in their groups. There are three major functions:

• Updating and broadcasting meta-data relating to users and their groups; for example, a user logs in and the server updates this user's status and broadcasts it to all online users in this user's groups.
• Acting as a proxy between two or more users; the client takes advantage of peer-to-peer transfer, but in the case that two users are unable to directly connect to each other, the server will act as a proxy between them.
• Storing data for offline users; if a client needs to send some data to a user who isn't online, the server will store that data for a period of time and then send it when the user next comes online.

I'm trying to modify this application to allow it to be distributed across multiple servers, not necessarily all on the same local network. However, I have a requirement that backwards compatibility with old clients cannot be broken; essentially, the distribution needs to be transparent to the client.

The biggest problem I'm having is handling the case of a user connected to Server A making an update that needs to be broadcast to a user on Server B.

By extension, an even bigger problem is when a user on Server A needs the server to act as a proxy between them and a user on Server B.

My initial idea was to try to assign each user a preferred server, using some algorithm that takes which users they need to communicate with into account. This could reduce the number of users who may need to communicate with users on other servers.

However, this only minimizes how often users on different servers will need to communicate. I still have the problem of achieving the communication between users on different servers.

The o开发者_开发技巧nly solution I could come up with for this is having the servers connect to each other, when they need to deal with a user connected to a different server.

For example, if I'm connected to Server A and I need a proxy with another user connected to Server B, I would ask Server A for a proxy connection to this user. Server A would see that the other user is connected to Server B, so it would make a 'relay' connection to Server B. This connection would just forward my requests to Server B and the responses to me.

The problem with this is that it would increase bandwidth usage, which is already extremely high. Unfortunately, I don't see any other solution.

Are there any well known or better solutions to this problem? It doesn't seem like it's very common for a distributed system to have the requirement of communication between users on different servers.

I don't know how much flexibility you have in modifying the existing server. The way I did this a long time ago was to have all the servers keep a TCP connection open to each other. I used a UDP broadcast which told the other servers about each other and allowed them to connect to new servers and remove servers that stopped sending the broadcast.

Then everytime a user connects to a server that server Unicasts a TCP message to all the servers it is connected to, and all the servers keeps a list of users and what server they are on.

Then as you suggest if you get a message from one user to another user on another server you have to relay that to the other server. The servers really need to be on the same LAN for this to work well.

You can run the server to server communications in a thread, and actually simulate the user being on the same server.

However maintaining the user lists and sending messages is prone to race conditions (like a user drops off while you are relaying the message from one server to another etc).

Maintaining the server code was a nightmare and this is really not the most efficient way to implement scalable servers. But if you have to use the legacy server code base then you really do not have too many options.

If you can look into using a language that supports remote processes and nodes like Erlang.

An alternative might be to use a message queue system like RabbitMQ or ActiveMQ, and have the servers talk to each other through that. Those system are designed to be scalable, and usually work off a Publish/Subscribe mechanism.