Finding good ways to transfer big files among Java servers

There in the project is a major DB Server, and other servers installed at different places maintain their own local Databases. We have to allow every system to update its local database to any version on the major DB. All the servers runs Java environment.

There seems many ways to transfer files: simple web download page, web-service, FTP, Socket.. What do you think fits better for this situation?

The general procedure looks like:

1, Client sends a string of version to the Server.

2, Server generates a patch and sends back.

(Since the patch maybe too huge, it should be separated to parts)

And what I'm thinking 开发者_JS百科about is inside the procedure 2.

There are really a lot of choices.

1, I should at first count the rows to be fetched and calculate how many parts it should be divided.

And then I meet a choice, should I just send back the number of parts or a list of links to the Client, and the Client uses the links to request each part if the connection based on web request, Or I can push parts back one by one if servers are connected with socket, or I can do the following step:

2, I can fetch needed records from the Server Database part by part, each time get a limited rows, return as Java List, serialize the List to a file immediately in order to empty memory for the next part, so I get a folder of files. And I compress the files to a single one and send back, avoiding the Client to request for a second time.

And the patch files generated can be kept for other Clients when requesting the same version.

I think all the ways will work, but which do you think is better? Or you know a better way?

Let's share.

You can try use Netty for large file transfer from server to client.

On Netty home page there is pretty good example how to send big files via web.

Finally I'm here to talk about the way I chose.

I made the major server to off a action that verify the incoming request. And prepare data to be sent serialized and package them into a single zip file. Send the zip file back to the requesting machine in a InputStream as the response.

Then the requesting machine unzip the file and deserialize and used them.