java download multiple files using threads

I am trying to download multiple file开发者_JAVA技巧s that matches a pattern using threads. The pattern could match 1 or 5 or 10 files of diff sizes.

lets say for simplicity sake the actual code that would download the file is in downloadFile() method and fileNames is the list of filenames that match the pattern. How do I do this using threads. Each thread will download only one file. Is it advisable to create a new thread inside the for loop.

for (String name : fileNames){
    downloadFile(name, toPath);
}

You really want to use an ExecutorService instead of individual threads, it's much cleaner, likely more performant and will enable you to change things more easily later on (thread counts, thread names, etc.):

ExecutorService pool = Executors.newFixedThreadPool(10);
for (String name : fileNames) {
    pool.submit(new DownloadTask(name, toPath));
}
pool.shutdown();
pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
// all tasks have now finished (unless an exception is thrown above)

And somewhere else in your class define the actual work horse DownloadTask:

private static class DownloadTask implements Runnable {

    private String name;
    private final String toPath;

    public DownloadTask(String name, String toPath) {
        this.name = name;
        this.toPath = toPath;
    }

    @Override
    public void run() {
        // surround with try-catch if downloadFile() throws something
        downloadFile(name, toPath);
    }
}

The shutdown() method has a very confusing name, because it "will allow previously submitted tasks to execute before terminating". awaitTermination() declares an InterruptedException you need to handle.

Yes, you certainly could create a new thread inside the for-loop. Something like this:

List<Thread> threads = new ArrayList<Thread>();
for (String name : fileNames) {
  Thread t = new Thread() {
    @Override public void run() { downloadFile(name, toPath); }
  };
  t.start();
  threads.add(t);
}
for (Thread t : threads) {
  t.join();
}
// Now all files are downloaded.

You should also consider using an Executor, for example, in a thread pool created by Executors.newFixedThreadPool(int).

Yes you can create the Threads inline.

for (final String name : fileNames){
    new Thread() {
       public void run() {
           downloadFile(name, toPath);
       }
    }.start();
}

Use Executor, Try this.

ExecutorService  exec= Executors.newCachedThreadPool()
for (String name : fileNames){
 exec.submit(new Runnable()
 {
  public void run()
  {
    downloadFile(name, toPath);
  }
 });
}

If you want say three download running concurrently, you can use:

Executors.newFixedThreadPool(3)

All the Above mentioned approach creates Threads but the actual Concurreny is not achieved.

ExecutorService pool = Executors.newFixedThreadPool(5);
final File folder = new File("YOUR_FILES_PATH");
int l = folder.listFiles().length;
System.out.println("Total Files----"+folder.listFiles().length);
long timeStartFuture = Calendar.getInstance().getTimeInMillis();
    pool.execute(new DownloadFile(folder,0,l/2));
    pool.execute(new DownloadFile(folder,(l/2),l));
    pool.shutdown();
    try {
        pool.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
    } catch (InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

    long timeEndFuture = Calendar.getInstance().getTimeInMillis();
    long timeNeededFuture = timeEndFuture - timeStartFuture;
    System.out.println("Parallel calculated in " + timeNeededFuture + " ms");

The above program is used to achieve concurreny and please modify as per your requirement.