What is the best multi-part base 64 encoder in java?

I have tested different base64 encoders mig64,iHarder,sun etc. Seems like these need to have the whole data need to be in memory for conversion.

If I want to encod开发者_运维百科e a large file (stream) > 1gb in a multi-threaded fashion, which codec implementation can be used without corrupting the file? commons codec seems to have the base64outputstream wrapper. any other solutions?

To make it clear, I have a 1TB file, and this file need to be encoded base64. Machine memory 2GB RAM, what is the fastest way to do it in Java?

I'm not sure which encoder is faster offhand, you'll have to measure each to determine that. However you can avoid the memory problem and accomplish the concurrency by splitting the file into chunks. Just make sure you split them on some 6-byte boundary (since it evenly turns into 8 bytes in Base64).

I'd recommend picking a reasonable chunk size and using an ExecutorService to manage a fixed number of threads to do the processing. You can share a RandomAccessFile between them and write to the appropriate places. You'll of course have to calculate the output chunk offsets (just multiple by 8 and divide by 6).

Honestly though you might not realize much performance gain here with concurrency. It could just overwhelm the hard drive with random access. I'd start with chunking the file up using a single thread. See how fast that is first. You can probably crunch a 1GB file faster than you think. As a rough guess I'd say 1 minute on modern hardware, even writing to the same drive you're reading from.