java - very fast writing to a file [duplicate]

This question already has answers here: Fastest way to 开发者_运维百科write huge data in text file Java (7 answers) Closed 6 years ago.

I get a fast stream of data (objects) and I would like to write it to a file.

This is a stand alone process so it doesn't do anything but read the data from a socket parse it to csv and write all to a file.

What is the best way to write a lot of csv lines to a file?

Is a buffer writing my solution?

Is there a buffered File object in Java ?

Should I manage it myself and use writeLines()?

---

Fastest way to write huge data in text file Java

---

If you're dealing with a huge throughput of data then I suggest you use a set of in-memory buffers where you deposit the data arriving and then have a thread/threadpool which uses Java NIO to "consume" these buffers and write them onto disk. You will however be limited by the disk writing speed -- bear in mind that it's not unusual for the speed of network to be faster than the speed of your hard disk! so you might want to consider a threadpool which writes in different physical locations and only "pastes" these files after all the data has been received and written.

---

As mentioned above, chances are that its disk I/O that limits you, not Java abstractions.

But beyond using a good lib to deal with CSV, you might consider using other (even more) efficient formats like JSON; as well as compression. GZIP is good at compressing things, but relatively slow; but there are faster ones too. For example, LZF (like this Java implementation) is fast enough to compress at speeds higher than typical disk I/O (and uncompress even faster). So compressing output may well increase throughput as well as reduce disk usage.