EOFException thrown by a Hadoop pipes program

First of all, I am a newbie of Hadoop.

I have a small Hadoop pipes program that throws java.io.EOFException. The program takes as input a small text file and uses hadoop.pipes.java.recordreader and hadoop.pipes.java.recordwriter. The input is very simple like:

1 262144 42.8084 15.9157 4.1324 0开发者_运维问答.06 0.1

However, Hadoop will throw an EOFException, which I can't see the reason. Below is the stack trace:

10/12/08 23:04:04 INFO mapred.JobClient: Running job: job_201012081252_0016
10/12/08 23:04:05 INFO mapred.JobClient:  map 0% reduce 0%
10/12/08 23:04:16 INFO mapred.JobClient: Task Id : attempt_201012081252_0016_m_000000_0, Status : FAILED 
java.io.IOException: pipe child exception
    at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
    at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.EOFException
    at java.io.DataInputStream.readByte(DataInputStream.java:267)
    at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
    at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
    at org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)

BTW, I ran this on a fully-distributed mode (a cluster with 3 work nodes).

Any help is appreciated! Thanks

Lessons learned: by all means, try to make sure there is no bug in your own program.

This stacktrace is usually indicative of running out of available file descriptors within your worker machines. This is exceedingly common, documented sparsely, and precisely why I have two related questions on the subject.

If you have root access on all of the machines, you should consider raising the file descriptor limit for your Hadoop user by editing /etc/sysctl.conf:

(Add) fs.file-max = 4096

Or issuing:

ulimit -Sn 4096
ulimit -Hn 4096

Ad infinitum. General information for raising this limit is available here.

However, from the perspective of long-term planning, this strategy is somewhat spurious. If you happen to discover more information on the problem, perhaps you can help me help you help us all? [Thank you, GLaDOS. -Ed]

(Edit: See commentary that follows.)