How to create and read directories in Hadoop - Mapreduce Job working directory

I want to create a directory inside the working directory of a MapRedu开发者_运维百科ce job in Hadoop.

For example by using: File setupFolder = new File(setupFolderName); setupFolder.mkdirs();

in my mapper class to write some intermediate files in it. Is it the right way to do it?.

Also after completion of the job how will I access this directory again if I wish so?

Please advice.

If you are using java, you can override the setup method and open the file handler there ( and close it in cleanup ) . This handle will be available to all mappers.

I am assuming that you are not writing all the map output here but some debug/stats. With this handler you can read and write as it is show in this example ( http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample )

if you want to read the whole directory, check out this example https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api

remember that you will not be able to depend on the the order of data written to the files.

You can override setupReduce() in reducer class, use mkdirs() to create folder and use create() to create file for outputstream.

@Override
    protected void setupReduce(Context context) throws IOException {
        Configuration conf = context.getConfiguration();
        FileSystem fs = FileSystem.get(conf);
        fs.mkdirs(new Path("your_path_here"));
    }