Tracking Hadoop job status via web interface? (Exposing Hadoop to internal clients in the company)

I want to develop a website that will allow开发者_JS百科 analysts within the company to run Hadoop jobs (choose from a set of defined jobs) and see their job's status\progress.

Is there an easy way to do this (get running jobs statuses etc.) via Ruby\Python? How do you expose your Hadoop cluster to internal clients on your company?

I have found one way to get information about jobs on JobTracker. This is the code:

    Configuration conf = new Configuration();
    conf.set("mapred.job.tracker", "URL");

    JobClient client = new JobClient(new JobConf(conf));

    JobStatus[] jobStatuses = client.getAllJobs();
    for (JobStatus jobStatus : jobStatuses) {

        long lastTaskEndTime = 0L;

        TaskReport[] mapReports = client.getMapTaskReports(jobStatus.getJobID());
        for (TaskReport r : mapReports) {
            if (lastTaskEndTime < r.getFinishTime()) {
                lastTaskEndTime = r.getFinishTime();
            }
        }

        TaskReport[] reduceReports = client.getReduceTaskReports(jobStatus.getJobID());
        for (TaskReport r : reduceReports) {
            if (lastTaskEndTime < r.getFinishTime()) {
                lastTaskEndTime = r.getFinishTime();
            }
        }
        client.getSetupTaskReports(jobStatus.getJobID());
        client.getCleanupTaskReports(jobStatus.getJobID());

        System.out.println("JobID: " + jobStatus.getJobID().toString() + 
                            ", username: " + jobStatus.getUsername() + 
                            ", startTime: " + jobStatus.getStartTime() + 
                            ", endTime: " + lastTaskEndTime + 
                            ", Durration: " + (lastTaskEndTime - jobStatus.getStartTime()));

    }

Since version 'beta 2' of Cloudera's Hadoop Distribution you can almost with no effort use Hadoop User Experience (HUE), which was earlier called Cloudera Desktop.

But since this version it has grown enormously. It comes with job designer,hive interface and many more. You should definitely check this out before deciding to build your own application.

Maybe a good place to start would be to take a look at Cloudera Destktop. It provides a web interface to enable cluster administration and job development tasks. Its free to download.

There is nothing like this that ships with hadoop. It should be trivial to build this functionality. Some of this is available via the JobTracker's page and some you will have to build yourself.