Hadoop job fails when invoked by cron

I have created the following shell script for invoking a hadoop job:

#!/bin/bash
/opt/hadoop/bin/hadoop jar /path/to/job.jar com.do.something <param-1> ... <param-n> &
wait %1
STATUS=$?
if [ $STATUS -eq 0 ]
then    
    echo "SUCCESS" | mailx -s "Status: "$STATUS -r "mail@mysite.com" "mail@mysite.com"
    exit $STATUS
else
    echo "FAILED" | mailx -s "Status: "$STATUS -r "mail@mysite.com" "mail@mysite.com"
    exit $STATUS
fi

When I run the above script manually like this:

$ ./path/to/job.sh

Hadoop job executes sucessfully and returns an exit status "0".

Now, to automate the job execution 开发者_如何学JAVAeveryday, I have configured a cron job to run the above script like this:

0 22 * * * /path/to/job.sh

But, now job is not submitted to Hadoop and I get a exit status "1".

Few things to note here:

• The user account under which cron job is configured is UserA
• UserA is also Hadoop System User
• The cluster is dedicated for running this job
• The script is executable

I would like to know why the job is not running when cron invokes it ?

0 22 * * * /path/to/job.sh

I think you lost the "." in your command.

0 22 * * * ./path/to/job.sh

does it work?

the env of running from cron could be different from your regular shell. You may want to check that, e.g. JAVA_HOME, PATH etc.

I have also encountered a similar problem. I have used $HOME/.bashrc to set environment variables such as JAVA_HOME, HADOOP_HOME and PATH. I can also run my job.sh manually. But hadoop related commands inside job.sh can't be invoked correctly when job.sh is invoked by cron.

The cause for my problem is that cron will not source $HOME/.bashrc. So environment variables inside it can't be seen by cron. After setting all these environment variables in job.sh, hadoop related commands are invoked correctly by cron.

If you set JAVA_HOME and HADOOP_HOME in your /etc/profile. Add

. /etc/profile

in your job.sh. This will help.

after set JAVA_HOME,HADOOP_HOME in job.sh, then:

0 22 * * * sh -x $HOME/path/to/job.sh > $HOME/job.log 2>&1

see what in your job.log