How to get datanode timeout?

I have a 3 node hadoop setup, with replication factor as 2.

When one of my datanode dies, namenode waits for 10 mins before removing it from live nodes. Till then my hdfs writes 开发者_运维百科fail saying bad ack from node.

Is there a way to set a smaller timeout( like 1 min) so that the node where datanode dies is discarded immediately ?

Setting up the following in your hdfs-site.xml will give you 1-minute timeout.

<property>
 <name>heartbeat.recheck.interval</name>
 <value>15</value>
 <description>Determines datanode heartbeat interval in seconds</description>
</property>

If above doesn't work - try the following (seems to be version-dependent):

<property>
 <name>dfs.heartbeat.recheck.interval</name>
 <value>15</value>
 <description>Determines datanode heartbeat interval in seconds.</description>
</property>

Timeout equals to 2 * heartbeat.recheck.interval + 10 * heartbeat.interval. Default for heartbeat.interval is 3 seconds.

In the version of Hadoop that we use, dfs.heartbeat.recheck.interval should be specified in milliseconds (check the code/doc of your version of Hadoop, to validate that).

I've managed to make this work. I'm using Hadoop version 0.2.2.

Here's what I added to my hdfs-site.xml:

 <property>
    <name>dfs.heartbeat.interval</name>
    <value>2</value>
    <description>Determines datanode heartbeat interval in seconds.</description>
  </property>

  <property>
    <name>dfs.heartbeat.recheck.interval</name>
    <value>1</value>
    <description>Determines when machines are marked dead</description>
  </property>

This parameters can differ for other versions of Hadoop. Here's how to check that you're using the right parameters: Once you set them, start your master, and check the configuration at :

http://your_master_machine:19888/conf

If you don't find "dfs.heartbeat.interval" and/or "dfs.heartbeat.recheck.interval" in there, that means you should try using their version without the "dfs." prefix:

"heartbeat.interval" and "heartbeat.recheck.interval"

Finally, to check that the dead datanode is no longer used after the desired amount of time, kill a datanode, then check repeatedly the console at:

http://your_master_machine:50070

For me, with the configuration shown here, I can see that a dead datanode is removed after about 20 seconds.