RejectedExecutionException when doing batched puts to an HBase table

I am trying to put rows into HBase (0.90.0) in batches of size ~ 1000 (rows) I have multiple producer threads writing data into a queue, and a single consumer thread which wakes up every couple of minutes, and writes everything that's in the queue to HBase as a batch. However, I am getting the following exception , and I'm not sure what this means.

Caused by: java.util.concurrent.RejectedExecutionException
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1760)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
    at java.util.concurrent.ThreadPoo开发者_开发百科lExecutor.execute(ThreadPoolExecutor.java:658)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1135)

Any help is greatly appreciated!

If you can, avoid doing Put or other operations on a HTable that has been close()'d in the past.

Normally there is no strong reason for doing htable.close() in the middle of your code, a htable.flushCommits() might be enough. Try reserving htable.close() as a finalizing step, when your code will not use HBase anymore.

This Jira goes deeper in details about this bug: https://issues.apache.org/jira/browse/HBASE-3692

So inspect your code, searching for calls to close() and either remove them or substitute it with flushCommits() if you need to. I've ran into the same problem you did, and removing the unnecessary htable.close() calls was enough to avoid the problem.

I have been chasing a very similar bug for months in our code using HBase 0.94.15 in CDH 4.7 and tonight I finally found the reason for the bug. It seems fixed in latest HBase, so a quick workaround would be to upgrade.

The problem was an improper implementation of HTablePool.PooledHTable.close(). Java's Closeable interface says calling close() more than once has no affect. However, if you follow the code, you'll see pooled table interfaces do not follow this rule and closing them a second time actually puts the wrapped table back in the pool again. At the time of the second close, the actual wrapped table might be in use in another thread. That can cause a connection to be closed while being used, or even another thread accessing the same wrapped table.

1. Thread A gets table T from the pool
2. Thread A closes table T
3. Thread B gets table T from the pool
4. Thread A closes table T again (and HTablePool.PooledHTable.close() doesn't complain)
5. Thread B starts using table T
6. Table T is marked as not being used and either:
   • Pool decides table T has not been used for long enough and can be really closed
   • Thread C gets table T from the pool and starts using it

The fix for us was to remove that second close() on the pooled table. No more RejectedExecutionException!

Latest HBase checks if the pooled table interface was already closed, and if so doesn't close the wrapped table again. It also throws an exception, which still doesn't conform to the Java Closeable interface, but that's another story.