activeMQ multiple consumer issue

We have two instances of activemq configured as a cluster and a cluster of four glassfish instances, that should be connected to ONE activemq at any given time. If activemq01 is unavailable, all four glassfish instances should failover on activemq02.

I've noticed several times, that for unknown reason, one of (random) glassfish instances will failover on activemq02, and remaining three instances will still be connected to activemq01, even though activemq01 w开发者_C百科as NOT down, and all glassfish instances should have been listening on activemq01.

Logs don't indicate anything that would explain this behavior, all I can see if that one glassfish instance couldn't connect to activemq01 and failedover on activemq02.

Have anybody experienced the same or similar issues? Any help/advise is greatly appreciated.

Here are my activemq configs :

activemq01 :

http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

file:${activemq.base}/conf/credentials.properties

<managementContext>
  <managementContext connectorPort="1093" createConnector="true"/>
</managementContext>

<networkConnectors>
  <networkConnector name="amq-prod" uri="static://(tcp://127.0.0.1:61616,tcp://192.168.0.167:61616)" />
</networkConnectors>


<persistenceAdapter>
  <kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>

<plugins>
  <loggingBrokerPlugin logAll="false" logConnectionEvents="true"/>
</plugins>


<systemUsage>
  <systemUsage>
    <memoryUsage>
      <memoryUsage limit="2048 mb"/>
    </memoryUsage>
    <storeUsage>
      <storeUsage limit="2 gb" name="prod"/>
    </storeUsage>
    <tempUsage>
      <tempUsage limit="2000 mb"/>
    </tempUsage>
  </systemUsage>
</systemUsage>

<transportConnectors>
  <transportConnector name="openwire" uri="tcp://0.0.0.0:61616" updateClusterClients="true" />
</transportConnectors>

activeMQ02:

http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

file:${activemq.base}/conf/credentials.properties

<managementContext>
  <managementContext connectorPort="1093" createConnector="true"/>
</managementContext>

<networkConnectors>
  <networkConnector name="amq-prod" uri="static://(tcp://127.0.0.1:61616,tcp://192.168.0.166:61616)"  />
</networkConnectors>


<persistenceAdapter>
  <kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>

<systemUsage>
  <systemUsage>
    <memoryUsage>
      <memoryUsage limit="2048 mb"/>
    </memoryUsage>
    <storeUsage>
      <storeUsage limit="2 gb" name="prod"/>
    </storeUsage>
    <tempUsage>
      <tempUsage limit="2000 mb"/>
    </tempUsage>
  </systemUsage>
</systemUsage>

<transportConnectors>
  <transportConnector name="openwire" uri="tcp://0.0.0.0:61616" updateClusterClients="true" />
</transportConnectors>

version of activeMQ - 5.4.1

Have your networkConnector tag defined like this:

acticemq01 config:

 <networkConnector
         name="amq1-nc" 
         uri="static:(failover:(tcp://192.168.0.167:61616))" 
         networkTTL="2" 
         duplex="true"
         dynamicOnly="true" 
  />

acticemq02 config:

 <networkConnector
         name="amq2-nc" 
         uri="static:(failover:(tcp://192.168.0.166:61616))" 
         networkTTL="2" 
         duplex="true"
         dynamicOnly="true" 
  />

And in your consumers use JMS provider URL like this:

failover:(tcp://192.168.0.166:61616,tcp://192.168.0.167:61616)?randomize=false&timeout=5000

Above failover URL will always connect to acticemq01 broker if it is available. If acticemq01 goes down it will automatically failover to acticemq02 broker. Also timeout=5000 will make sure that consumer throws back error in 5 sec in an attempt to connect to any of the 2 brokers.