Multiple Blocked threads on UnInvertedField.getUnInvertedField() & SegmentReader$CoreReaders.getTermsReader

We are upgrading from Solr 1.3 to Solr 1.4.1. While using Solr 1.3 , we were seeing multiple blocking active threads on "org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal() ".

To utilize the benefits of NIO, on upgrading to Solr 1.4.1, we see other type of multiple blocking threads on "org.apache.solr.request.UnInvertedField.getUnInvertedField() &

SegmentReader$CoreReaders.getTermsReader". Due to this, the QTimes shoots up from few hundreds to thousand of msec.. even going upto 30-40 secs for a single query.

• The multiple blocking threads show up after few thousands of queries.
• We do not have faceting and sorting on the same fields.
• Our facet fields are multivalued text fields, but no large text values are present.
• Index size - around 10 GB
• We have not specified any method for faceting in our schema.xml.
• Our field value cache settings are:

Can someone please tell us the why we are seeing these blocked threads ?

Also if they are related to our field value cache , then a cache of s开发者_JAVA技巧ize 175 will be filled up with very few initial queries and right after that we should see multiple blocking threads ?

What difference it will make if we have "facet.method = enum" ?

Is this all related to fieldValueCache or is there some other configuration which we need to set to avoid these blocking threads?

Thanks,

Rachita

Cache values example:

facetField1_27443 :

{field=facet1_27443,memSize=4214884,tindexSize=52,time=22,phase1=15,nTerms=4,bigTerms=0,termInstances=6,uses=1}

facetField1_70 :

{field=facetField1_70,memSize=4223310,tindexSize=308,time=28,phase1=21,nTerms=636,bigTerms=0,termInstances=14404,uses=1}

facetField2 : {field=facetField2,memSize=4262644,tindexSize=3156,time=273,phase1=267,nTerms=12188,bigTerms=0,termInstances=1255522,uses=7031}

Stack trace for "org.apache.solr.request.UnInvertedField.getUnInvertedField() - BLOCKED"

at org.apache.solr.request.UnInvertedField.getUnInvertedField (UnInvertedField.java:837) at org.apache.solr.request.SimpleFacets.getTermCounts (SimpleFacets.java:250) at org.apache.solr.request.SimpleFacets.getFacetFieldCounts (SimpleFacets.java:283) at org.apache.solr.request.SimpleFacets.getFacetCounts (SimpleFacets.java:166) at org.apache.solr.handler.component.FacetComponent.process (FacetComponent.java:72) at org.apache.solr.handler.component.SearchHandler.handleRequestBody (SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest (RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute (SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute (SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter (SolrDispatchFilter.java:241) at com.caucho.server.dispatch.FilterFilterChain.doFilter (FilterFilterChain.java:87) at com.caucho.server.webapp.WebAppFilterChain.doFilter (WebAppFilterChain.java:187) at com.caucho.server.dispatch.ServletInvocation.service (ServletInvocation.java:266) at com.caucho.server.http.HttpRequest.handleRequest (HttpRequest.java:270) at com.caucho.server.port.TcpConnection.run (TcpConnection.java:678) at com.caucho.util.ThreadPool$Item.runTasks (ThreadPool.java:721) at com.caucho.util.ThreadPool$Item.run (ThreadPool.java:643) at java.lang.Thread.run (Thread.java:595)

org.apache.lucene.index.SegmentReader$CoreReaders.getTermsReader() - BLOCKED

at org.apache.lucene.index.SegmentReader$CoreReaders.getTermsReader (SegmentReader.java:170) at org.apache.lucene.index.SegmentTermDocs. (SegmentTermDocs.java:52) at org.apache.lucene.index.SegmentReader.termDocs (SegmentReader.java:987) at org.apache.lucene.index.IndexReader.termDocs (IndexReader.java:1102) at org.apache.lucene.index.SegmentReader.termDocs (SegmentReader.java:981) at org.apache.solr.search.SolrIndexReader.termDocs (SolrIndexReader.java:320) at org.apache.solr.search.SolrIndexSearcher.getDocSetNC (SolrIndexSearcher.java:640) at org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet (SolrIndexSearcher.java:563) at org.apache.solr.search.SolrIndexSearcher.numDocs (SolrIndexSearcher.java:1422) at com.askme.solrenhancements.facet.ExtendedFacet.getCustomFacetCount (ExtendedFacet.java:132) at com.askme.solrenhancements.facet.ExtendedFacet.getCustomFacetCount (ExtendedFacet.java:92) at com.askme.solrenhancements.facet.ExtendedFacet.getFacetAdditionalInfo (ExtendedFacet.java:69) at com.askme.solrenhancements.facet.ExtendedFacet.getFacetInfo (ExtendedFacet.java:56) at com.askme.solrenhancements.facet.CustomFacetComponent.process (CustomFacetComponent.java:43) at org.apache.solr.handler.component.SearchHandler.handleRequestBody (SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest (RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute (SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute (SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter (SolrDispatchFilter.java:241) at com.caucho.server.dispatch.FilterFilterChain.doFilter (FilterFilterChain.java:87) at com.caucho.server.webapp.WebAppFilterChain.doFilter (WebAppFilterChain.java:187) at com.caucho.server.dispatch.ServletInvocation.service (ServletInvocation.java:266) at com.caucho.server.http.HttpRequest.handleRequest (HttpRequest.java:270) at com.caucho.server.port.TcpConnection.run (TcpConnection.java:678) at com.caucho.util.ThreadPool$Item.runTasks (ThreadPool.java:721) at com.caucho.util.ThreadPool$Item.run (ThreadPool.java:643) at java.lang.Thread.run (Thread.java:595)

I have met this problem in my test as well. I found the block near following code is interesting:

SimpleFSDirectory.readInternal

protected void readInternal(byte[] b, int offset, int len)
     throws IOException {
  synchronized (file) {
    long position = getFilePointer();
    if (position != file.position) {
      file.seek(position);
      file.position = position;
    }
    int total = 0;

    try {
      do {
        final int readLength;
        if (total + chunkSize > len) {
          readLength = len - total;
        } else {
          // LUCENE-1566 - work around JVM Bug by breaking very large reads into chunks
          readLength = chunkSize;
        }
        final int i = file.read(b, offset + total, readLength);
        if (i == -1) {
          throw new IOException("read past EOF");
        }
        file.position += i;
        total += i;
      } while (total < len);
    } catch (OutOfMemoryError e) {
      // propagate OOM up and add a hint for 32bit VM Users hitting the bug
      // with a large chunk size in the fast path.
      final OutOfMemoryError outOfMemoryError = new OutOfMemoryError(
          "OutOfMemoryError likely caused by the Sun VM Bug described in "
          + "https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize "
          + "with a a value smaller than the current chunks size (" + chunkSize + ")");
      outOfMemoryError.initCause(e);
      throw outOfMemoryError;
    }
  }
}

I guess your solr is built at windows as I did. You can use the "code patch" in the following post: Use NIO positional read to avoid synchronization in FSIndexInput