Tools for finding solr performance issues

I have a solr server, which runs into performance issue at random. The queries which result in longer qtimes, dont take the same time if I try tho开发者_C百科se once again. The performance issue even arises sometimes when the load on server is not high.

So I wanted to know if there are any tools available or are there any options in solr available to pin-point the problem leading to performance issues.

some of the parameters in solrconfig.xml used are -

<filterCache class="solr.LRUCache" size="1024" initialSize="512" autowarmCount="1000"/>
<queryResultCache class="solr.LRUCache" size="512" initialSize="128" autowarmCount="0"/>
<documentCache class="solr.LRUCache" size="1024" initialSize="512" autowarmCount="0"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<queryResultWindowSize>30</queryResultWindowSize>
<queryResultMaxDocsCached>100</queryResultMaxDocsCached>
<HashDocSet maxSize="10000" loadFactor="0.75"/>

I think you should start by reviewing Solr Performance Factors on the Solr Wiki. You can also use the tool SolrMeter to help with stress testing your changes.

If you are looking for long-term monitoring you may also look into tools/services like Scalable Performance Monitoring (SPM) from my company, Sematext or others. http://sematext.com/spm/index.html

This service will expose request rates, latency, all Solr cache info, JVM memory, GC, CPU, load, disk and network IO, etc.

We eat our own dog food and use this tool to monitor Solr performance of Solr instances behind search-lucene.com and search-hadoop.com/ and we use it regularly with our customers when we need to help them around Solr performance tuning.

If you don't want "proper" performance monitoring a la above, then you can use tools like vmstat, iostat, sar, top, jstack, etc. for troubleshooting Solr performance, assuming you run it under UNIX.

The 'sar' utility from the sysstat package is great for debugging disk IO issues. If you notice that any particular disk is getting more than 30% or so utilization consistently, you're probably IO bound.

Maybe this tool can help you: LucidGaze.

I have no experience with it but it looks promising. The people from Lucid also published some interesting articles which are really worth reading.

I did a stress testing of the application by firing concurrently multiple requests to the solr server(requests were done by subsequent wgets)

We started seeing the performance impact and narrowed down our problem to two causes:

• the cache size was too small, so there were too many evictions and inserts happening in cache.
• We were using solrversion1.3 which apparently uses blocking reads for the index files. Switching to solr 1.4 removed that problem as it uses java nio which uses non blocking reads of index files.

[also we tried putting the index files in ram and directly using it from there, which improved the performance(but obviously not scalable)]

I was experiecing SOLR memory leaks for over a year and had to recysle the server very five days as a workaround to the problem, eventually I found out that the monitoring tool RELIC was causing the problem, we removed the monitoring and the problem was resolved, so my advice is to be careful when using too many monittoring tools, it puts an overhead on performance.