Sitecore only publishing to one of two servers

I'm having problems with Sitecore (6.1.0) only publishing an item to one of our two web servers.

Unfortunately I'm still pretty new to Sitecore and still haven't done the training (although it's suppose to happen soon!) so my knowledge of how Sitecore works is limited (I'm also new to this project so not familiar with how it's been setup yet). Please bear with me as I'm sure I've missed out checking obvious things or left out vital information in my question.

Basically what happened was that a client tried to publish an item but couldn't see it on the live site (they tried to publish a开发者_StackOverflowt least twice). I checked in the web database and it was there, then logged on to the both webservers where I could see that it worked on one but not the other. I later published again, and I could then see it on both, but I'm not sure if it might have been a coincidence. As far as I know this hasn't happened before, but I can't be sure. There's been no error messages, and I guess it's possible that it's happened before but they client hasn't noticed as they have gotten the server it worked on.

I'd like to understand why this happened, and obviously try to prevent it from happening in the future! If anyone has any ideas I'd be very grateful.

I checked the logs and this is the entry from my first publish:

4764 11:15:07 INFO  AUDIT (sitecore\admin): Publish, languages:en, targets:Internet, databases:web, incremental:false, smart:true, republish:false, children:true
ManagedPoolThread #16 11:15:08 INFO  Job started: Publish
ManagedPoolThread #1 11:15:08 INFO  Job started: Publish to 'web'
ManagedPoolThread #1 11:15:08 INFO  HtmlCacheClearer clearing HTML caches for all sites (1).
ManagedPoolThread #1 11:15:08 INFO  HtmlCacheClearer done.
ManagedPoolThread #1 11:15:08 INFO  Staging server found: Web1
ManagedPoolThread #1 11:15:08 INFO  SourceDatabase: master
ManagedPoolThread #1 11:15:08 INFO  TargetDatabase: web
ManagedPoolThread #1 11:15:08 INFO  Staging server found: Web2
ManagedPoolThread #1 11:15:08 INFO  SourceDatabase: master
ManagedPoolThread #1 11:15:08 INFO  TargetDatabase: web
ManagedPoolThread #1 11:15:08 INFO  Job ended: Publish to 'web' (units processed: 2)
ManagedPoolThread #16 11:15:08 INFO  Job ended: Publish
ManagedPoolThread #14 11:15:10 INFO  Job started: Sitecore.Modules.Staging.StagingAgent
ManagedPoolThread #17 11:15:10 INFO  Starting update of index for the database 'master' (2 pending).
ManagedPoolThread #14 11:15:10 INFO  Partial cache cleared on Web1
ManagedPoolThread #14 11:15:10 INFO  Staging on server Web1 finished
ManagedPoolThread #17 11:15:10 INFO  Update of index for the database 'master' done.
ManagedPoolThread #14 11:15:10 INFO  Partial cache cleared on Web2
ManagedPoolThread #14 11:15:10 INFO  Staging on server Web2 finished
ManagedPoolThread #14 11:15:10 INFO  Job ended: Sitecore.Modules.Staging.StagingAgent

I've also compared the web.configs on Web1 and Web2, and they are identical. Not sure what to check next.

UPDATE: This is from the log files on the web servers.

The one that didn't show the page:

2180 11:15:01 WARN  Memory usage exceeded the MemoryMonitor threshold.
2180 11:15:01 WARN  Memory usage: 2,058,067,968
2180 11:15:01 WARN  Number of suppressed logs due to the minimum time between log entries: 5
4624 11:15:10 INFO  DEBUG: STAGING History storage entries grouped: 0
4624 11:15:10 INFO  DEBUG: STAGING History storage entries grouped: 0
2180 11:16:11 WARN  Memory usage exceeded the MemoryMonitor threshold.
2180 11:16:11 WARN  Memory usage: 2,060,402,688
2180 11:16:11 WARN  Number of suppressed logs due to the minimum time between log entries: 6
4968 11:16:37 INFO  HttpModule is being initialized

The one that showed the page:

776 11:15:08 ERROR Announcement Error
Exception: System.IndexOutOfRangeException
Message: Index was outside the bounds of the array.
Source: 

 776 11:15:10 INFO  DEBUG: STAGING History storage entries grouped: 0
 776 11:15:10 INFO  DEBUG: STAGING History storage entries grouped: 0
3308 11:15:35 WARN  Memory usage exceeded the MemoryMonitor threshold.
3308 11:15:35 WARN  Memory usage: 2,632,605,696
3308 11:15:35 WARN  Number of suppressed logs due to the minimum time between log entries: 6
3308 11:15:45 INFO  Health.PrivateBytes: 2,632,630,272
3308 11:15:45 INFO  Health.CacheInstances: 654
3308 11:15:45 INFO  Health.CacheTotalCount: 332,491
3308 11:15:45 INFO  Health.CacheTotalSize: 417,760,120
3308 11:15:45 INFO  Health.Counter('Process\Private Bytes'): 2,632,630,272
3308 11:15:45 INFO  Health.Counter('Process\Virtual Bytes'): 5,925,634,048
3308 11:15:45 INFO  Health.Counter('Process\Page File Bytes'): 2,632,630,272
3308 11:15:45 INFO  Health.Counter('.net CLR Memory\# Bytes in all Heaps'): 1,568,948,208
3308 11:15:45 INFO  Health.Counter('.net CLR Memory\% Time in GC'): 1,080,607
3308 11:15:45 INFO  Health.Counter('.net CLR Memory\Large Object Heap size'): 343,910,104
3308 11:15:45 INFO  Health.Counter('.net CLR Loading\Bytes in Loader Heap'): 15,257,600
3308 11:15:45 INFO  Health.Counter('.net CLR Loading\Current Assemblies'): 107

Not sure if the exception has anything to do with it considering it was thrown on the server that did publish correctly? (The same exception has been throw several other times on both servers.)

Thanks,

Annelie

I recommend you review the documentation for the Staging module. Here it is for Sitecore 6.0-6.2 (PDF link)

I think one of the biggest issues I've experienced with the staging module is the correct configuration of the HistoryEngine. This helps Sitecore keep track of what publishes have gone through and will help with the cache clearing. In my case it was needed for Lucene index updates but its also useful here for the cache clearing mechanism.

Straight from the above linked doc, summarized:

On your content-management server and on the slave prod servers, make sure this is in the web.config under the live web database (e.g. webtarget in this example):

<database id="webtarget">
  <Engines.HistoryEngine.Storage>
    <obj type="Sitecore.Data.$(database).$(database)HistoryStorage, Sitecore.Kernel">
      <param connectionStringName="$(id)" />
      <EntryLifeTime>30.00:00:00</EntryLifeTime>
    </obj>
  </Engines.HistoryEngine.Storage>
</database>
...
<hooks>
  <hook type="Sitecore.Modules.Staging.InitializeEngines, Staging.Kernel" />
</hooks>

Can you try an AppPool recycle on the server that's not showing the updates. We've had similar things happen previously.

If you upgrade to Sitecore 6.3, I think the caching mechanism is improved, whereby they bypass the cache and look at a distribution table, to watch for item changes so they know when to flush the cache.

There could be any reasons why this is happening, if the data they a trying to retrieve is relying on searching functionality based on Lucene engine, then this could be indexing issue.

Make sure your indexes are automatically rebuilded when items are published. Double check web.config schedule timing, also verify your settings are correct on CD (Content Delivery) instances: Indexing.UpdateInterval Indexing.UpdateJobThrottle

Refer to ScalabilitySettings documentation available on sdn.sitecore.net. If you are still on Sitecore 6.1.0 - then it has a common issue when publishing to multiple CD instances, cache is not cleared automatically, even if you set handler

<event name="publish:end:remote">
    <handler type="Sitecore.Publishing.HtmlCacheClearer, Sitecore.Kernel" method="ClearCache">
      <sites hint="list">
        <site>website</site>
      </sites>
    </handler>
  </event>

To solve that problem, either update your version to 6.6.0 update 5 - the most stable or download Stager Module from shared sources. This will solve the issue with publishing and cache.