Creating objects suddenly begins failing after they have been loaded in memory successfully

Behavior:
Application is loaded and being used as expected. Suddenly, a particular DLL can no longer be loaded. The error message is:

ActiveX component cannot create object.
In each case, the object had been created successfully many times before failure. All objects are marked for "retain in memory".

This error is cleared when the application pool is recycled. It may be hours or months before it is seen again.

Issue has happened within two hours of a refresh, as well as never happened in months of uptime.

Issue has happened with hundreds of simultaneous users (heavy usage) and also with 1-3 users.

While the issue is occurring, the process running that application pool cannot create the object that is failing. However it can create any other objects. Memory, CPU, and other resources all remain at normal usage. In addition, other processes (such as a stand-alone exe) can successfully create the object.

The first instance of the issue appeared in mid 2008. There have been less than fifty instances since then, despite a pool of hundreds of servers for it to occur on. All instances except one have failed on the same DLL.

DLL Failure Info:

most common - generic data structure implementing a b-tree, has no references other than to its interface. Code consists of arrays and one use of the vb6 Event functionality. The object has not been changed in any way since 2005.

one-time - interop to a .NET module. the failure is occurring when trying to create the interop object, not the .NET object. This object is updated a few times each year.

Application Environment:
IIS hosted application

VB6, classic ASP, some interop to minor .NET components

Windows Server 2003 / Windows Server 2008 (both have independently had the problem)

Attempts to Reproduce:

Using scripts (and real-life humans) to run the same end-user workflows that our logs reported the days before the issue occurred.

Using scripts to create/destroy suspected objects as fast as possible from multiple simultaneous sessions.

Wild speculation. No intentional success, but it does manifest randomly on the servers on its own.

Troubleshooting:

Code reviews

Test harnesses to investigate upper limits of object creation / destruction

Verification of ability to create object outside of the process experienci开发者_如何学Gong the issue

Monitoring of resources over time on servers under load
Review of IIS, error, and event logs to determine events leading up to issue

Questions:

Any ideas on how to reproduce the issue?

What could cause this behavior?

Ideas for bypassing the first two questions in favor of a fast solution?

The DLL isn't on a network drive is it? You can get "glitches" where the drive is not available momentarily that then means COM can't do what it needs and could then fail to notice the drive is available again.

I used Process Monitor to debug similar problem when accessing ADO/OLEDB stack. Turned out environment got corrupted at some point and ADO classes are registered with InprocServer32 being REG_EXPAND_SZ pointing to %CommonProgramFiles%\System\ado\msado15.dll or similar ot x64 OSes.

Also when you register an application with Restart Manager, on failure the process gets restarted by winlogon process whose environment is different than explorer's one and unfortunately is missing %CommonProgramFiles% -- ouch!

This seems like a random failure; some race condition.
Try VMWARE to record the state of the machine you run this dll on. When the error happens you can then replay the record and inspect the memory contents. That why you won't have to play try and catch the error. At least you will have a solid record of it.

While I can't provide a solution, try catching the error and retry loading the dll when this happens after a refresh to the environment.