New worker role instances hang when dev-fabric Azure started by Visual Studio

I have created a new Azure project with a single worker role, the code of which is as produced by the Visual Studio Windows Azure Project wizard. When I start it from within Visual Studio (using Ctrl-F5 - not debugging) and then use csrun.exe to increase the number of instances, the new instances do not start properly. If I start the deployment using csrun.exe then everything works as expected.

DETAILS

The original instances start fine:

[fabric] Role Instance: deployment(237).Recon.WorkerRole1.0
[fabric] Role state Busy
[runtime] Role entrypoint . CALLING   OnStart()
[runtime] Role entrypoint . COMPLETED OnStart()
[runtime] Role entrypoint . CALLING   Run()
Information: WorkerRole1 entry point called
[fabric] Role state Started
Information: Working
Information: Working
....

To add instances I increase the number in ServiceConfiguration.cscfg and run

csrun /update:237;ServiceConfiguration.cscfg

and I can see a new instance with a green dot in the Windows Azure Compute Emulator, but the new instance produces this:

[fabric] Role Instance: deployment(237).Recon.WorkerRole1.3
[fabric] Role state Busy
[Diagnostics]: UpdateState(Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorStartupInfo, Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorConfiguration, )
[Diagnostics]: Acquired mutex
[Diagnostics] Information: C:\Users\oliver\AppData\Local\dftmp\s0\deployment(237)\res\deployment(237).Recon.WorkerRole1.3\directory\DiagnosticStore\Monitor
[Diagnostics] Information: C:\Recon\bin\Debug\Recon.csx\roles\WorkerRole1\diagnostics\x64\monitor\MonAgentHost.exe -LocalPath "C:\Users\oliver\AppData\Local\dftmp\s0\deployment(237)\res\deployment(237).Recon.WorkerRole1.3\directory\DiagnosticStore\Monitor" -StaticConfigFile "C:\Users\oliver\AppData\Local\dftmp\s0\deployment(237)\res\deployment(237).Recon.WorkerRole1.3\directory\DiagnosticStore\Monitor\Configuration\mastaticconfig.xml" -ConfigFile "C:\Users\oliver\AppData\Local\dftmp\s0\deployment(237)\res\deployment(237).Recon.WorkerRole1.3\directory\DiagnosticStore\Monitor\Configuration\maconfig.xml" -ShutDownEvent WADDM-ShutDown-f5dba0349f5d45769204950f21e9c6a5 -InitializedEvent WADM-StartUp-f5dba0349f5d45769204950f21e9c6a5 -parent 2328 -events
[Diagnostics]: Creating config channel server
[MonAgentHost] Output: Agent will exit when WADDM-ShutDown-f5dba0349f5d45769204950f21e9c6a5 is signaled.
[MonAgentHost] Output: Will signal WADM-StartUp-f5dba0349f5d45769204950f21e9c6a5 after the agent is initialized.
[MonAgentHost] Output: Registered as an event consumer.
[MonAgentHost] Output: Agent will exit when parent process 2328 exits.
[MonAgentHost] Output: Monitoring Agent Started
[Diagnostics]: Starting configuration channel polling

Note that role state never gets to "Started", and OnStart() is not called. So I saw Microsoft's instructions and realised that perhaps the new role is being started suspended, so I clicked the green arrow. Unfortunately I get this transitory error message, and nothing happens:

Failed to start service deploymentg deployment(237).  Start role failed for one or more roles.

I am using Visual Studio 2010 SP1 w/Azure SDK 1.4.1 and the Windows Azure Tools for Microsoft Visual Studio 2010 1.3 (the latest, AFAIK). I have reinstalled all the Azure bits to no avail.

(After a while I get this extra output, which seems to be the diagnostic stuff restarting itself. I don't know why. I think I see this stuff regularly enough with other projects.

[Diagnostics]: Checking for configuration updates 17/05/2011 2:18:40 PM.
[Diagnostics]: Signalling process restart on event = WADDM-ShutDown-f5dba0349f5d45769204950f21e9c6a5
[MonAgentHost] Output: Exiting the moni开发者_StackOverflow社区toring agent after the shutdown event was signaled.
[MonAgentHost] Output: Monitoring Agent Stopped
[Diagnostics] Information: Diagnostic process exited with code: 0
[Diagnostics] Information: C:\Users\oliver\AppData\Local\dftmp\s0\deployment(237)\res\deployment(237).Recon.WorkerRole1.3\directory\DiagnosticStore\Monitor
[Diagnostics] Information: C:\Recon\bin\Debug\Recon.csx\roles\WorkerRole1\diagnostics\x64\monitor\MonAgentHost.exe -LocalPath "C:\Users\oliver\AppData\Local\dftmp\s0\deployment(237)\res\deployment(237).Recon.WorkerRole1.3\directory\DiagnosticStore\Monitor" -StaticConfigFile "C:\Users\oliver\AppData\Local\dftmp\s0\deployment(237)\res\deployment(237).Recon.WorkerRole1.3\directory\DiagnosticStore\Monitor\Configuration\mastaticconfig.xml" -ConfigFile "C:\Users\oliver\AppData\Local\dftmp\s0\deployment(237)\res\deployment(237).Recon.WorkerRole1.3\directory\DiagnosticStore\Monitor\Configuration\maconfig.xml" -ShutDownEvent WADDM-ShutDown-96f71a44d4b0434b8da78f1ac6f802c3 -InitializedEvent WADM-StartUp-96f71a44d4b0434b8da78f1ac6f802c3 -parent 2328 -events
[MonAgentHost] Output: Agent will exit when WADDM-ShutDown-96f71a44d4b0434b8da78f1ac6f802c3 is signaled.
[MonAgentHost] Output: Will signal WADM-StartUp-96f71a44d4b0434b8da78f1ac6f802c3 after the agent is initialized.
[MonAgentHost] Output: Registered as an event consumer.
[MonAgentHost] Output: Agent will exit when parent process 2328 exits.
[MonAgentHost] Output: Monitoring Agent Started
[Diagnostics]: Starting configuration channel polling
[Diagnostics]: Checking for configuration updates 17/05/2011 2:19:42 PM.

I have seen this when your temp path where Azure puts the sites is too long. You might want to check your EventViewer to see what is going on. You will see more info in the EventViewer than in the Compute Emulator. If you are running into the path too long problem you can set the _CSRUN_STATE_DIRECTORY environment variable to a path like c:\Azure. You can see more info here on that. You can also check the troubleshooting area for Windows Azure here.

I was able to solve my problem with the Azure roles hanging by using this post from 'Just another .NET guy'.

The key points of the article being:

• Check your EventViewer for Azure related exceptions. Here you might see something like Provider Microsoft-Windows-Application Server-System Services IIS Manager{{SOME-GUID}} is missing channels under the channelreferances registry key. (typo is also present :P )
• Navigate to [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Publisher] and find the folder with the same GUID as above.
• Check to see if your development machine has the same values as your worker role on your remote machine.

After following those steps, I found that I had 1, while the other machine had 3. Once I corrected my registry, everything worked like normal.