.net app crashes when parallelizing a sequential operation - how is this possible?

Folks,

I'm iterating through 4k objects in an IEnumerable and calling a method to process each object. When I iterate sequentially, no problem. When I use Parallel.ForEach, the program crashes (windows shows "Program has stopped working"...). When I hookup WinDBG and run !analyze -v, I get the following:

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

GetPageUrlData failed, server returned HTTP status 404
URL requested: http://watson.microsoft.com/StageOne/DataUpdater_exe/1_1_1279_0/4d1e6b7d/ntdll_dll/6_1_7600_16559/4ba9b802/80000003/0004ef90.htm?Retriage=1

FAULTING_IP: 
+10ebb260
ffffffff`ffffffff ??              ???

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000000007745ef90 (ntdll!DbgBreakPoint)
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 0000000000000000

FAULTING_THREAD:  00000000000004e0

PROCESS_NAME:  DataUpdater.exe

OVERLAPPED_MODULE: Address regions for 'System_Data_ni' and 'certcli.dll' overlap

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

EXCEPTION_PARAMETER1:  0000000000000000

MOD_LIST: <ANALYSIS/>

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x4e0 (17)
Current frame: 
Child-SP         RetAddr          Caller, Callee

ADDITIONAL_DEBUG_TEXT:  Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]

LAST_CONTROL_TRANSFER:  from 0000000077508778 to 000000007745ef90

BUGCHECK_STR:  APPLICATION_FAULT_STACKIMMUNE_WRONG_SYMBOLS_FILL_PATTERN_ffffffff

PRIMARY_PROBLEM_CLASS:  STACKIMMUNE_FILL_PATTERN_ffffffff

DEFAULT_BUCKET_ID:  STACKIMMUNE_FILL_PATTERN_ffffffff

STACK_TEXT:  
00000000`00000000 00000000`00000000 dataupdater.exe+0x0


STACK_COMMAND:  .cxr 000000003F5B0000 ; kb ; ** Pseudo Context ** ; kb

SYMBOL_NAME:  dataupdater.exe

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: dataupdater

DEBUG_FLR_IMAGE_TIMESTAMP:  4d1e6b7d

BUCKET_ID:  X64_APPLICATION_FAULT_STACKIMMUNE_WRONG_SYMBOLS_FILL_PATTERN_ffffffff_dataupdater.exe

IMAGE_NAME:  C:\Users\WarGames\Documents\Applications\DataUpdater\DataUpdater.exe

FAILURE_BUCKET_ID:  STACKIMMUNE_FILL_PATTERN_ffffffff_80000003_C:_Users_WarGames_Documents_Applications_DataUpdater_DataUpdater.exe!Unknown

FOLLOWUP_IP: 
DataUpdater!get_xmlUpdater+0 [C:\TeamCity\buildAgent\work\57279efa3d42f599\trunk\Apps\DataUpdater\DataUpdater\ConfigHolder.vb @ 19]
00000000`00110000 4d5a            pop     r10

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/DataUpdater_exe/1_1_1279_0/4d1e6b7d/ntdll_dll/6_1_7600_16559/4ba9b802/80000003/0004ef90.htm?Retriage=1

Followup: MachineOwner
---------

and !mk (with sosex loaded)

Thread 17:
The current thread is unmanaged.

and !threads

ThreadCount:      11
UnstartedThread:  0
BackgroundThread: 8
PendingThread:    0
DeadThread:       2
Hosted Runtime:   no
                                           PreEmptive                                                   Lock
       ID  OSID        ThreadOBJ     State GC       GC Alloc Context                  Domain           Count APT Exception
   0    1   be4 0000000000724c30      6020 Enabled  0000000000000000:0000000000000000 00000000007184f0     0 STA
   2    2   44c 000000000072af70      b220 Enabled  0000000000000000:0000000000000000 00000000007184f0     0 MTA (Finalizer)
   7    4   b54 000000001c428170   a009220 Enabled  0000000000000000:0000000000000000 00000000007184f0     0 MTA (Threadpool Completion Port)
   8    9   69c 000000001bfc0c40   1009220 Enabled  0000000181327080:0000000181327098 00000000007184f0     0 MTA (Threadpool Worker) System.AccessViolationException (00000001812ca038)
   9    5   904 000000001bfc1350   100a220 Enabled  0000000000000000:0000000000000000 00000000007184f0     0 MTA (Threadpool Worker)
  13    b   278 000000001c4a9d70   1009220 Enabled  00000001814973c0:0000000181499098 00000000007184f0     0 MTA (Threadpool Worker)
  14    a   afc 000000001c4a8f50   1009220 Enabled  0000000000000000:0000000000000000 00000000007184f0     0 MTA (Threadpool Worker)
  15    7   2bc 000000001c4ab2a0   1009220 Enabled  00000001814e0da0:00000001814e1098 00000000007184f0     0 MTA (Threadpool Worker)
XXXX    c       000000001c4ab9b0   1019820 Enabled  0000000000000000:0000000000000000 00000000007184f0     0 MTA (Threadpool Worker)
XXXX    6       000000001c4aa480   1019820 Enabled  0000000000000000:0000000000000000 00000000007184f0     0 MTA (Threadpool Worker)
  16    8   8bc 000000001c452f50   1009220 Disabled 00000001817c3de8:00000001817c45c0 00000000007184f0     1 MTA (Threadpool Worker)

How is this possible? I've traced every line of code and there are no calls to unmanaged code. Does this have anything to do with the AccessViolation shown in the !threads command? How should I proceed? I'm completely lost.

EDIT Per Brian's suggestion I raw !pe, !clrstack. Here's the output:

0:019> !pe 0000000102b74288
Exception object: 0000000102b74288
Exception type:   System.AccessViolationException
Message:          Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
InnerException:   <none>
StackTrace (generated):
    SP               IP               Function
    000000001E86C300 000007FEF163703B System_Core_ni!System.Linq.Enumerable+WhereSelectEnumerableIterator`2[[System.__Canon, mscorlib],[System.Decimal, mscorlib]].MoveNext()+0x27c2eb
    000000001E86C360 000007FEF11D1661 System_Core_ni!System.Linq.Enumerable.Average(System.Collections.Generic.IEnumerable`1<System.Decimal>)+0x161
    000000001E86C470 000007FF00402F74 MyEngine!MyCompany.Libs.MyEngine.EndOfDayMessages.EndOfDayMessageBuilder.AddPortfolio(System.DateTime, System.DateTime, System.Collections.Generic.IEnumerable`1<MyCompany.Libs.MyEngine.Position>, MyCompany.Utility.WebHelpers.Proxy.LiveDataServicesClient)+0x10c4
    000000001E86C920 000007FF003F8C91 MyEngine!MyCompany.Libs.MyEngine.Simulation.MySimulation.RunSimulation()+0x3e1
    000000001E86C9E0 000007FF003F8815 MyEngine!MyCompany.Libs.MyEngine.Simulation.MySimulation.Run(System.DateTime)+0xc5
    000000001E86CA30 000007FF003F1D81 MyUpdater!MyCompany.MyUpdater.Modeling.Retrainer.SimulateTradingStrategy(MyCompany.ModelEngine.TradingStrategy, MyCompany.MyUpdater.Modeling.TradingStrategyParallelSimulationParams)+0x281
    000000001E86CB30 000007FEF2EB197E mscorlib_ni!System.Threading.Tasks.Parallel+<>c__DisplayClassf`1[[System.__Canon, mscorlib]].<ForWorker>b__c()+0x3ce
    000000001E86CC50 000007FEF2CDECB9 mscorlib_ni!System.Threading.Tasks.Task.InnerInvokeWithArg(System.Threading.Tasks.Task)+0x19
    000000001E86CC80 000007FEF2CE82E7 mscorlib_ni!System.Threading.Tasks.Task+<>c__DisplayClass7.<ExecuteSelfReplicating>b__6(System.Object)+0x197
    000000001E86CD60 000007FEF2CDD6A7 mscorlib_ni!System.Threading.Tasks.Task.Execute()+0x57
    000000001E86CDC0 000007FEF25B3179 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0x179
    000000001E86D610 0000000000000001 mscorlib_ni!System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object)+0x2
    000000001E86D610 000007FEF25A17E2 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0xa2
    000000001E86D670 000007FEF2CDD976 mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)+0x1b6
    000000001E86D6F0 000007FEF2CDEF31 mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry(Boolean)+0xb1
    000000001E86D730 000007FEF2CF71E4 mscorlib_ni!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean)+0xa4
    000000001E86D780 000007FEF2CE1DFB mscorlib_ni!System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean, System.Object)+0xbb
    000000001E86D800 000007FEF2CE0BE8 mscorlib_ni!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler)+0xe8
    000000001E86D870 000007FEF2CF2F99 mscorlib_ni!System.Threading.Tasks.Parallel.ForWorker[[System.__Canon, mscorlib]](Int32, Int32, System.Threading.Tasks.ParallelOptions, System.Action`1<Int32>, System.Action`2<Int32,System.Threading.Tasks.ParallelLoopState>, System.Func`4<Int32,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>)+0x729
    000000001E86DA50 000007FEF2CF15FD mscorlib_ni!System.Threading.Tasks.Parallel.ForEachWorker[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.__Canon>, System.Action`2<System.__Canon,System.Threading.Tasks.ParallelLoopState>, System.Action`3<System.__Canon,System.Threading.Tasks.ParallelLoopState,Int64>, System.Func`4<System.__Canon,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`5<System.__Canon,System.Threading.Tasks.ParallelLoopState,Int64,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>)+0x20d
    000000001E86DAF0 000007FEF2CF1D82 mscorlib_ni!System.Threading.Tasks.Parallel.ForEach[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.__Canon>)+0xb2
    000000001E86DB80 000007FF0016FF1B MyUpdater!MyCompany.MyUpdater.Modeling.Retrainer.SimulateTradesAndCreateJournal(Int32, System.String, MyCompany.Utility.WebHelpers.Proxy.LiveDataServicesClient, System.String)+0x7db
    000000001E86DCD0 000007FF0016C8D7 MyUpdater!MyCompany.MyUpdater.Modeling.Retrainer.TradingSimulation(Int32, Boolean, Boolean)+0x1d7
    000000001E86DD80 000007FF0016C463 MyUpdater!MyCompany.MyUpdater.Form1.TradingSimulation(Int32, Boolean, Boolean)+0x33
    000000001E86E650 0000000000000001 mscorlib_ni!System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr, System.Object[], System.Object, Int32, Boolean, System.Object[] ByRef)+0x2
    000000001E86E650 000007FEF2E34643 mscorlib_ni!System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(System.RuntimeMethodHandle, System.Object[], System.Object, Int32, Boolean, System.Object[] ByRef)+0x93
    000000001E86E6C0 000007FEF2E34D86 mscorlib_ni!System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessageSink)+0x2f6
    000000001E86E7B0 000007FEF25B3179 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0x179
    000000001E86F000 0000000000000001 mscorlib_ni!System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object)+0x2
    000000001E86F000 000007FEF25A17E2 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0xa2
    000000001E86F060 000007FEF25EC41A mscorlib_ni!System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()+0x6a
    000000001E86F0B0 000007FEF25EBA84 mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+开发者_如何学Python0x194
    000000001E86F150 000007FEF25EB8D6 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()+0x36

StackTraceString: <none>
HResult: 80004003
The current thread is unmanaged
0:019> ~8e!clrstack
OS Thread Id: 0x888 (8)
Child SP         IP               Call Site
000000001dfdf058 0000000077ae030a [HelperMethodFrame_1OBJ: 000000001dfdf058] System.Threading.WaitHandle.WaitMultiple(System.Threading.WaitHandle[], Int32, Boolean, Boolean)
000000001dfdf1a0 000007fef2cba946 System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle[], Int32, Boolean)
000000001dfdf200 000007feedaf7bb3 System.Runtime.IOThreadTimer+TimerManager.OnWaitCallback(System.Object)
000000001dfdf260 000007feedafdb81 System.Runtime.IOThreadScheduler+ScheduledOverlapped.IOCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
000000001dfdf2c0 000007feedaf9c80 System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
000000001dfdf320 000007fef2cc88e6 System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
000000001dfdf5c0 000007fef34f10b4 [GCFrame: 000000001dfdf5c0] 
000000001dfdf790 000007fef34f10b4 [DebuggerU2MCatchHandlerFrame: 000000001dfdf790] 
0:019> ~8e!kb
No export kb found

There are a couple of things you can do to investigate further. To look at the AccessViolationException, you can do a !pe 00000001812ca038, which will print the details of the exception. You may also want to look at the stack for that thread. Do a ~8e!clrstack to get the managed stack for that thread. To look at the native stack use ~8ekb.

A lot of managed code is built on top of system APIs so eventually your code will hit native code at some point. The CLR maps a couple of access violation exceptions to specific managed exceptions such as NullReferenceException. An AccessViolationException is an access violation that could not be mapped. There's a good chance that it comes from interacting with native code. The call stack of the exception should give you additional info.