ListBox scrolls to end when adding new content

I have a listbox which is bound to a CollectionViewSource (backed by an ObservableCollection). When I add items to my backend collection, the listbox automatically scrolls them into view. How do I disable this behavior?

<ListBox ItemsSource="{Binding}"
                 ScrollViewer.VerticalScrollBarVisibility="Hidden"
                 ScrollViewer.HorizontalScrollBarVisibility="Visible"
                 HorizontalContentAlignment="Center"
                 VerticalContentAlignment="Stretch">
            <ListBox.ItemsPanel>
                <ItemsPanelTemplate>
                    <StackPanel Orientation="Horizontal" Margin="0"
                                CanVerticallyScroll="False"
                                CanHorizontallyScroll="True"
                                />
                </ItemsPanelTemplate>
            </ListBox.ItemsPanel>
        </ListBox>

EDIT: This is with .NET 4.0, Visual Studio 2010. This behavior does not occur when using AddRange, but the elements are expensive, so I add them one at a time using Dispatcher.Invoke in order to keep the UI responsive.

EDIT2: Because its been suggested three times, the reason I do not use a VirtualizingStackPanel is because it virtualizes "too hard". I find it performing more work than a regular stack panel, because using my method, all items are created once, and the UI is responsive. With the virtualizing panel, the app locks up when new items are scrolled into view as it pauses to create a few more. Furthermore, there are issues with items that aren't loaded not receiving events properly, and if one scrolls too far, earlier items appear to be discarded and are forced to 开发者_开发百科be created later. So I use a stack panel, where I avoid all of these issues, except for the initial load time, which I've almost managed to overcome save this one question.

Check to see what other elements are doing. In this case the elements I was adding (3rd party software) had a bug where one of the elements was calling RequestBringIntoView. As the UI couldn't update between all these calls, the appearance was that the listbox would appear to jump towards the end. Swallowing the event lead to correct behavior.