Load an assembly without locking file AND keep the right Binding Context

I've comp up against a formidable conundrum. Here's my situation:

I'm building an application with a plugin framework. There's a base plugin class that all plugins must extend. Within the same same assembly I have a helper class that will serialize and deserialize classes. It's a generic开发者_开发百科 class and it's used all over the place. The structure is like this:

MyApp.dll
|_ App.cs
|_ HelperCollection.cs
|_ PluginBase.cs

MyPlugin.dll
|_MyPlugin.cs (this extends PluginBase)
|_Foo.cs

The Problem

My problem is the assembly loading and locking files. A requirement of the app is that Plugins can be overwritten at any time. If so, they need to be reloaded. It seems the best way to load an assembly so that it's not locked (that is, I can overwrite it or blow away while the app is still running) is this:

byte[] readAllBytes = File.ReadAllBytes("MyPlugin.dll");
Assembly assembly = Assembly.Load(readAllBytes);

Loading the plugin assembly works just fine, no problems there. I get an exception when, from within MyPlugin.cs, which is in the plugin assembly, I tried to use the HelperCollection to do deserialization. An example could be something like this:

// HelperCollection uses XmlSerializer under the covers
List<Foo> settingCollection = HelperCollection<Foo>.Deserialize("mysettings.xml");

It's blowing up and throwing an InvalidCastException saying that it's "Unable to cast object of type 'List[Foo]' to 'List[Foo]'". After much research I finally found why. It's getting loaded in the LoadNeither binding context.

When Foo is loaded (from MyPlugin.dll) it's in the LoadNeither binding context while the assembly containing the type for the type conversion (in my case, MyApp.dll) is loaded in Default context. So even though they have the same name they are not considered the same type. It's doing this because I'm using Assembly.Load(byte[]).

Questions

How can I get around this? How can I,

1. Load an assembly and not lock the file, and
2. Provide the right binding context so I can cast objects which are located in the loaded assembly.

Sorry for the wall of text, just wanted to get all the relevant info out there.

Have you tried shadow copying?

When you're using shadow copying, .NET copies the assembly to a temp directory and loads it from there (so it's the temp file that's locked, not the original assembly) -- but all the binding rules are based on the original file location.

Just an idea from the top of my head:

What about having a Plugin directory where the users may drop their Plugins into. Then in your application, you monitor that folder and copy the assemblies to the bin or whatever. Then you load them the "normal" way, which may lock them.

When the user drops new plugins, you kill the plugin app domain and reload them.

Just an idea.