How do I return a managed object from another AppDomain when using managed C++?

I'm using an unmanaged library written in C++. The library has a managed C++ (CLI) wrapper, and I'm using the library from managed code. The unmanaged library (including the CLI wrapper) is written by a third party but I have access to the source code.

Unfortunately the managed wrapper does not play well with AppDomains. The unmanaged library creates threads and will call into managed code from these threads. This leads to problems when the managed code is running in a non-default AppDomain. And I need cross-AppDomain calls to be able to unit test using standard tools.

To solve that I have introduced delegates in the managed wrapper and used Marshal.GetFunctionPointerForDelegate() to get a function pointer that allows cross-AppDomain calls to succeed.

This generally work well but now I have a new problem. I have the following sequence of events.

Unmanaged thread ->
Unmanaged code 1 ->
Managed wrapper 1 ->
AppDomain transition (via delegate) ->
Managed wrapper 2 ->
Unmanaged code 2 ->

I have left out some details on how the library allows you to override some functionality in managed code above managed wrapper 2 which is the whole point of doing the unmanaged to managed transition in the first place.

Eventually unmanaged code 2 will have to return an unmanaged object to unmanaged code 1.

Without the delegate and AppDomain stuff managed wrapper 2 will wrap the unmanaged object and return it to managed wrapper 1 that will then transfer the state to the unmanaged object used by unmanaged code 1.

Unfortunately I'm having a hard time returning a managed object across the AppDomain transition.

I figured that I had to make the managed object passed across the AppDomain boundary serializable. However, that is not easily done. Instead I've created a simple class where I can store the type of the object I want to tran开发者_StackOverflow中文版sfer and a string representing the state of the object. Both Type and String are easily marshalled and luckily I can always create an instance of the object using the default constructor and then initialize it from a string:

// Message is the base class of large hierarchy of managed classes.

[Serializable]
// Sorry for the "oldsyntax", but that is what the C++ library uses.
__gc class SerializedMessage {
public:
  SerializedMessage(Message* message)
    : _type(message->GetType()), _string(message->ToString()) { }

  Message* Create() {
    Message* message = static_cast<Message*>(Activator::CreateInstance(_type));
    message->InitializeFromString(_string);
    return message;
  }

private:
  Type* _type;
  String* _string;
};

In managed wrapper 2 I return a SerializedMessage and in managed wrapper 1 I then get back a copy of the original message by calling SerializedMessage::Create. Or at least that was what I was hoping to achieve.

Unfortunately the AppDomain transition fails with a InvalidCastException having the message Unable to cast object of type 'SerializedMessage' to type 'SerializedMessage'.

I'm not sure what is going on but the error message could indicate that the SerializedMessage object is being accessed from the wrong AppDomain. However, the whole point of using Marshal.GetFunctionPointerForDelegate is to be able to call across AppDomains.

I also tried to derive SerializedMessage from MarshalByRefObject but then I get and InvalidCastException having the message Unable to cast object of type 'System.MarshalByRefObject' to type 'SerializedMessage'.

What do I need to do to be able to pass a managed object back from another AppDomain when I call through a pointer returned by Marshal.GetFunctionPointerForDelegate()?

Comments on the accepted answer

The part about "how the assembly got loaded into the second AppDomain" is correct.

The C++ code executes in the default AppDomain which is controlled by the unit test runner. The managed assembly is loaded into a second AppDomain created by this unit test runner. I'm using Marshal.GetFunctionPointerForDelegate to enable managed C++ calls from the first to the second AppDomain.

Initially I got some FileNotFoundException's trying to load my managed assembly and to work around this I copied my managed assembly to the AppBase of the unit test runner. I'm still a bit confused why .NET insist on loading the very assembly that is executing, but decided to work on that problem later and simply copied the missing file as a kludge.

Unfortunately this loads the same type from two different copies of the same assembly in two different AppDomains and I assume this is the root cause of the InvalidCastException's.

My conclusion is that I'm unable to use a "standard" unit test runner to test the managed C++ library because the callbacks from this library is from the wrong AppDomain which I have no control over.

Unable to cast object of type 'SerializedMessage' to type 'SerializedMessage'.

I'd focus on this problem for the core of your issue. The real message should be "of type Foo.SerializedMessage to type Bar.SerializedMessage". In other words, there are two types involved here, from different assemblies. A .NET type's identity is not just the class name, it also includes the fully qualified assembly name. Which is the assembly display name and assembly version and culture. A DLL Hell counter-measure.

Check how your assemblies are built and verify that SerializedMessage only appears once in any of your assemblies. It can also be caused by the way the assembly got loaded into the second AppDomain, using LoadFile() will cause it for example. It loads assemblies without a loading context and any types loaded from such an assembly are not even compatible with the exact same type from the exact same assembly that was loaded normally.

And you'd better stay away from GetFunctionPointerForDelegate(), unmanaged pointers don't observe AppDomain boundaries. Albeit that I have no clue why you're using it.