Delphi TOpenDialog hangs in windows 2008 when run as remote desktop application

I have a Delphi 2010 exe that launches a second exe. In the second exe, there is a dialog that calls openDialog.execute. When this runs under Windows 2008 Enterprise R2 under a remote desktop, it runs as expected, but when run as a remote application, as soon as the file dialog pops up, the application hangs, turning all of the application windows white. The only way to get out of it is to terminate the application. I tried replacing TOpenDialog with TFileOpenDialog, the results are the same. I've looked into modifying the RDP file that launches the main application, but cannot see any parameters there that would make a difference. Has anyone ever seen this kind of behavior before?

2010.07.13 Updated

This is reproducable using a simple example. There are two executable files in the example. The first is a file launcher, called m_module.exe, which contains one edit, one button, and the code below. I change the name of the executable file in the edit to match the second executable before I click the launch button:

procedure TForm1.Button1Click(Sender: TObject);
begin
     ShellExecute(Handle, 'open', stringToOLEstr(edit1.text) , nil, nil, SW_SHOWNORMAL) ; 
end;

procedure TForm1.FormShow(Sender: TObject);
begin
     edit1.text:=application.exename;
end;

The second executable contains a button and the code below:

procedure TForm1.Button1Click(Sender: TObject);
begin
     OpenDialog1.execute;
end;

The first module is launched from an RDP file.

2010.07.14 Updated

I have discovered that if I copy the following dlls:

thumbcache.dll 
dtsh.dll 
wkscli.dll 

from the \Windows\System32 folder into the application folder, the problem is eliminated.

I've further discovered that changing ownership and permission levels of these dlls in the \Windows\System32 folder from TrustedInstaller to the Administrator's group has the same result (Copying them to the application directory is changing ownership and permission I think)

To confirm this, I verified that the errors reappeared if I changed the ownership and permission levels back to TrustedInstaller away from the Administrator's group.

So it appears that this is an access issue of some kind. Perhaps this will help in discovering the cause of the issue.

2010.07.18 Updated

Some additional information that might be helpful (provided by Embarcadero):

This MSDN article for GetWindowsDirectory http://msdn.microsoft.com/en-us/library/ms724454%28VS.85%29.aspx documents some interesting behavior of applications running under Terminal Services. While GetWindowsDirectory is not being called directly the sandboxing of the Windows System directory per user could be causing some sort of problem. Perhaps one of the DLLs in the calling chain to GetOpenFileNameA is trying to reference the real DLL in the real System directory instead of the sandboxed one thus causing a rights violation. It is just speculation but it is worth investigati开发者_如何学Cng. If you were able to get the SysInternals Process Monitor or Process Explorer working on the server you should be able to see commdlg32 and the other DLLs in the stack trace being loaded.

All legacy applications (i.e. all applications not created for Terminal Services or Remote Desktop Services) run under an Application Compatibility Layer. See this MSDN article http://msdn.microsoft.com/en-us/library/cc834995%28VS.85%29.aspx . The IMAGE_DLLCHARACTERISTICS_TERMINAL_SERVER_AWARE flag is defined in Windows.PAS. For testing purposes you can add it to your application's PE header by adding Windows to your application's USES section and right under the USES section put:

{$SetPEOptFlags IMAGE_DLLCHARACTERISTICS_TERMINAL_SERVER_AWARE}

This will cause your application to bypass the compatibility layer. I am currently investigating if spawned processes (e.g. your second exe) retain all of the rights and settings of the application defined under RDS.

Windows reports AV (c0000005) in thumbcache.dll module.

I think that thumbcache.dll have something to do with building/caching thumbnails for files. Building thumbnails may mean using 3rd party extensions to Explorer, which may not behave nicely with RDP.

Try that on clean system. Use VMWare or similar virtual machine to setup test configuration.

P.S. See also this article: How to debug application’s hang? But I think that hang is just consequence of another problem in your case.

FWIW, we have a similar situation, but it's driven by a security need, and not a crash. When our app runs via Citrix, we are forbidden to ever show the regular windows "open" or "save as" dialogs. So we rolled our own. It's got a combo of drive letters (local drives only), folder selector (restricted to the approved drives), filename selector, and filename edit box.

For us, this gets around any active directory issues, and keeps security happy. And it keeps the users from trying to drop files into our filesystem or see things they shouldn't.

If they're not running in the sandbox, we show the regular windows file dialogs. A wrapper function allows us to call it from anywhere and leave the "sandboxed vs windows" decision in one spot.

I recommend you to use Process Explorer tool to view properties of your process. Check, which exactly DLLs are loaded in both cases (you can do it by selecting your process and opening lower pane in modules view).

You can also use Process Monitor tool to monitor process startup (again: in both cases) and see any references to DLLs in question.

You seems to have narrowed your problem to an access issue of some kind, so the following explanation might not help you. But there seems to exist a problem with popup windows on RemoteApp and I could imagine that it could lead (at least theoretically) to a similar problem, that's why I would like to mention it: http://social.technet.microsoft.com/Forums/en-US/winserverTS/thread/0a88919f-2d72-4340-abd7-fbe0e9545f25/

Apparently the Z-order of the windows isn't always correct when using RemoteApp. In your case TOpenDialog should be a modal popup window. Due to the bug, I could imagine that TOpenDialog could appear in the background. Your main window would remain in the foreground but would be disabled as TOpenDialog is modal. Windows might then not know how to redraw a disabled window and simply draw a white box.

We were having problems on the OpenDialog.Execute but only on one computer - and it seemed to be random I found that adding the exe the Windows DEP may resolve the problem we haven't had any issues since changing it

here is the link on how to change the windows DEP settings http://www.itechtalk.com/thread3591.html

this is a workaround - if anyone knows how to keep the DEP happy please add a comment below

It the Z-order is incorrect (which I often see in Citrix, without having a proper fix to it) you would still be able to close the form with ctrl-F4 or alt-f4. Furthermore the application would not be "not responding". Sometimes the order will correct itself when switching between tasks