CreateThread() fails on 64 bit Windows, works on 32 bit Windows. Why?

Operating System: Windows XP 64 bit, SP2.

I have an unusual problem. I am porting some code from 32 bit to 64 bit. The 32 bit code works just fine. But when I call CreateThread() for the 64 bit version the call fails. I have three places where this fails. 2 call CreateThread(). 1 calls beginthreadex() which calls CreateThread().

All three calls fail with error code 0x3E6, "Invalid access to memory location".

The problem is all the input parameters are correct.

HANDLE  h;
DWORD   threadID;

h = CreateThread(0,            // default security
                 0,            // default stack size
                 myThreadFunc, // valid function to call
                 myParam,      // my param
                 0,            // no flags, start thread immediately
                 &threadID);

All three calls to CreateThread() are made from a DLL I've injected into the target program at the start of the program execution (this is before the program has got to the start of main()/WinMain()). If I call CreateThread() from the target program (same params) via say a menu, it works. Same parameters etc. Bizarre.

If I pass NULL instead of &threadID, it still fails.

If I pass NULL as myParam, it still fails.

I'm not calling CreateThread from inside DllMain(), so that isn't the problem. I'm confused and searching on Google etc hasn't shown any relevant answers.

If anyone has seen this before or has any ideas, please let me know.

Thanks for reading.

ANSWER

Short answer: Stack Frames on x64 need to be 16 byte aligned.

Longer answer: After much banging my head against the debugger wall and posting responses to the various suggestions (all of which helped in someway, prodding me to try new directions) I started exploring what-ifs about what was on the stack prior to calling CreateThread(). This proved to be a red-herring but it did lead to the solution.

Adding extra data to the stack changes the stack frame alignment. Sooner or later one of the tests gets you to 16 byte stack frame alignment. At that point the code worked. So I retraced my steps and started putting NULL data onto the stack rather than what I thought was the correct values (I had been pushing return addresses to fake up a call frame). It still worked - so the data isn't 开发者_运维知识库important, it must be the actual stack addresses.

I quickly realised it was 16 byte alignment for the stack. Previously I was only aware of 8 byte alignment for data. This microsoft document explains all the alignment requirements.

If the stackframe is not 16 byte aligned on x64 the compiler may put large (8 byte or more) data on the wrong alignment boundaries when it pushes data onto the stack.

Hence the problem I faced - the hooking code was called with a stack that was not aligned on a 16 byte boundary.

Quick summary of alignment requirements, expressed as size : alignment

• 1 : 1
• 2 : 2
• 4 : 4
• 8 : 8
• 10 : 16
• 16 : 16

Anything larger than 8 bytes is aligned on the next power of 2 boundary.

I think Microsoft's error code is a bit misleading. The initial STATUS_DATATYPE_MISALIGNMENT could be expressed as a STATUS_STACK_MISALIGNMENT which would be more helpful. But then turning STATUS_DATATYPE_MISALIGNMENT into ERROR_NOACCESS - that actually disguises and misleads as to what the problem is. Very unhelpful.

Thank you to everyone that posted suggestions. Even if I disagreed with the suggestions, they prompted me to test in a wide variety of directions (including the ones I disagreed with).

Written a more detailed description of the problem of datatype misalignment here: 64 bit porting gotcha #1! x64 Datatype misalignment.

The only reason that 64bit would make a difference is that threading on 64bit requires 64bit aligned values. If threadID isn't 64bit aligned, you could cause this problem.

Ok, that idea's not it. Are you sure it's valid to call CreateThread before main/WinMain? It would explain why it works in a menu- because that's after main/WinMain.

In addition, I'd triple-check the lifetime of myParam. CreateThread returns (this I know from experience) long before the function you pass in is called.

Post the thread routine's code (or just a few lines).

It suddenly occurs to me: Are you sure that you're injecting your 64bit code into a 64bit process? Because if you had a 64bit CreateThread call and tried to inject that into a 32bit process running under WOW64, bad things could happen.

Starting to seriously run out of ideas. Does the compiler report any warnings?

Could the bug be due to a bug in the host program, rather than the DLL? There's some other code, such as loading a DLL if you used __declspec(import/export), that occurs before main/WinMain. If that DLLMain, for example, had a bug in it.

I ran into this issue today. And I checked every argument feed into _beginthread/CreateThread/NtCreateThread via rohitab's Windows API Monitor v2. Every argument is aligned properly (AFAIK).

So, where does STATUS_DATATYPE_MISALIGNMENT come from?

The first few lines of NtCreateThread validate parameters passed from user mode.

ProbeForReadSmallStructure (ThreadContext, sizeof (CONTEXT), CONTEXT_ALIGN);

for i386

#define CONTEXT_ALIGN   (sizeof(ULONG))

for amd64

#define STACK_ALIGN (16UI64)
...
#define CONTEXT_ALIGN STACK_ALIGN

On amd64, if the ThreadContext pointer is not aligned to 16 bytes, NtCreateThread will return STATUS_DATATYPE_MISALIGNMENT.

CreateThread (actually CreateRemoteThread) allocated ThreadContext from stack, and did nothing special to guarantee the alignment requirement is satisfied. Things will work smoothly if every piece of your code followed Microsoft x64 calling convention, which unfortunately not true for me.

PS: The same code may work on newer Windows (say Vista and newer). I didn't check though. I'm facing this issue on Windows Server 2003 R2 x64.

I'm in the business of using parallel threads under windows for calculations. No funny business, no dll-calls, and certainly no call-back's. The following works in 32 bits windows. I set up the stack for my calculation, well within the area reserved for my program. All releveant data about area's and start addresses is contained in a data structure that is passed to CreateThread as parameter 3. The address that is called contains a small assembler routine that uses this data stucture. Indeed this routine finds the address to return to on the stack, then the address of the data structure. There is no reason to go far into this. It just works and it calculates the number of primes below 2,000,000,000 just fine, in one thread, in two threads or in 20 threads.

Now CreateThread in 64 bits doesn't push the address of the data structure. That seems implausible so I show you the smoking gun, a dump of a debug session.

In the subwindow at the bottom right you see the stack, and there is merely the return address, amidst a sea of zeroes. The mechanism I use to fill in parameters is portable between 32 and 64 bits. No other call exhibits a difference between word-sizes. Moreover why would the code address work but not the data address?

The bottom line: one would expect that CreateThread passes the data parameter on the stack in the same way in 64 bits as in 32 bits, then does a subroutine call. At the assembler level it doesn't work that way. If there are any hidden requirements to e.g. RSP that are automatically fullfilled in C++ that would be very nasty.

P.S. No there are no 16 byte alignment problems. That lies ages behind me.

Try using _beginthread() or _beginthreadex() instead, you shouldn't be using CreateThread directly.

See this previous question.