Call/Ret in x86 assembly embedded in C++

This is probably trivial, but for some reason I can't it to work. Its supposed to be a simple function that changes the last byte of a dword to 'AA' (101010开发者_StackOverflow10), but nothing happens when I call the function. It just returns my original dword.

__declspec(naked) long
    function(unsigned long inputDWord, unsigned long *outputDWord)
    {
      _asm{
        mov ebx, dword ptr[esp+4]

  push ebx
  call SET_AA
  pop ebx

  mov eax, dword ptr[esp+8]
  mov dword ptr[eax], ebx
       }
}

__declspec(naked) unsigned long 

SET_AA( unsigned long inputDWord )

{

       __asm{
          mov eax, [esp+4]
                mov al, 0xAA //10101010 didn't work either 
                ret
             }
}

You seem to be confusing returning a value and having an out variable.

Here:

push ebx
call SET_AA
pop ebx

you are acting like ebx is an out variable.

and here:

mov eax, [esp+4]
mov al, 0xAA //10101010 didn't work either 
ret

you are simply writing stuff into eax twice (once with the parameter, then you overwrite it with your 0xAA). eax is traditionally the return value register. You need to pick which one you want to use.

If you want it to be an out variable you need to do something like this:

__declspec(naked) long function(unsigned long inputDWord, unsigned long *outputDWord) {
    _asm{
        mov ebx, dword ptr[esp+4]

        push ebx
        call SET_AA
        pop ebx

        mov eax, dword ptr[esp+8]
        mov dword ptr[eax], ebx
    }
}

__declspec(naked) void SET_AA( unsigned long inputDWord ) {
    __asm{
        mov [esp+4], 0xAA // put 0xAA into the variable passed on the stack
        ret
    }
}

If you want a return value, you can do something like this:

__declspec(naked) long function(unsigned long inputDWord, unsigned long *outputDWord) {
    _asm{
        mov ebx, dword ptr[esp+4]

        call SET_AA
        mov ebx, eax

        mov eax, dword ptr[esp+8]
        mov dword ptr[eax], ebx
    }
}

__declspec(naked) unsigned long SET_AA(/* input param not needed, we are just returning a value */) {
    __asm{
        mov eax, 0xAA // return 0xAA via the eax register
        ret
    }
}

I think this is more what you meant. One important thing: as MSDN says,

Naked functions must provide their own prolog...and epilog

Your SET_AA()is fine. It leaves the result in eax. (You can get away without the prolog/epilog because you're calling it from _asm, not C.)

__declspec(naked) unsigned long 
SET_AA(unsigned long inputDWord )
{
    __asm
    {
        mov eax, [esp+4]
        mov al, 0xAA
        ret               // final value is in eax
    }
}

function() should return void, since you want the result in *outputDWord. Also, you might as well use inputDWord instead of [esp+4]:

__declspec(naked) void
function(unsigned long inputDWord, unsigned long *outputDWord)
{
    _asm
    {
    // you need a prolog/epilog to make C happy
    // here's the prolog:
    push ebp
    mov ebp, esp

    mov ebx, inputDWord    // the value you're going to change
    mov ecx, outputDWord   // address of where to put the result

    push ebx
    call SET_AA // puts the result in eax
    pop ebx

    // copy the result to the thing ecx points to (*outputDWord)
    mov [ecx], eax

    // epilog to keep C happy
    pop ebp
    ret
    }
}

Your function SET_AA modifies the value in EAX (and in the register only), but after you call SET_AA, you move another value into EAX in your first function, thus overwriting the result from the SET_AA call. Therefore, changing your use of registers in such a way as not to overwrite EAX (as indicated in your answer) solves your problem.

I agree with what other users said.

1. SET_AA stores the return value in EAX register. However instead of returning it you return the parameter that you pass it (EBX).
2. Your function doesn't have a RET instruction at the end. Neither you manually implement the code the returns to the caller.

In addition I'd like to note that in function you overwrite the value of the EBX register without saving it (in the stack) and restoring at the end.

You don't specify which calling convention you assume for your function. (Since you don't even use RET or RET(8) instruction I can't guess what it had to be). However according to the most of the calling conventions I know, it's illegal to overwrite the EBX register without restoring it at the end.

Registers available for use by function (in most of the conventions) are EAX, ECX, EDX. All other registers may also be used at will, however they must be restored.

Why not just write the function in C++? It happily provides bit operations.

unless you're using __cdecl as your call convetion, both your functions are missing stack cleanups, this will cause problems when the returns unwind the stack frames, as well as the fact that the registers aren't preserved correctly.

using something a little more structured, clear and concise:

__declspec(naked) unsigned long __stdcall SET_AA(unsigned long inputDWord )
{
    __asm
    {
        mov eax, [esp+4]
        mov al, 0xAA
        retn 4
    }
}

__declspec(naked) void __fastcall function(unsigned long inputDWord, unsigned long *outputDWord)
{
    _asm
    {
    push ecx //push inputDWord
    call SET_AA // puts the result in eax

    // copy the result to the thing ecx points to (*outputDWord)
    mov [edx], eax
    retn//fastcall has no cleaup for the first 2 args
    }
}