x86 asm crashing application

char program[3] = { 0x6a,  0x0a, 0xC3 }; //push 10; ret;
char (*p)(void) = program;
printf("char = %c \n", p());

This开发者_如何学Go code crashes. Why?

The following code works ok:

char program[3] = { 0x90,  0x90, 0xC3 }; //nop; nop; ret
void (*p)(void) = program;
p();

Because you need to cleanup your stack by popping from it. Read more on the ret instruction - it will load cs:ip with values from the stack!

Firstly, you should understand that you cannot just "do things" in assembly and expect it to work. There is something called an Application Binary Interface with specifies how programs across the operating system and indeed inside your code are to behave.

For example, in C on most x86-32 platforms, one common rule is that eax should contain the return value. There will also be a set of values that are pushed onto the stack (called the stack frame for a function). Another is that certain registers need to be preserved whilst others (sometimes called scratch registers) can be left with your garbage in them.

In short, if you violate these rules, things go wrong and the OS tidies up for you. It really depends not only on your processor but your operating system; even linux and windows on x64 are different, for example.

I should also add char program is not a program, it is a function inside an existing program.

Finally, nop, nop, ret does nothing. So you are fine to execute these instructions and return because you have in fact not done anything to the stack frame.

I highly recommend you learn about calling conventions. For 32-bit x86, a function that modifies the stack should look more like this (cdecl):

push ebp
mov ebp, esp ; or ENTER instruction for the push+mov

; Function code!

mov esp, ebp
pop ebp ; or LEAVE instruction for the mov+pop
ret

The mov esp, ebp ensures your stack returns to the way it was at the beginning of the function, which means the return address is on the stack for ret to load and jump to.

Local variables in the function are placed on the stack (sub esp, 0x4 would allocate space for one 32-bit variable, usable as [esp + 0]).

64-bit x86 (not Itanium) has its own calling convention. Then there's fastcall and such.

The Wikipedia Article on x86 Calling Conventions is worth taking a look at.

char program[] = {
    0x55,              /* push %ebp */
    0x89, 0xe5,        /* mov %esp, %ebp */
    0x6a, 0x0a,        /* push $0xa */
    0x8b, 0x04, 0x24,  /* mov (%esp), %eax */
    0x89, 0xec,        /* mov %ebp, %esp */
    0x5d,              /* pop %ebp */
    0xc3,              /* ret */
};

That'll work*. Your program crashes because you're messing up the stack layout.

* as long as program resides in an executable part of memory, which is not necessarily true, and you're running on x86-32 with the typical C calling convention

And aside from cleaning up the stack, based on the intent(expecting output of 10) of your program...

char program[3] = { 0x6a,  0x0a, 0xC3 }; //push 10; ret;
char (*p)(void) = program;
printf("char = %c \n", p())

..., you shouldn't push 10, return value from functions are stored in AX(16 bit), EAX if 32 bit

If passing parameter is intended, cleaning up the stack depends on calling convention of function, if it is __fastcall (or pascal), it is the invoked routine that must do the cleanup; if it is C calling convention (_cdecl), it is the caller who should clean up the stack