OS X video memory from x86_64 assembly

I am working on a C++ app on Intel Mac OS X 10.6.x. I have a variable which contains pixel data which was obtained using OpenGL call glReadPixels. I want to do some operations directly on the pixel data using x86_64 assembly in开发者_高级运维structions. The assembly routine works fine in test programs but when I try to use it on the pixel data, it only gets zeroes in the memory location pointed by the pixel data variable. I am guessing this is since I am trying to access video memory directly from x86_64 assembly. Is there a way to access x86_64 video memory directly from assembly? Otherwise how can I resolve this situation?

Appreciate any pointers. Thanks in advance.

See below for code sample to flip last n and 1st n bytes. Same code works well in test program.

void Flip(void *b, unsigned int w, unsigned int h)
  {
    __asm {
    mov r8, rdi //rdi b
    mov r9, rsi //W
    mov r10,rdx //H
    mov r11, 0 // h <- 0
    mov r12, 0 // w<- 0
    outloop:
    ------------
    .............
    .............
  }

This isn't really an answer but the comments bit is too short to post this.

Your inline assembly is problematic, in multiple ways:

• it assumes by the time the compiler gets to the inline block, the function arguments are still in the arg registers. This isn't guaranteed.
• it uses MS-VC++ style inline assembly; I'm unsure about OSX Clang, but gcc proper refuses to even compile this.

It'd also be good to have a complete (compilable) source fragment. Something like (32bit code):

int mogrifyFramebufferContents(unsigned char *fb, int width, int height)
{
    int i, sum;
    glReadPixels(1, 1, width, height, GL_RGBA, GL_UNSIGNED_BYTE, fb);
    for (i = 0, sum = 0; i < 4 * width * height; i++)
        sum += fb[i];

    printf("sum over fb via C: %d\n", sum);

    asm("xorl     %0, %0\n"
        "xorl     %1, %1\n"
        "0:\n"
        "movsbl   (%2, %1), %ebx\n"
        "addl     %ebx, %0\n"
        "incl     %1\n"
        "cmpl     %1, %3\n"
        "jl       0b"
        : "=r"(sum)
        : "r"(i), "r"(fb), "r"(4 * width * height)
        : "cc", "%ebx");

    printf("sum over fb via inline asm: %d\n", sum);

    return (sum);
}

If I haven't made a one-off error, this code should result in the same output for C and assembly. Try something similar, right at the place where you access the read data within C, and compare assembly results - or even singlestep through the generated assembly with a debugger.

A stackoverflow search for "gcc inline assembly syntax" will give you a starting point for what the %0...%2 placeholders mean, and/or how the register assignment constraints like "=r"(sum) above work.