How does Microsoft Detours work and how do I use it to get a stack trace?

I am new to Microsoft Detours. I have installed it to trace the system calls a process makes. I run the following commands which I got from the web

syelogd.exe /q C:\Users\xxx\Desktop\log.txt 
withdll.exe /d:traceapi.dll C:\Program Files\Google\Google Talk\googletalk.exe

I get the log file. The problem is I don't fully understand what is happening here. How does detours work? How does it trace the system calls? Also I don't know how to read the output in log.txt. Here is one line开发者_如何学运维 in log.txt

20101221060413329 2912 50.60: traceapi: 001 GetCurrentThreadId()

Finally I want to get the stack trace of the process. How can I get that?

Detours lets you intercept any function. It places a jmp in the address that you specify creating a trampoline to your code. Finally, you call the old function if you want to do it. To use Detours you have to inject your code in the process you want to intercept.

To simplify this process you can use Deviare API Hook which does all the injection staff and you can use intercept applications from any programming language that supports COM technology, including .NET, Delphi, C++, Python, etc.. After downloading the package you will find some examples in it. There is a console named DeviareCSharpConsole that let you intercept any API of any process showing full stack trace information.

This is the way Deviare API Hook works but is what you need to do if you want to create an application that hooks another process:

An agent should be created in the target process to intercept the APIs you want. To intercept these APIs you can use Detours but you have to code IPC staff that is not included in that library.

If you need to write code inside the target process using Deviare API Hook you can use Deviare Custom Hooks. This feature lets you intercept APIs and handle processed parameters asynchronously.

Instead of detours (which is free for 32-bit only) or easyhook (which is, khm, a little bit messy code) you may want to check out mhook 2.4 which is very neat code and BSD-licensed. Works on x86 and x64, handles IP-relative code, etc.

There's also a thorough description on how it works at the site.

As for the stack backtrace, you can use CaptureStackBackTrace() from kernel32, or if you want to get fancy, use StackWalk64() from dbghelp.

First of all, I would HIGHLY advise, that if you want to perform API hooking, I would go with easyhook: http://easyhook.codeplex.com/ (open source). It is a VERY good and easy api-hooking framework.

About how to get the stack trace, I don't remember exactly how to do it, but check out WinAPIOverride32: http://jacquelin.potier.free.fr/winapioverride32/ (open source). He's doing exactly that, and it is open source. Besides, if you need the traces for research, WinAPIOverride32 is a great application to use in order to study how applications work.

EDIT: Just adding one more application. http://www.rohitab.com/ is like WinAPIOVerride32, but it supports 64bit and it really improved since I wrote this answer. I must point out that it in some cases it missed API calls that I found in WinAPIOverride32, but its still pretty good. Unfortunately the source is not published.

About how api-hooking works, Well its a long explanation, I would point you to this article: http://www.codeproject.com/KB/system/hooksys.aspx It gives a pretty good explanation of how it is done under the hood (there are other methods besides what is written there, but still, it is a very good article).

Hope it helps! :-)

If you are allowed to use something other than Detours, you could install a debugger like WinDbg and attach it to the process to get a callstack.

You could also try other tools like Process Monitor and Windows Performance Toolkit as explained here.