How do function detour packages circumvent security

I am looking at some code that uses a function detour package called DetourXS. My application is targeted for Microsoft Server operating 开发者_Go百科systems. Microsoft Research also has a Detours package and they have an article on how it works. They patch the machine code that is loaded in memory and insert code that makes an unconditional jump into the newly injected code.

If this code works by modifying the machine code at run time, they should be facing security restrictions by the Operating System. This would be a serious security lapse on the OS as I can modify any critical DLL like kernel32 to do anything I want. My understanding is that if a user process attempts to modify the code of a dll already loaded into memory, it should be stopped by the OS. Is there a setting in Widows Server OS to enable/disable this check?

How do they overcome this?

Does anybody have experience on using this kind of detour packages in any application in enterprise production environment?

First important thing is that with detours you modify the instructions of your own process. In your process - you can do whatever you want anyways and you don't even have to detour anything, from OS point of view userspace code (e.g. code in your DLL and code in system's kernel32.dll loaded into your process) is exactly the same from security point of view. The reason is simple - hacking security of OS is not the same as changing some code of your process in userspace. OS secures itself by not giving your process too much power (unless you're running as an administrator). For security purposes it is more interesting how to modify code of another process (to achieve its privileges or interesting data, like passwords), but it is a subject for a different discussion.

Secondly, detouring had been considered as a way to implement hot-patching, i.e. patching critical system services on the fly, without reboot. I don't know whether it is currently used/supported by Microsoft (according to google it is), but it is not by chance that standard prolog has length of 5 bytes (yes, it is ideal for putting your jmp instruction).

Despite its name, kernel32.dll is not actually the kernel; it contains the entry points to Win32 API functions that implement Win32's interface to NT system calls.

Furthermore, Windows (like any modern OS) supports copy-on-write, so if you use detours to patch a shared DLL like kernel32.dll, the OS will first make a private copy for your process and patch that. The NT kernel itself is perfectly safe.