Detecting debugger on Mac OS X

I am trying to detect whether my process is being run in a debugger or not and, while in Windows there are many solutions a开发者_运维技巧nd in Linux I use:

ptrace(PTRACE_ME,0,0,0) 

and check its return value, I did not manage to perform the same basic check on Mac OS X. I tried to use the

ptrace(PT_TRACE_ME,0,0,0)

call but it always returns 0 even when run under gdb.

If I change the request to PT_DENY_ATTACH it correctly stops the debugging but that is not what I want to achieve. Any ideas?

You can just call the function AmIBeingDebugged() from Apple Technical Q&A QA1361, which is reproduced here because Apple sometimes breaks documentation links and makes old documentation hard to find:

#include <assert.h>
#include <stdbool.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/sysctl.h>

static bool AmIBeingDebugged(void)
    // Returns true if the current process is being debugged (either 
    // running under the debugger or has a debugger attached post facto).
{
    int                 junk;
    int                 mib[4];
    struct kinfo_proc   info;
    size_t              size;

    // Initialize the flags so that, if sysctl fails for some bizarre 
    // reason, we get a predictable result.

    info.kp_proc.p_flag = 0;

    // Initialize mib, which tells sysctl the info we want, in this case
    // we're looking for information about a specific process ID.

    mib[0] = CTL_KERN;
    mib[1] = KERN_PROC;
    mib[2] = KERN_PROC_PID;
    mib[3] = getpid();

    // Call sysctl.

    size = sizeof(info);
    junk = sysctl(mib, sizeof(mib) / sizeof(*mib), &info, &size, NULL, 0);
    assert(junk == 0);

    // We're being debugged if the P_TRACED flag is set.

    return ( (info.kp_proc.p_flag & P_TRACED) != 0 );
}

#include <mach/task.h>
#include <mach/mach_init.h>
#include <stdbool.h>

static bool amIAnInferior(void)
{
    mach_msg_type_number_t count = 0;
    exception_mask_t masks[EXC_TYPES_COUNT];
    mach_port_t ports[EXC_TYPES_COUNT];
    exception_behavior_t behaviors[EXC_TYPES_COUNT];
    thread_state_flavor_t flavors[EXC_TYPES_COUNT];

    exception_mask_t mask = EXC_MASK_ALL & ~(EXC_MASK_RESOURCE | EXC_MASK_GUARD);
    kern_return_t result = task_get_exception_ports(mach_task_self(), mask, masks, &count, ports, behaviors, flavors);
    if (result == KERN_SUCCESS)
    {
        for (mach_msg_type_number_t portIndex = 0; portIndex < count; portIndex++)
        {
            if (MACH_PORT_VALID(ports[portIndex]))
            {
                return true;
            }
        }
    }
    return false;
}

This looks and sees if there is an active exception handler in our process (for EXC_BREAKPOINT, EXC_BAD_ACCESS, etc). Ptrace is not required to achieve this in a debugger, thus relying only on a ptrace flag to be set is not quite ideal.

This approach is mentioned in http://reverse.put.as/wp-content/uploads/2012/07/Secuinside-2012-Presentation.pdf

My blog post describes this in more detail.

Here's a Swift version of the function from Apple Technical Q&A QA1361:

import Foundation

extension ProcessInfo {
    /// - returns: true if the process is being debugged, else false.
    public func isBeingDebugged() -> Bool {
        // https://developer.apple.com/library/archive/qa/qa1361/_index.html
        // Technical Q&A QA1361: Detecting the Debugger

        var mib: [Int32] = [
            CTL_KERN,
            KERN_PROC,
            KERN_PROC_PID,
            processIdentifier
        ]

        var info = kinfo_proc()
        var size: size_t = MemoryLayout.size(ofValue: info)

        let rc = sysctl(&mib, UInt32(mib.count), &info, &size, nil, 0)
        assert(rc == 0)

        return (info.kp_proc.p_flag & P_TRACED) != 0
    }
}

Usage:

if ProcessInfo.processInfo.isBeingDebugged() {
     print("running under the debugger")
}