Unboxing uint/int without knowing what's inside the box

I have an object o that is known to be a boxed int or uint:

object o = int.MinValue
object o = (uint)int.MinValue // same bytes as above

I don't know what's in the box, all I care about is that there's 4 bytes in there that I want to coerce to an int or uint. This works fine in an unchecked context when I have values (instead of boxes):

unchecked
{
    int a = (int)0x80000000u; // will be int.MinValue, the literal is a uint
    uint b = (uint)int.MinValue;
}

Note: By default everything in C# is unchecked, the unchecked context is only necessary here because we are dealing with literals and the compiler wants to know if we really want to shoot ourselves in the foot.

The problem is now that I don't know whats inside the box (besides it's 4 bytes), but the runtime does so when I try to unbox to the wrong type I get an InvalidCastException. I know this is reasonable runtime behavior, but in this case I know what I'm doing and want a "unchecked unbox". Does something like that exist?

I know I could catch and retry, so that doesn't count as an answer.

You could use Convert.ToInt32 to convert any object to an int if possible, although it will also do conversion or parsing so it may be slower than you want.

If you know it's an int or uint, you could do this:

int x = (o is int) ? (int)o : (int)(uint)o;