Questions about OpenGL Settings and drawing over a mask in a window

I would like to know the OpenGL Rendering settings for having a program render OpenGL over top of any window on screen that has a specific color code (screen-level buffer?)

I.E. VLC Media Player and Media Player Classic both have rendering modes which allow you to full-screen then minimize player, but maintain watching media via allowing a specific color to act as a transparent mask. For example, you could set the background color of a terminal application to be 0x000010 for VLC 0x000001 for MPC and you could then type over the media using text (as it is in it's original color). When you try to do a "printscreen" all you get is the mask color, However, this is an accept开发者_JS百科able side-effect.

Is it possible to do this as well with any OpenGL application with the right settings and hardware? If so, what are the settings or at least the terminology of this effect to further research it?

What you are trying to implement is called "overlay". You can try this angelcode tutorial. If I remember correctly, there was also a tutorial in DirectX SDK.

If you need to use OpenGL, you will need to perform offscreen rendering (using FBO or P-buffer), read the results using glReadPixels() and display using overlay.