How can i get this kind of window?

In eclipse you can detach tabs and they will displayed in a special kind of window.

Win7 with aero displays it as follows

http://www.oliholz.de/data/1.jpg

How can i produce this frame? i have tried many styles of JFrame,JWi开发者_高级运维ndow,JDialog, ...

As other people mentioned eclipse uses it's own library for UI. That library is based on SWT which is better integrated with the native window system. See this article for explanation on how SWT, AWT, and Swing relate to each other. Eclipse Rich Client Platform which is also mentioned here can be observed as a layer above SWT that can make your life much easier if you intend to develop a complex, feature rich application (for example an IDE for the new programming language you made).

Take a look at the Eclipse Rich Client Platform. If you want to emulate their look and feel, you might consider building your user interface upon their free (and rather good) open source framework.