How to create a custom WPF Window like this one?

Add the following properties to your Window definition to get rid of the default style

WindowStyle="None"
AllowsTransparency="True"
Background="Transparent"

I'm not sure the exact details of how they created their border but I'm guessing it has something to do with a Border which has a Transparent background, a CornerRadius, and perhaps a Blur Effect. It might actually be 2-3 borders (one black 1px one with a ~ 5 corner radius, a transparent one with the blur effect, and a third square one for around the content box)

Using Microsoft Expression Blend