Problem with getline and "strange characters"

I have a strange problem, I use

wifstream a("a.txt");
wstring line;
while (a.good()) //!a.eof()  not helping
{
     getline (a,line);
      //...
     wcout<<line<<endl;

}

and it works nicely for txt file like this http://www.speedyshare.com/files/29833132/a.txt (sorry for the link, but it is just 80 bytes so it shouldn't be a problem to get it , if i c/p on SO newlines get lost) BUT when I add for example 水 (from http://en.wikipedia.org/wiki/UTF-16/UCS-2#Examples )to any line that is the line where loading stops. I was under the wrong impression that getline that takes wstring as one input and wifstream as other can chew any txt input... Is there any way to read every single line in the file even if it contains funky characters?

The not-very-satisfying answer is that you need to imbue the input stream with a locale which understands the particular character encoding in question. If you don't know which locale to choose, you can use the empty locale.

For example (untested):

std::wifstream a("a.txt");
std::locale loc("");
a.imbue(loc);

Unfortunately, there is no standard way to determine what locales are available for a given platform, let alone select one based on the character encoding.

The above code puts the locale selection in the hands of the user, and if they set it to something plausible (e.g. en_AU.UTF-8) it might all Just Work.

Failing this, you probably need to resort to third-party libraries such as iconv or ICU.

Also relevant this blog entry (apologies for the self-promotion).

The problem is with your call to the global function getline (a,line). This takes a std::string. Use the std::wistream::getline method instead of the getline function.

C++ fstreams delegeate I/O to their filebufs. filebufs always read "raw bytes" from disk and then use the stream locale's codecvt facet to convert between these raw bytes into their "internal encoding".

A wfstream is a basic_fstream<wchar_t> and thus has a basic_filebuf<wchar_t> which uses the locale's codecvt<wchar_t, char> to convert the bytes read from disk into wchar_ts. If you read a UCS-2 encoded file, the conversion must thus be performed with a codecvt who "knows" that the external encoding is UCS-2. You thus need a locale with such a codecvt (see, for example, this SO question)

By default, the stream's locale is the global locale at the stream construction. To use a specific locale, it should be imbue()-d on the stream.