Android: Unicode/Charset problems when sending an SMS (sendTextMessage)

Basically I have a working application that sends an SMS after receiving an SMS.

Everything works fine, except when the SMS text to send has "special chars", ie "é,à,í,ç", etc.

I've tried many things including charset conversion but I simply can't make it work... the msgText always comes back with charset encoding problems.

Here's the part where the message is sent:

if (msgText.length() > 160) {
    ArrayList msgTexts = sm.divideMessage(msgText);
    sm.sendMultipartTextMessage(PhoneNumber, null, msgTexts, null, null);
} else {
    try {
        sm.sendTextMessage(PhoneNumber, null, msgText, null, null);
    } catch (IllegalArgumentException e) {
        e.printStackTrace();
    }
}

Here's the charset conversion function I tried (but didn't help), that I applied on msgText:

public static String formatCharset(String txtInicial) {
    //-- Please notice this is just for reference, I tried every charset from/to conversion possibility. Even stupid ones and nothing helped.

    /*try {//-- Seems simpler, it should do the same as below, but didn't help
        msgText = new String(msgText.getBytes("UTF-8"), "ISO-8859-1");
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
    }*/

    Charset charsetOrigem = Charset.forName("UTF-8");
    CharsetEncoder encoderOrigem = charsetOrigem.newEncoder();
    Charset charsetDestino = Charset.forName("ISO-8859-1");
    CharsetDecoder decoderDestino = charsetDestino.newDecoder();

    String txtFinal = "";

    try {
        ByteBuffer bbuf = encoderOrigem.encode(CharBuffer.wrap( txtInicial ));
        CharBuffer cbuf = decoderDestino.decode(bbuf);
        txtFinal = cbuf.toString();
    } catch (CharacterCodingException e) {
        e.printStackTrace();
    }

    if (txtFinal.length() == 0) txtFinal = txtInicial;

    return txtFinal;
}

Near desperation I even tried the solution for unicode messaging in here (didn't help as well):

http://since2006.com/blog/android-send-unicode-message/

Anyway, here's the (cleaned up - package is com.THE.APPLICATION, main activity is MAINACT) LogCat for when it crashes (when trying to send the message, after receiving one):

WARN/dalvikvm(28218): threadid=1: thread exiting with uncaught exception (group=0x4001d7f0)
ERROR/AndroidRuntime(28218): FATAL EXCEPTION: main
ERROR/AndroidRuntime(28218): java.lang.RuntimeException: Error receiving broadcast Intent { act=android.provider.Telephony.SMS_RECEIVED (has extras) } in com.THE.APPLICATION.SMSReceiver@44acd880
ERROR/AndroidRuntime(28218):     at android.app.ActivityThread$PackageInfo$ReceiverDispatcher$Args.run(ActivityThread.java:905)
ERROR/AndroidRuntime(28218):     at android.os.Handler.handleCallback(Handler.java:587)
ERROR/AndroidRuntime(28218):     at android.os.Handler.dispatchMessage(Handler.java:92)
ERROR/AndroidRuntime(28218):     at android.os.Looper.loop(Looper.java:123)
ERROR/AndroidRuntime(28218):     at android.app.ActivityThread.main(ActivityThread.java:4627)
ERROR/AndroidRuntime(28218):     at java.lang.reflect.Method.invokeNative(Native Method)
ERROR/AndroidRuntime(28218):     at java.lang.reflect.Method.invoke(Method.java:521)
ERROR/AndroidRuntime(28218):     at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:868)
ERROR/AndroidRuntime(28218):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:626)
ERROR/AndroidRuntime(28218):     at dalvik.system.NativeStart.main(Native Method)
ERROR/AndroidRuntime(28218): Caused by: java.lang.NullPointerException
ERROR/AndroidRuntime(28218):     at android.os.Parcel.readException(Parcel.java:1253)
ERROR/AndroidRuntime(28218):     at android.os.Parcel.readException(Parcel.java:1235)
ERROR/AndroidRuntime(28218):     at com.android.internal.telephony.ISms$Stub$Proxy.sendText(ISms.java:369)
ERROR/AndroidRuntime(28218):     at android.telephony.SmsManager.sendTextMessage(SmsManager.java:87)
ERROR/AndroidR开发者_如何学Gountime(28218):     at com.THE.APPLICATION.MAINACT.sendMessage(MAINACT.java:214)
ERROR/AndroidRuntime(28218):     at com.THE.APPLICATION.SMSReceiver.onReceive(SMSReceiver.java:24)
ERROR/AndroidRuntime(28218):     at android.app.ActivityThread$PackageInfo$ReceiverDispatcher$Args.run(ActivityThread.java:892)
ERROR/AndroidRuntime(28218):     ... 9 more

Sample of message text to send with issues:

VERBOSE/debug_tag(28218): msgText is: possível.

So, it reads possível when it should be possível

Please some enlightened soul help me out. He/She'll have a special place in my heart! :)

Edit: If the special place in my heart doesn't cut it, I'm willing to pay a few bucks for a working solution...

Ok, this seems to have been solved by simply using sendMultipartTextMessage instead of sendTextMessage for the messages.

Who would've thought... it kind of makes sense because unicode characters use more "space" than "normal" ones.

I have used this code to convert UTF-8 characters into ASCII. Then sending of SMS works and I can use 160 characters:

private static final String PLAIN_ASCII = "AaEeIiOoUu" // grave
        + "AaEeIiOoUuYy" // acute
        + "AaEeIiOoUuYy" // circumflex
        + "AaOoNn" // tilde
        + "AaEeIiOoUuYy" // umlaut
        + "Aa" // ring
        + "Cc" // cedilla
        + "OoUu" // double acute
;

private static final String UNICODE = "\u00C0\u00E0\u00C8\u00E8\u00CC\u00EC\u00D2\u00F2\u00D9\u00F9"
        + "\u00C1\u00E1\u00C9\u00E9\u00CD\u00ED\u00D3\u00F3\u00DA\u00FA\u00DD\u00FD"
        + "\u00C2\u00E2\u00CA\u00EA\u00CE\u00EE\u00D4\u00F4\u00DB\u00FB\u0176\u0177"
        + "\u00C3\u00E3\u00D5\u00F5\u00D1\u00F1"
        + "\u00C4\u00E4\u00CB\u00EB\u00CF\u00EF\u00D6\u00F6\u00DC\u00FC\u0178\u00FF"
        + "\u00C5\u00E5" + "\u00C7\u00E7" + "\u0150\u0151\u0170\u0171";

// remove accentued from a string and replace with ascii equivalent
public static String convertNonAscii(String s) {
    if (s == null)
        return null;
    StringBuilder sb = new StringBuilder();
    int n = s.length();
    for (int i = 0; i < n; i++) {
        char c = s.charAt(i);
        int pos = UNICODE.indexOf(c);
        if (pos > -1) {
            sb.append(PLAIN_ASCII.charAt(pos));
        } else {
            sb.append(c);
        }
    }
    return sb.toString();
}

I am using this command line:

Html.fromHtml(new String(myString.getBytes("UTF-8"))).toString();

and my SMS message looks perfect.

You can only use 160 chars if it's 7-bit encoding. If you use 140 chars, it should work without sendMultipartTextMessage because you're using 8-bit chars (UTF-8).

I has the same problem with special characters. When I change message MAX_SMS_MESSAGE_LENGTH to 70 everything works well. Look at that link:

https://forums.macrumors.com/threads/special-characters-in-sms-turn-them-shorter-70-characters-old-issue-never-solved.1030577/

This is my code:

    public static void sendSMS(String phoneNumber, String message, Context context) {

    String SENT = "SMS_SENT";
    int MAX_SMS_MESSAGE_LENGTH = 70;

    SmsManager smsManager = SmsManager.getDefault();
    PendingIntent sentPI;
    sentPI = PendingIntent.getBroadcast(context, 0,new Intent(SENT), 0);

    try {
        if(message.length() > MAX_SMS_MESSAGE_LENGTH) {
            ArrayList<String> messageList = SmsManager.getDefault().divideMessage(message);
            smsManager.sendMultipartTextMessage(phoneNumber, null, messageList, null, null);
        } else {
            smsManager.sendTextMessage(phoneNumber, null, message, sentPI, null);
        }
    } catch (Exception e) {
        Log.e("SmsProvider", "" + e);
    }
}

Of course you can insert a controller that will check if the message contains some special character and then change MAX_SMS_MESSAGE_LENGTH from 160 to 70. I always have special characters in my app.