Search code examples
javaunicodesurrogate-pairs

Handling Unicode surrogate values in Java strings

Consider the following code:

byte aBytes[] = { (byte)0xff,0x01,0,0,
                  (byte)0xd9,(byte)0x65,
                  (byte)0x03,(byte)0x04, (byte)0x05, (byte)0x06, (byte)0x07,
                  (byte)0x17,(byte)0x33, (byte)0x74, (byte)0x6f,
                   0, 1, 2, 3, 4, 5,
                   0 };
String sCompressedBytes = new String(aBytes, "UTF-16");
for (int i=0; i<sCompressedBytes.length; i++) {
    System.out.println(Integer.toHexString(sCompressedBytes.codePointAt(i)));
}

Gets the following incorrect output:

ff01, 0, fffd, 506, 717, 3374, 6f00, 102, 304, 500.

However, if the 0xd9 in the input data is changed to 0x9d, then the following correct output is obtained:

ff01, 0, 9d65, 304, 506, 717, 3374, 6f00, 102, 304, 500.

I realize that the functionality is because of the fact that the byte 0xd9 is a high-surrogate Unicode marker.

Question: Is there a way to feed, identify and extract surrogate bytes (0xd800 to 0xdfff) in a Java Unicode string?
Thanks

Solution

  • Is there a way to feed, identify and extract surrogate bytes (0xd800 to 0xdfff) in a Java Unicode string?

    Just because no one has mentioned it, I'll point out that the Character class includes the methods for working with surrogate pairs. E.g. isHighSurrogate(char), codePointAt(CharSequence, int) and toChars(int). I realise that this is besides the point of the stated problem.

    new String(aBytes, "UTF-16");

    This is a decoding operation that will transform the input data. I'm pretty sure it is not legal because the chosen decoding operation requires the input to start with either 0xfe 0xff or 0xff 0xfe (the byte order mark). In addition, not every possible byte value can be decoded correctly because UTF-16 is a variable width encoding.

    If you wanted a symmetric transformation of arbitrary bytes to String and back, you are better off with an 8-bit, single-byte encoding because every byte value is a valid character:

    Charset iso8859_15 = Charset.forName("ISO-8859-15");
byte[] data = new byte[256];
for (int i = Byte.MIN_VALUE; i <= Byte.MAX_VALUE; i++) {
  data[i - Byte.MIN_VALUE] = (byte) i;
}
String asString = new String(data, iso8859_15);
byte[] encoded = asString.getBytes(iso8859_15);
System.out.println(Arrays.equals(data, encoded));

    Note: the number of characters is going to equal the number of bytes (doubling the size of the data); the resultant string isn't necessarily going to be printable (containing as it might, a bunch of control characters).

    I'm with Jon, though - putting arbitrary byte sequences into Java strings is almost always a bad idea.