Comments on: Unicode and You

By: Avi

Avi — Tue, 23 Jun 2009 13:31:47 +0000

Cool, thanks for the first answer, Kalid. I’ll look over the article that you mentioned.

Avi

By: Kalid

Kalid — Mon, 22 Jun 2009 19:27:43 +0000

@Kai: Thanks, glad you liked it.

@Avi: Thank you! Great questions

1) Unicode gives a number (called a code point) to every symbol, so “a” “b” and “c” each have their own number. Sometimes the same exact symbol (like a) will have two different numbers that represent it, to be compatible with the old formats like ASCII.

From a purist point of view, it’d be nice to have every symbol have exactly 1 number, but from a practical standpoint the system needs to be backwards compatible. So latin characters appear where they are today, in the ASCII range under 127, and also in another “proper” location defined by the unicode standard.

2) Great question. You’re right, in UTF-8, avi should only have 3 bytes (61 76 69). The preceding ones (EF BB BF) are part of the BOM (byte order mark) that defines whether the data is big or little endian. You can read more about it here:

http://betterexplained.com/articles/understanding-big-and-little-endian-byte-order/

(Scroll down for the part about Unicode).

By: Avi

Avi — Sun, 21 Jun 2009 12:08:52 +0000

Very good article – the most clear article I have encountered on the web.
Keep up the good work Kalid!
Two things I didn’t understand:
1. “Purists probably didn’t like this, because the full Latin character sets were defined elsewhere, and now one letter had 2 codepoints”. “Defines elsewhere”, “2 codepoints” – I didn’t get it.
2. Regarding the first byte count which indicates on the number of bytes in. I wrote “avi” and saved it as UTF8. I looked at hex editor and saw that the first byte was 1111…
The bytes there were: EF BB BF 61 76 69 so it amounts to 6 bytes -can somebody explain this issue to me.