Home · Blog · UTF-8

UTF-8: how text became universal

21 Sep 2026 · Data · 6 min read

Anyone who used the internet in the 2000s remembers the symptoms: a résumé arriving as "José" instead of "José", a Japanese website rendering as rows of empty boxes, an email subject line that looked like it had been through a blender. This was mojibake — garbled text — and it happened because the sender and the receiver disagreed about how bytes map to characters. Then UTF-8 won, almost completely, and the blender went quiet.

UTF-8 is now used by well over 95% of websites, and it's the default in nearly every modern system. This article explains the problem it solved, the elegant trick at its core, why "one character" stopped meaning "one byte", and where the remaining sharp edges are.

The problem: 128 characters weren't enough

Early computing standardised on ASCII: 128 characters — English letters, digits, basic punctuation — each fitting in 7 bits. It was enough for American English and nothing else. The rest of the world improvised with code pages: 256-slot tables where slots 128–255 held whatever a region needed. Latin-1 covered Western Europe, Windows-1252 was Microsoft's near-compatible variant, Shift_JIS covered Japanese — and they all disagreed with each other.

The result was chaos by design. A file was just bytes; which characters those bytes meant depended on which table you consulted. Byte 233 is "é" in Latin-1, something else entirely in another table, and meaningless in a third. Send a file without saying which table you used — and nothing reliably said it — and the receiver guessed. Guessing wrong produced mojibake. There was no way to mix languages in one document at all: a file couldn't contain both Greek and Japanese, because one byte couldn't mean two things.

Unicode's idea, UTF-8's trick

Unicode solved the identity problem by giving every character in every writing system a permanent number called a code point: U+0041 is "A", U+00E9 is "é", U+1F600 is "😀". One catalogue, no regional variants, room for over a million entries.

But a catalogue isn't an encoding — you still have to write those numbers as bytes. UTF-8's trick is variable length with a genius compatibility property:

The first row is the masterstroke: plain ASCII text is already valid UTF-8, byte for byte. Every English text file ever written needed zero conversion. That backwards compatibility is the single biggest reason UTF-8 conquered the world — adoption cost nothing for the huge installed base of ASCII, and everything else came along for the ride.

Bytes are not characters anymore

This is the mental shift UTF-8 demands: length in bytes and length in characters are different numbers. Concrete examples:

• "café" is 4 characters but 5 bytes (the é takes 2).
• "€100" is 4 characters but 6 bytes (the € takes 3).
• "😀" is 1 character but 4 bytes.
• A tweet-length string of 280 emoji is 280 characters but 1,120 bytes.

This bites in practice. A database column limited to 255 bytes holds fewer than 255 characters of non-English text. Cutting a string at a byte boundary can split a multi-byte character in half, producing the replacement character "�". And string-length limits in forms ("maximum 140 characters") must count characters, not bytes, or users writing in Thai or Arabic get mysteriously cut off early.

Emoji and the four-byte club

Emoji live in the supplementary planes (U+10000 and up), so every emoji costs 4 bytes in UTF-8. Older systems built before these planes mattered famously fumbled this: MySQL's original utf8 character set only supported 3 bytes per character — enough for nearly everything except emoji — and inserting a 😀 would fail or corrupt. The fix was utf8mb4 ("UTF-8 max bytes 4"), and to this day, encountering a database column still on 3-byte utf8 is a reliable sign of a system old enough to predate emoji being load-bearing infrastructure. Some emoji are even sequences: a family or a flag is multiple code points joined together, so one visible "character" can be 7+ code points and 25+ bytes.

Where UTF-8 still breaks

• Mojibake's last stand. UTF-8 bytes read as Latin-1 produce the unmistakable "é" pattern. If you see it, the bytes are almost certainly fine — the label is wrong. Fix the declared encoding, not the data.
• The BOM. Some Windows programs prepend an invisible byte-order mark (EF BB BF) to UTF-8 files. Most tools tolerate it; a few choke — notably, it can break scripts and confuse parsers that expect the file to start with content.
• Look-alikes. Unicode contains characters that look identical but aren't: Latin "A" (U+0041) vs Cyrillic "А" (U+0410). This enables phishing ("paypaI.com" with a capital i) and is a text-processing problem no encoding can solve.

UTF-8 vs UTF-16: why two encodings survived

If UTF-8 won so completely, why does UTF-16 still exist? History and inertia. Java, JavaScript, C# and Windows internals standardised on UTF-16 (then called simply "Unicode") in the 1990s, when 65,536 characters seemed like plenty and fixed two-byte units were convenient. Then the supplementary planes arrived, and UTF-16 needed surrogate pairs — two 16-bit units for one character — making it variable-length anyway, with all of UTF-8's complexity and none of its ASCII compatibility. New systems choose UTF-8; old platforms carry UTF-16 in their internals while speaking UTF-8 on the wire. You'll meet UTF-16 mainly as an implementation detail — JavaScript strings are UTF-16 under the hood, which is why an emoji counts as length 2 in "😀".length.

How to identify a mystery encoding

Handed a garbled file with no declared encoding? Work the clues in order. First, check for a byte-order mark at the start — EF BB BF means UTF-8, FF FE means UTF-16. Second, look at the damage pattern: the "é"/"ä" signature almost always means UTF-8 bytes misread as Latin-1 or Windows-1252 — reinterpret the bytes as UTF-8 and it usually snaps into focus. Third, try UTF-8 first as a decoder; it's strict enough that invalid sequences fail loudly rather than silently producing wrong characters. For genuinely unknown files, detection libraries (like Python's chardet) guess statistically, but they're heuristics — right most of the time, confidently wrong some of the time. The durable fix is always the same: declare UTF-8 explicitly at every boundary, and the mystery never arises.

The honest caveat

UTF-8 solved interchange — bytes now travel the world and arrive as the intended characters. It did not solve everything about text. The same visual character can have multiple code-point representations (é as one code point vs e + combining accent), which breaks naive string comparison — that's normalisation, a separate layer. Right-to-left scripts, line breaking in languages without spaces, and font coverage are all above UTF-8's pay grade. Think of UTF-8 as the foundation: necessary, nearly universal, and no longer the thing you should worry about — so you can spend that worry on the genuinely hard problems it uncovered.

Try the Base64 Encoder