Mojibake: Why Text Turns Into é and ’, and How to Fix It
Mojibake is text that has turned into nonsense symbols: an é that became é, an apostrophe that became ’. The letters were never damaged. They were stored with one character encoding and read with another. This guide shows the usual patterns, explains the byte order mark, and gives you ways to repair the text and stop it happening again.
What is mojibake?
The word is Japanese and means roughly transformed characters. Japan met the problem early, because several incompatible encodings for Japanese were in use at the same time. In English the same thing is called garbled text.
A computer stores text as bytes. An encoding is the table that says which bytes stand for which characters. As long as the program that reads the bytes uses the same table as the program that wrote them, everything works. When the reader assumes a different table, the bytes are still intact but they are shown as the wrong characters.
The most common cause: UTF-8 read as Windows-1252
UTF-8 is the standard encoding of the web. It stores the basic English letters, digits and punctuation in one byte each, and every other character in two, three or four bytes. Windows-1252 and its close relative Latin-1 (ISO 8859-1) are older Western European encodings that store every character in exactly one byte.
When UTF-8 text is read as Windows-1252, each of those multi-byte characters is shown as two or three separate characters. English text survives almost untouched, because its letters are the same in both tables. Only accented letters, curly quotes, dashes and symbols break, which is why the damage looks so scattered.
| Meant | UTF-8 bytes | Shown as |
|---|---|---|
| é | C3 A9 | é |
| ü | C3 BC | ü |
| ñ | C3 B1 | ñ |
| ’ (curly apostrophe) | E2 80 99 | ’ |
| “ (opening curly quote) | E2 80 9C | “ |
| – (en dash) | E2 80 93 | – |
| — (em dash) | E2 80 94 | — |
| … (ellipsis) | E2 80 A6 | … |
| • (bullet) | E2 80 A2 | • |
| € | E2 82 AC | € |
| £ | C2 A3 | £ |
| © | C2 A9 | © |
| ° | C2 B0 | ° |
| Non-breaking space | C2 A0 | Â followed by a space |
Two patterns are worth remembering. A stray à in front of a symbol means an accented letter. The sequence †followed by one more character means a curly quote, a dash or another piece of typographic punctuation. The closing curly quote is a special case: its third byte has no character in Windows-1252, so you often see †with nothing visible after it.
Other patterns you may see
The replacement character
The opposite mistake, older one-byte text read as UTF-8, produces a different symptom. A lone byte such as the Latin-1 E9 for é is not valid UTF-8, so the reader substitutes the replacement character U+FFFD, drawn as a question mark inside a black diamond. The word café comes out with that diamond in place of its last letter.
Double encoding
If garbled text is saved as UTF-8 and misread a second time, the damage stacks up. The é that became é turns into é. Long runs of à and  are the signature of text that has been through the mistake more than once.
Plain question marks
When a program converts text into an encoding that has no code for a character, it often writes an ordinary question mark instead. This is not mojibake in the strict sense, and it cannot be undone, because the original byte is gone.
East Asian text
The same mechanism affects every script. Chinese readers know a famous three-character sequence, 锟斤拷, which appears when two replacement characters saved as UTF-8 are read with the Chinese GBK encoding.
The byte order mark (BOM)
The byte order mark is the character U+FEFF placed at the very start of a file. In UTF-16, which stores text in two-byte units, it has a real job: it tells the reader which byte of each unit comes first. In UTF-8 there is no byte order to mark, so there it serves only as a label saying that the file is UTF-8.
| Encoding | BOM bytes |
|---|---|
| UTF-8 | EF BB BF |
| UTF-16, little-endian | FF FE |
| UTF-16, big-endian | FE FF |
A UTF-8 BOM is optional, and whether you want one depends on who reads the file:
- It helps Excel. When you open a CSV file by double-clicking it, Excel uses the BOM as its cue that the file is UTF-8. Without it, accented names can come out garbled. Excel's own format CSV UTF-8 writes the mark for this reason.
- It hurts scripts and strict parsers. A shell script that begins with a BOM may not run, because the first line no longer starts with the expected two characters. The JSON standard says a BOM must not be added, and some parsers reject a file that has one.
- It hides in the first field. A program that does not strip the mark treats it as part of the first value, so the first column header of a CSV file silently stops matching its name. In Python, opening the file with the encoding
utf-8-sigremoves it.
When a UTF-8 BOM is itself misread as Windows-1252, it appears as the three characters  at the top of the page or file.
How to fix garbled text
The first rule is not to retype or search-and-replace in a hurry. If the bytes are intact, the right fix is to read them again with the correct encoding.
- Reopen the file with the right encoding. In Visual Studio Code, click the encoding name in the status bar, choose Reopen with Encoding and pick the encoding the file is really in. Once it looks right, use Save with Encoding and choose UTF-8.
- Import instead of opening. In Excel, load a CSV through Data > From Text/CSV, where you can set the file origin to UTF-8, instead of double-clicking the file.
- Convert on the command line. The
iconvtool rewrites a file from one encoding to another.
iconv -f WINDOWS-1252 -t UTF-8 old.txt > new.txt
If the garbled characters have already been saved as text, for instance in a database, you can often reverse the mistake in code. In Python, this turns é back into é:
fixed = broken.encode("cp1252").decode("utf-8")
It raises an error when the text contains characters that cannot be mapped back, and text that was damaged twice needs the step twice. The open-source Python library ftfy automates the detective work for mixed cases.
How to prevent it
- Use UTF-8 everywhere, and say so. On a web page, put
<meta charset="utf-8">near the top of the head and send the matching header from the server. - Name the encoding when you open a file in code. In Python, write
open(path, encoding="utf-8")so the result does not depend on the computer's default. - Check the database. In MySQL, use the
utf8mb4character set for tables and for the connection. - Be careful at every hand-over. Exports, imports, email attachments and copy and paste between old and new systems are where the encoding label gets lost.
What a text cleaner can and cannot do
The AI Text Cleaner does not repair mojibake. Undoing it means reinterpreting bytes, and by the time text is pasted into a web page, the bytes have already been turned into characters. What the cleaner can do is remove a byte order mark that came along with pasted text. With Remove invisible characters switched on, a U+FEFF is deleted and shown in the list of characters found. It leaves the replacement character U+FFFD in place, since deleting it would hide the fact that a letter is missing.
Quick answers
Why does an apostrophe turn into three strange symbols? A curly apostrophe takes three bytes in UTF-8. Read as Windows-1252, each byte is shown as its own character.
Can mojibake always be reversed? No. It can be reversed while the original bytes are still there. Once characters have been replaced by question marks or replacement characters, that information is lost.
Should I save my files with or without a UTF-8 BOM? Without, in most cases, and especially for code, JSON and web pages. Add it only for a program that needs it, such as a CSV file meant to be opened directly in Excel.