Byte Order Mark (BOM): What It Is and How to Remove It
A byte order mark, usually shortened to BOM, is an invisible character that some programs write at the very start of a text file to say which encoding the file uses. It is harmless in many places and a nuisance in others: it turns the first column of a CSV into an unfindable name, makes some JSON parsers refuse a file, and can appear on screen as the strange string . This guide explains where it comes from and how to remove it.
The BOM in one table
| Encoding | BOM bytes | How it looks if misread as Windows-1252 |
|---|---|---|
| UTF-8 | EF BB BF |  |
| UTF-16, little-endian | FF FE | ÿþ |
| UTF-16, big-endian | FE FF | þÿ |
All three are the same character, U+FEFF, written in different encodings. In UTF-16 the order of the two bytes tells a program whether the file is little-endian or big-endian, which is where the name comes from. UTF-8 has no byte order to signal, so there the mark only says "this file is UTF-8".
Where BOMs come from
- Windows tools. Older versions of Notepad saved UTF-8 files with a BOM by default. Current Notepad offers both "UTF-8" and "UTF-8 with BOM".
- Spreadsheet exports. Excel's "CSV UTF-8" format writes a BOM so that Excel can recognise the file as UTF-8 when it is opened again.
- Some editors and IDEs, depending on their settings.
- Copy and paste. When text is copied from the start of a file that has a BOM, the invisible character can come along.
Problems a BOM causes
| Where | Symptom |
|---|---|
| CSV files | The first column header is not found. A script looking for id gets \ufeffid |
| JSON | Some parsers reject the file. The JSON standard says a BOM must not be added, though parsers may choose to ignore one |
| PHP | "Headers already sent" errors, because the BOM is output before the page starts |
| Shell scripts | The #! line is not recognised, so the script does not run |
| Web pages |  appears at the top of the page when the encoding is declared wrongly |
| Comparisons | Two values look the same but do not match, because one starts with an invisible character |
When you should keep it
The BOM is not always a mistake. Excel relies on it to open UTF-8 CSV files correctly; without it, accented letters and other non-English characters can appear garbled. If people will open your CSV in Excel by double-clicking it, saving it with a BOM is often the right choice. For files read by programs, websites and APIs, UTF-8 without a BOM is the safer default.
How to check whether a file has one
On the command line
head -c 3 file.csv | xxd
If the output starts with efbb bf, the file begins with a UTF-8 BOM.
In a code editor
Visual Studio Code shows the encoding in the status bar, for example "UTF-8 with BOM". Many other editors show it in the same place or in the save dialog.
In Python
with open("file.csv", "rb") as f:
print(f.read(3) == b"\xef\xbb\xbf")
How to remove it
Visual Studio Code
Click the encoding in the status bar, choose "Save with Encoding", then "UTF-8".
Notepad on Windows
Choose Save As and pick "UTF-8" rather than "UTF-8 with BOM" in the encoding list.
Command line (GNU sed)
sed -i '1s/^\xEF\xBB\xBF//' file.csv
Python, when reading
Open the file with the encoding utf-8-sig. It reads files with or without a BOM and drops the mark if it is there:
with open("file.csv", encoding="utf-8-sig") as f:
text = f.read()
Pasted text
The option Remove invisible characters in the AI Text Cleaner removes U+FEFF wherever it appears in the text you paste, and lists it in the summary as a byte order mark. That fixes values copied from the top of a file; to fix the file itself, save it again without a BOM as described above.
A BOM in the middle of text
The same character, U+FEFF, was once also used inside text as a "zero-width no-break space" to stop a line from breaking. That use is deprecated; the word joiner, U+2060, does the job now. When U+FEFF turns up in the middle of a sentence today, it is almost always an accident left over from joining files together. It behaves like the other invisible characters described in our invisible character list.
Quick answers
What does  mean at the start of a file? It is a UTF-8 byte order mark being read with the wrong encoding. Remove the BOM or open the file as UTF-8.
Does UTF-8 need a BOM? No. UTF-8 works without one, and most programs expect it to be absent. Excel is the main exception.
Is the BOM the same as a zero-width space? No. The zero-width space is U+200B; the BOM is U+FEFF. Both are invisible, and both are covered in our guide to the zero-width space.