Hidden characters
What are zero-width spaces, and how do you remove them?
Zero-width spaces are invisible characters that break search, regex, and duplicate checks. Here is what they are, where they come from, and how to remove them properly.
A zero-width space is a Unicode character (U+200B) that takes up no visible width. It exists so that text in scripts without spaces can be broken across lines, but it also appears accidentally in copied text, where it splits words invisibly and breaks search, regex, string comparison, and duplicate detection. You remove it by filtering the specific code points — not by deleting whitespace generally.
That distinction matters: a tool that strips “extra spaces” will not touch U+200B, because it is not a space. It is a formatting character that happens to be invisible.
The characters people mean
When someone says “zero-width space”, they usually mean a small family of invisible characters. The common ones:
| Code point | Name | Typical effect |
|---|---|---|
| U+200B | Zero-width space | Allows a line break; splits words invisibly |
| U+200C | Zero-width non-joiner | Prevents a ligature or joining form |
| U+200D | Zero-width joiner | Joins characters, including emoji sequences |
| U+2060 | Word joiner | Prevents a line break at that position |
| U+FEFF | Byte-order mark | Marks encoding; sometimes appears mid-string after a copy |
| U+00AD | Soft hyphen | A hidden hyphenation point |
Two of these are legitimate in the right context. U+200D is how emoji sequences such as 👨👩👧 stay a single glyph, and U+200C is required in some Persian and Indic text. Removing every invisible character unconditionally is therefore not the goal; removing the ones that do not belong is.
Where they come from
Copying from a web page. Invisible characters travel with the text. A page that uses them for line-breaking control hands them to your clipboard.
Copying from an AI chat. Models sometimes reproduce characters they saw in training data, including zero-width spaces inside words. They also use them when formatting long strings.
Emoji sequences. A single emoji family can contain three joiner characters. If you copy part of a sequence, the joiners come along.
Encoding conversions. A UTF-8 file read as another encoding can produce a byte-order mark in the middle of a string, which is invisible in most editors.
Deliberate insertion. Some tools insert invisible characters to detect copied content, and some people use them to evade filters. If a phrase looks identical but does not match, this is one explanation.
What breaks when they are present
Search. On-site search compares strings. A zero-width space inside a product name means the name typed by a customer will not match the name in your database.
Regex and validation. Patterns anchored with ^ and $ fail when an invisible character sits at the start or end of a field. Email and URL validation are common casualties.
Code. String equality, JSON keys, environment variables, and command-line arguments all compare exact bytes. One invisible character produces a mismatch that is almost impossible to spot by reading.
Content operations. Duplicate-content detection, translation memories, and diff tools treat the two strings as different. A heading with a hidden character will not be found by “find and replace” across a document.
Copy protection schemes. Some publishers insert zero-width characters to trace copied text. If you paste such text into a document, the markers come with it.
How to find them
You cannot see them, so counting code points is the only reliable test.
In the browser. Paste into the hidden character remover. The cleaning report shows how many invisible characters were removed, which is the count you want.
In an editor. VS Code can render control characters with editor.renderControlCharacters, and most editors let you paste a character into the find box if you can obtain it.
In the terminal. On macOS or Linux, grep -P "[\x{200B}-\x{200D}\x{2060}]" file.txt lists matching lines. perl -CSD -ne 'print if /[\x{200B}-\x{200D}]/' does the same more portably.
In Python. [c for c in text if unicodedata.category(c) == "Cf"] returns every format character, which is a superset of what you probably want to remove.
How to remove them
The safe method is a targeted filter over the specific code points, followed by a normalisation pass for the whitespace characters that usually travel with them.
- Copy the text.
- Paste it into JustCleanText.
- Keep Clean invisible Unicode on and turn the other toggles off if you want to change nothing else.
- Check the report for the hidden-character count.
- Copy plain text if the destination is a code editor or CMS field.
Do not use a general “remove all non-ASCII” filter. It will delete accented letters, curly quotes you may want, and every character from a non-Latin script. Filtering by code point is precise; filtering by “not ASCII” is destructive.
What should be preserved
A careful cleaner keeps:
- Emoji presentation selectors (U+FE0F), which control whether an emoji renders in colour.
- The Arabic letter mark (U+061C), which is required for correct right-to-left rendering.
- Bidi isolates in text that legitimately mixes directions.
- The ideographic space (U+3000) used in CJK typography.
JustCleanText removes the zero-width family, legacy bidi embeddings, soft hyphens, non-emoji variation selectors, and stray control characters, and leaves the four categories above alone. That is why it can be used on multilingual text without fear.
Before and after
The two lines below look identical. The first contains a zero-width space between “pay” and “ment”.
The payment failed.
The payment failed.
After cleaning, both are the same string, and a search for “payment failed” finds both.
A quick sanity check
If a paste behaves strangely — a word that will not match, a link that will not resolve, a form that rejects a valid value — copy the text into the cleaner and look at the hidden-character count before you debug anything else. It takes ten seconds and rules out an entire class of problem.
Open the hidden character remover · Read why AI text contains hidden Unicode
Put this into practice
Clean the text before you paste it
Paste the text into JustCleanText, keep the toggles that match your problem, and copy either rich text or plain text. Everything runs in this browser tab.
Open the cleaner