Remove Duplicate Lines

How to use the duplicate line remover

Paste your text into the box. The result updates as you type, and the counters show how many lines went in, how many came out and how many duplicates were removed. When a line appears more than once, the first occurrence is kept in its original position and later copies are dropped.

• Case-sensitive (on by default) treats Apple and apple as different lines. Turn it off to treat them as duplicates; the spelling of the first occurrence is kept.
• Trim spaces ignores spaces and tabs at the start and end of each line, and removes them from the output. This catches duplicates that only differ by a trailing space, which are invisible on screen.
• Remove empty lines drops blank lines entirely. Without it, repeated blank lines count as duplicates too, so only the first blank line stays.
• Sort orders the unique lines alphabetically after deduplication. Sorting is language-aware and treats numbers naturally, so item 2 comes before item 10.

Worked example

Suppose you merged two email lists:

anna@example.com
ben@example.com
Anna@example.com
ben@example.com 
carla@example.com

With the default settings the second ben@example.com (which has a trailing space) is removed, because spaces are trimmed. Anna@example.com stays, because the comparison is case-sensitive. Turn off Case-sensitive and it is removed as well, leaving three addresses. Since email addresses are not case-sensitive in practice, that is the setting you want here.

Typical uses

• Cleaning up keyword lists, tags or product SKUs before importing them.
• Merging mailing lists or lists of URLs from several sources.
• Getting the unique values of a spreadsheet column: copy the column, paste it here and copy the result back.
• Reducing log output to distinct messages.

Everything happens in your browser, so lists with customer data never leave your computer. Lists with hundreds of thousands of lines are processed in a fraction of a second, because each line is looked up in a hash set instead of being compared with every other line.

Common mistakes

• Invisible differences. Non-breaking spaces, tabs or a zero-width character make two lines look identical while being different. Trimming handles leading and trailing whitespace, but not characters in the middle of the line.
• Expecting the last occurrence. This tool keeps the first. To keep the last, reverse the line order first with the reverse text tool, deduplicate, then reverse again.
• Several items per line. Items separated by commas need to be on their own lines first. Replace the commas with line breaks in your editor.
• Different line endings. Windows (\r\n) and Unix (\n) line endings are both handled, so mixed files are deduplicated correctly.