Remove Duplicate Lines
Remove duplicate lines from a list while keeping the original order. Options for case sensitivity, trimming spaces and removing empty lines. Free and private.
Runs entirely in your browser. Your data is never uploaded.
What is Duplicate Line Remover?
Deletes repeated lines from a list and keeps the first occurrence of each, in the original order. It reports how many lines were removed and can ignore letter case, surrounding spaces and empty lines.
How to use Duplicate Line Remover
- Paste your list (one item per line) into the Input box.
- Choose whether matching is Case sensitive and whether to Trim spaces at line start/end and Remove empty lines.
- Copy the de-duplicated list from Output; the line below shows how many lines were removed.
Example
a B b a
a B
Default settings (case-insensitive): 2 line(s) removed
When to use Duplicate Line Remover
Use it to clean e-mail lists, keyword lists, IDs, URLs or log lines. It compares whole lines only; to find duplicates inside a line or across columns of a CSV, use a spreadsheet instead.
How Duplicate Line Remover works
Lines are processed top to bottom. Each line (trimmed if that option is on) is turned into a comparison key – lower-cased unless case-sensitive matching is selected – and stored in a Set. A line whose key is already in the set is dropped; otherwise it is kept. Because only the first occurrence is kept, the order of the remaining lines is unchanged. With trimming on, the output contains the trimmed lines.
Common Use Cases
- Cleaning a mailing list before an import
- Removing repeated keywords from an SEO keyword list
- Getting unique values from a pasted column
- De-duplicating URLs or IDs copied from logs
FAQs
Does it keep the original order?
Yes. The first occurrence of each line stays in place and later repeats are removed. Use the Text Sorter if you also want to sort.
Which version of a duplicate is kept when case is ignored?
The first one. With "a" followed by "A", the output keeps "a".
Are empty lines treated as duplicates?
With Remove empty lines on, all empty lines are removed. With it off, the first empty line is kept and later ones are removed as duplicates.