Removing duplicate lines and sorting lists the right way
To remove duplicate lines, delete every line that repeats an earlier one, and decide first whether "the same" means exactly identical or identical after cleaning. Exact removal is safe but misses most real duplicates. Trimming spaces and ignoring case catches far more, and it is what most people actually need for lists of emails, keywords or product codes.
Remove duplicate linesList has duplicatesOpenA worked example
Here is a list of six email addresses copied from a spreadsheet:
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]Three things are going on. The first and last lines are exact copies. The second line differs only by capital A. The third line has a trailing space that you cannot see. Here are the counts under each rule:
| Rule | Lines left | Why |
|---|---|---|
| Remove exact duplicates | 5 | Only the last "anna" line matches the first exactly |
| Trim spaces and ignore case | 3 | anna, ben and cara are the only unique addresses |
Email addresses are almost always case-insensitive in practice, so the second rule is the one you want here. Use extract emails and links first if the list came from a messy page, so that only real addresses reach the duplicate step.
Near-duplicates to watch for
Near-duplicates are lines that look different to a computer but mean the same thing to you. Common causes:
- Capital letters: "Blue Widget" and "blue widget".
- Trailing or leading spaces, often invisible in an editor.
- Non-breaking spaces or other hidden characters, which look like ordinary spaces (hidden characters explains them).
- Different punctuation: "e-mail" and "email".
- Plural forms, which need a human decision and should not be merged by a tool.
Clean the list in this order: trim, then ignore case where case means nothing, then remove duplicates. Keep a copy of the original, so you can check what was removed.
Case can matter
Ignoring case is not always safe. Product codes, file names on Linux and some identifiers are case-sensitive, so "AB-101" and "ab-101" may be two different items. Before you merge items that differ only by case, check whether the system you are feeding the list into treats them as the same.
Alphabetical sort versus natural sort
Sorting a list of items that contain numbers often gives an unexpected order. A plain alphabetical sort compares characters one at a time, so "item10" comes before "item2", because the character 1 comes before 2.
Here is the difference for three items:
- Alphabetical: item1, item10, item2
- Natural: item1, item2, item10
Natural sort compares the numbers as numbers, so 2 comes before 10. Use natural sort for file names, shelf codes, version numbers and anything that ends in a count. Use plain alphabetical sort for names and words, where it is what people expect. To compare both orders on your own list, paste it into sort lines and switch between A→Z and Numbers.
Two more points help:
- Sorting does not remove duplicates. Remove them first, then sort, or the sort will place duplicates next to each other, which makes them easier to see but not removed.
- Keep the original order if the list has meaning. A list of steps or a ranking should not be sorted at all.
Keeping the first occurrence
When you remove duplicates, most tools keep the first occurrence of each line and drop the later ones. That matters when lines carry extra information, such as a name and a date. If the first copy is the outdated one, sort the list by date first, or check the duplicates by hand, before you remove them.
Check the result
Count the lines before and after. In the email example, six lines became three unique addresses, so three lines were removed. If the count is lower than you expected, look at the removed lines. A duplicate removal matches the whole line, so two different people with the same name in a list of rows, or two products with the same title but different codes, can look identical once the codes are cut off. Remove duplicates from the field that defines an item, not from a field that only describes it.
Edge cases
- Blank lines. Several blank lines in a row can count as duplicates. Remove them as a separate step if you want to keep one for spacing.
- Multi-line records. Duplicate removal works line by line. If each record spans several lines, it can split a record and leave the parts in the wrong order.
- Very large lists. Large files are fine, but check the output count against the input count so you know what changed.
Quick answers
Does removing duplicate lines also sort the list?
Not necessarily. Many tools keep the original order, so sort separately if you need an order.
Why does item10 come before item2 when I sort?
Plain alphabetical sort compares one character at a time, and the character 1 comes before 2. Use natural sort, which compares the numbers as numbers.
Should I ignore case when removing duplicate emails?
Usually yes, because email addresses are treated as case-insensitive in practice. Do not ignore case for identifiers where case changes the meaning.
