Text Deduplicator
Remove duplicate lines from any list with options for trimming whitespace, ignoring case and dropping empty lines. Shows before/after counts and what was removed.
Notes
Uniqueness is decided on the cleaned value (after trim and case-folding, if enabled) but the original first occurrence is what gets output — “Keep original text” is automatic. Empty lines are considered duplicates of each other and by default removed entirely. Sorting happens after deduplication.
How to use Text Deduplicator
Paste lines of text and the duplicates vanish in real time, preserving first-seen order by default. Options cover case-insensitive comparison, trim-before-compare (so trailing spaces do not cheat identity) and ignore-blank-lines, with live counters showing how many lines were removed.
The sorted-output toggle gives a deduplicated, ordered view ready for checklists and logs alike. Useful well beyond logs: cleaning keyword lists, merging export files where repeated rows are common, or flattening survey responses before they go into another system.
Frequently asked questions
What exactly gets removed?
Exact duplicate lines, or repeated words in word mode. Whitespace-only duplicates are removed too, and the original count is reported so you can compare before and after.
Is the comparison case sensitive?
By default lines must match exactly. A case-insensitive option folds A-Z, so duplicates that differ only in capitalization merge as well.
Can I keep first-occurrence order?
Yes. The default preserves the order of first appearance, which keeps log chronology intact. A sorted-output mode is available when you need canonical ordering instead.
What do people use it for?
Cleaning scraped email lists, merging overlapping exports, removing repeated error lines from logs and collapsing duplicated terms before translation work.