It seems like an endless quest.
Some years ago everyone was writing bots, kind of like a rite of passage.
It also seems like everyone was making a duplicate remover.
Etc...
Now we're back at writing bots.
And you're bringing back duplicate removers.

Jokes aside, I've also written one of those, in Perl this time: https://github.com/venam/duplicate-remover
I don't remember at all the quality of the code though, it might be bad (warning).
There are probably more efficient ways to solve this taking edge cases in consideration like links (hard/soft).
EDIT: this is probably the total opposite of what z3bra proposes, full of colors and block characters, can't be used in a pipeline.