Eliminate duplications through a bash pipe 2010-09-27

I receive data from a continuous stream (a tail -f or rsstail output for example). I want to remove duplications. | sort | uniq won't work since it's a non-ending stream (and I don't want to sort them). So much drama.

Solution: Launch Vim and code.

Result, tadaaa:


import sys

anti_doublons = set()
a = sys.stdin.readline()
while a:
    if a not in anti_doublons:
    a = sys.stdin.readline()

(I will create a pipe/stream toolkit to distribute all this kind of stuff to play with bash pipe in the next days).