Eliminate duplications through a bash pipe 2010-09-27

Problem:
I receive data from a continuous stream (a tail -f or rsstail output for example). I want to remove duplications. | sort | uniq won't work since it's a non-ending stream (and I don't want to sort them). So much drama.

Solution: Launch Vim and code.

Result, tadaaa:

#!/usr/bin/python

import sys

anti_doublons = set()
a = sys.stdin.readline()
while a:
    if a not in anti_doublons:
        sys.stdout.write(a)
        sys.stdout.flush()
        anti_doublons.add(a)
    a = sys.stdin.readline()

(I will create a pipe/stream toolkit to distribute all this kind of stuff to play with bash pipe in the next days).