I just found two new *nix commands that i've been missing:
- cut
- uniq


CUT to print second column

old:
cat $FILE | awk -f"$DELIMITOR" '{print $2}'

new:
cat $FILE | cut -d"$DELIMITOR" -f2


UNIQ to find dups

old:
traditionally to find dup is a file one might imploy some bash looping tricker:
for i in `cat $FILE | cut -d"." -f1,2 | uniq`
do
num=`cat $FILE | grep $i | wc -l`
if (( $num > 1 ))
then
echo $num $i
fi
done

now:
cat $FILE|sort |uniq -c |sort


sources:

View Original: http://www.unix.com/unix-dummies-questions-answers/10617-counting-number-duplicate-lines-file.html#ixzz0llI3UEwi