November 20th, 2011, 02:45 AM
hello all, I have an input file with four columns like this with a lot of lines 2GOX03.
November 21st, 2011, 02:38 PM
Welcome to the forums. Please read the forum rules and expand upon your question so it makes sense.
HEY! YOU! Read the New User Guide and Forum Rules
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin
"The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002
Think we're being rude? Maybe you asked a bad question
or you're a Help Vampire.
Trying to argue intelligently? Please read this.
December 2nd, 2011, 07:39 PM
Learn gawk. (awk, nawk you'll have some flavor available to you)
sort | uniq
removes duplicate lines but may change the order of lines.
If you want to discard lines with a duplicate column entry gawk is quite good for this. If the other columns differed how would you merge them?
Suppose lines 3 and 6 have duplicate column 3 entries, and you want therefor neither line 3 nor line 6. gawk is good for this too.