1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2011
    Rep Power
    hello all, I have an input file with four columns like this with a lot of lines 2GOX03.
  2. #2
  3. Sarcky
    Devshed Supreme Being (6500+ posts)

    Join Date
    Oct 2006
    Pennsylvania, USA
    Rep Power
    Welcome to the forums. Please read the forum rules and expand upon your question so it makes sense.
    HEY! YOU! Read the New User Guide and Forum Rules

    "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin

    "The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002

    Think we're being rude? Maybe you asked a bad question or you're a Help Vampire. Trying to argue intelligently? Please read this.
  4. #3
  5. Contributing User

    Join Date
    Aug 2011
    Rep Power

    Learn gawk. (awk, nawk you'll have some flavor available to you)

    sort | uniq
    removes duplicate lines but may change the order of lines.

    If you want to discard lines with a duplicate column entry gawk is quite good for this. If the other columns differed how would you merge them?

    Suppose lines 3 and 6 have duplicate column 3 entries, and you want therefor neither line 3 nor line 6. gawk is good for this too.

IMN logo majestic logo threadwatch logo seochat tools logo