#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2011
    Posts
    3
    Rep Power
    0
    hello all, I have an input file with four columns like this with a lot of lines 2GOX03.
  2. #2
  3. Sarcky
    Devshed Supreme Being (6500+ posts)

    Join Date
    Oct 2006
    Location
    Pennsylvania, USA
    Posts
    10,908
    Rep Power
    6351
    Welcome to the forums. Please read the forum rules and expand upon your question so it makes sense.
    HEY! YOU! Read the New User Guide and Forum Rules

    "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin

    "The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002

    Think we're being rude? Maybe you asked a bad question or you're a Help Vampire. Trying to argue intelligently? Please read this.
  4. #3
  5. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,854
    Rep Power
    481

    Learn gawk. (awk, nawk you'll have some flavor available to you)


    sort | uniq
    removes duplicate lines but may change the order of lines.

    If you want to discard lines with a duplicate column entry gawk is quite good for this. If the other columns differed how would you merge them?

    Suppose lines 3 and 6 have duplicate column 3 entries, and you want therefor neither line 3 nor line 6. gawk is good for this too.

IMN logo majestic logo threadwatch logo seochat tools logo