
May 26th, 2001, 03:12 PM
|
|
Junior Member
|
|
Join Date: Feb 2001
Posts: 4
Time spent in forums: < 1 sec
Reputation Power: 0
|
|
|
Eliminating duplicate lines in TXT files
Anybody know of a program to eliminate duplicate lines from CSV files ( usually ranging from 1-20MB)? I've found one but it is actually a HTML parser and doesn't support files over 2.5MB in size.
ie
Barbara Meyers,Associate,97401
Barbara Yvette Santiago,Member,90210
Barbara J Green,Associate,67541
Barbara Jean Hall,Moderator,43521
Barbara Yvette Santiago,Member,90210
Barbara Yvette Santiago,Member,90210
Barbara Jean Hall,Moderator,43521
Barbara Yvette Santiago,Member,90210
after being parsed I'd want all original records intact, but the exact duplicates removed.
Barbara Meyers,Associate,97401
Barbara Yvette Santiago,Member,90210
Barbara J Green,Associate,67541
Barbara Jean Hall,Moderator,43521
Thanks,
Zach Sniezko
|