#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2011
    Posts
    2
    Rep Power
    0

    Combining Files issue


    Hi all,

    I am in need of a method and commands to do the following. I have 18,000 files that all start with the same naming convention. They have a counter at the end.
    So I have file1.txt - file18000.txt

    I need to place sections of these files into their own file. I don't want to put them all into a single file. Just subsets.
    i.e. I don't want to use the *. cat files*.txt > bigfile.txt

    I don't think this will work without the extra work of creating a filelist and then trying to break it down by ftp'ing it and seperating the files.

    ls testfile*.txt > filelist1
    cat filelist1 | xargs cat >> bigfile1.txt

    I would like to try something more like this below but I don't know these commands well enough to put it together. Such as regular expressions which might be the answer but not sure.

    Anyone have a command that will accomplish the following by grouping the range of files?

    <grab all files filecountx and put into file list>
    <run through filelist0 and grab *1 thru *1000 (since filename is the same except the count) and place into Filelist1>
    < run through filelist0 and grab *1001 thru *1500 and placeinto Filelist2>
    <grab filelist1 and maybe filelist2 (depending on size of files) and create Bigfile1>
    <Then do same for the rest into separate Bigfiles>

    Thanks!
  2. #2
  3. Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Sep 2006
    Posts
    841
    Rep Power
    387

    Cool


    Try this:
    Code:
    nf=1000
    i=$nf
    bf=0
    for fn in `ls -1|egrep 'File[0-9]+'`
    do
      (( i += 1 ))
      if [ $i -gt $nf ]
      then
        (( bf+= 1 ))
        bigfn="BigFile$bf"
        i=1
      fi
      cat $fn >>$bigfn
    done
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2011
    Posts
    2
    Rep Power
    0
    Thanks, I'll give it a try!

IMN logo majestic logo threadwatch logo seochat tools logo