March 25th, 2011, 12:35 PM
Combining Files issue
I am in need of a method and commands to do the following. I have 18,000 files that all start with the same naming convention. They have a counter at the end.
So I have file1.txt - file18000.txt
I need to place sections of these files into their own file. I don't want to put them all into a single file. Just subsets.
i.e. I don't want to use the *. cat files*.txt > bigfile.txt
I don't think this will work without the extra work of creating a filelist and then trying to break it down by ftp'ing it and seperating the files.
ls testfile*.txt > filelist1
cat filelist1 | xargs cat >> bigfile1.txt
I would like to try something more like this below but I don't know these commands well enough to put it together. Such as regular expressions which might be the answer but not sure.
Anyone have a command that will accomplish the following by grouping the range of files?
<grab all files filecountx and put into file list>
<run through filelist0 and grab *1 thru *1000 (since filename is the same except the count) and place into Filelist1>
< run through filelist0 and grab *1001 thru *1500 and placeinto Filelist2>
<grab filelist1 and maybe filelist2 (depending on size of files) and create Bigfile1>
<Then do same for the rest into separate Bigfiles>
March 29th, 2011, 07:45 AM
for fn in `ls -1|egrep 'File[0-9]+'`
(( i += 1 ))
if [ $i -gt $nf ]
(( bf+= 1 ))
cat $fn >>$bigfn
March 29th, 2011, 08:14 AM
Thanks, I'll give it a try!