Site Copier/Mover (non FTP)

Dev Shed Forums Sponsor:

December 5th, 2002, 03:23 PM

Webbywarehouse

Contributing User

Join Date: Dec 2002

Posts: 53

Time spent in forums: 1 h 40 m 15 sec
Reputation Power: 6

Site Copier/Mover (non FTP)

Somewhat newer working with PHP. I have successfully written a script that will upload a file from a URL address (i.e. http://www.webbywarehouse.com/image.gif ).

For my clients I will need a site mover/copier where they provide an address such as http://www.wtv-zone.com/woordrack/bars, which btw is a one of the sites/directories that I need to move, and the script opens the URL given and copies the contents of the given directory.

I need to be able to copy ONLY from directories where directory lists are enabled. In the above mentioned site it is on an Apache server with DL enabled.

I wrote a script to read the contents of the directory but how to then move the files, typically only images and sound files, to my clients new locaiton at my site/server.

To be honest, I don't even know where to begin. Any help on how to start, complete, approach this would be greatly appreciated. I know for many this is not hard, for me though it is a leap.

Thank you for your assistance.

__________________
- Brian

December 5th, 2002, 03:57 PM

kicken

Wiser? Not exactly.

Join Date: May 2001

Location: Ft Myers, FL

Posts: 4,149

Folding Points: 115670 Folding Title: Super Ultimate Folder - Level 1

Time spent in forums: 1 Month 2 Weeks 6 Days 16 h 9 m 13 sec
Reputation Power: 546

PHP Code:


		
			
 $site='http://www.wtv-zone.com/woordrack/bars';

$savedir='';

$files=array();



$con=file($site);

//Do something to parse the html page into the different links you want to copy over and get them info the $files array.  You mentioned you had this part, so I won't bother.





foreach ($files as $url){

  $savepath=$savedir.basename($url);

  $remote=fopen($url, 'rb');

  $local=fopen($savepath, 'wb');

  while($data=fread($remote, 2048)){

    fwrite($local, $data);

  }

  fclose($remote);

  fclose($local);

}

That should do the trick.

December 5th, 2002, 04:16 PM

Webbywarehouse

Contributing User

Join Date: Dec 2002

Posts: 53

Time spent in forums: 1 h 40 m 15 sec
Reputation Power: 6

Thank you.

Thank you for you very speedy reply and assistance.

I may have mis spoken, I do not know how to parse the the contents in such a way that I can then either transfer all of the files in the directory or create a list of the files.

Honestly, I just want to be able to offer a very basic script, nothing fancy. I think they could simple drop the URL of the directory in and then the files within that directory then be copied, that is fine.

So, I can now read the directory, yes. Where I get lost still is what to do with the contents of the file that I am reading. In my URL uploader it is so easy becuase the user provides the exact file name withing the URL, but with this, there is no filename given.

So, should I then seem the best way to parse the file or is there a better way to do this?

Thanks again for your assistance. I hope this benefits others as well.

December 5th, 2002, 07:49 PM

kicken

Wiser? Not exactly.

Join Date: May 2001

Location: Ft Myers, FL

Posts: 4,149

Time spent in forums: 1 Month 2 Weeks 6 Days 16 h 9 m 13 sec
Reputation Power: 546

This is a script for the command line. It should give you the idea of how to do it though and you can make a web front for it.

[I]Copydir.php[I]

PHP Code:


		
			
<?php

if (!isset($_SERVER['argv'][1])){

        echo "Usage: {$_SERVER['argv'][0]} <url>[ <save path>]\n";

        exit;

}





$url=$_SERVER['argv'][1];

$savepath=(isset($_SERVER['argv'][2]))?$_SERVER['argv'][2]:'./';



//Make sure save path has a / on the end.

if ($savepath{strlen($savepath)-1} != '/' && $savepath{strlen($savepath)-1} != '\\'){

        $savepath.='/';

}



//Make sure url path has a / on the end.

if ($url{strlen($url)-1} != '/' && $url{strlen($url)-1} != '\\'){

        $url.='/';

}



$con=implode('', file($url)); //Get the directory listing for the given url into a string.



//Remove the html tags from the dir, except the <a> tags.  Everything else is not needed.

$con=strip_tags($con, '<A>');



//Get all the HREF="..." attribute values from the A tags.

//Ignore values starting with / (parent directory)  and ? (sorting functions)

$files=array();

preg_match_all('/HREF="([^\/?].[^"]*)"/', $con, $files);





//Now, loop through the files, fetch them from the URL, and save them locally.

foreach ($files[1] as $file){

        $remotefile=$url.$file;

        //Make sure url path has a / on the end.

        if ($remotefile{strlen($remotefile)-1} == '/' || $remotefile{strlen($remotefile)-1} == '\\'){

                echo "$remotefile is a directory, skipping\n";

                continue;

        }

        $currentSavePath=$savepath.basename($remotefile);



        //Open the remote file for reading only.  Use the 'b' mode for binary safe.

        $remote=fopen($remotefile, 'rb');

        //Open the local file for writting only.  Use the 'b' mode for binary safe.

        $local=fopen($currentSavePath, 'wb');

        //Read the data in 2K intervals and write it to the local file

        while ($data=fread($remote, 2048)){

                fwrite($local, $data);

        }



        //Close both files.

        fclose($remote);

        fclose($local);



        //Print a done message.

        echo "Done copying file $remotefile to $currentSavePath\n";

}

echo "Done copying all files.\n";

?>

December 5th, 2002, 08:22 PM

AlCapone

Mobbing Gangster

Join Date: Sep 2001

Location: "Best City" 2002 and 2003- Melbourne, Australia

Posts: 4,913

Time spent in forums: 5 h 36 m 31 sec
Reputation Power: 18

Not to be nitpicking, kicken, but fopen() is extremely ineficient comparing to fsockopen()

__________________
And you know I mean that.

December 5th, 2002, 10:18 PM

kicken

Wiser? Not exactly.

Join Date: May 2001

Location: Ft Myers, FL

Posts: 4,149

Time spent in forums: 1 Month 2 Weeks 6 Days 16 h 9 m 13 sec
Reputation Power: 546

Yea, for most the stuff I do w/ http and php I use fsockopen myself, but for the sake of simplicity, I figured it'd be eaiser to just use fopen than have to teach the HTTP protocol as well as PHP.

December 6th, 2002, 01:22 AM

Webbywarehouse

Contributing User

Join Date: Dec 2002

Posts: 53

Time spent in forums: 1 h 40 m 15 sec
Reputation Power: 6

Thank you again.

While the script that you provided does seem for me overwhelming, I am not proud nor shy that I am a newbie to PHP.

I see how the extraction of the individual files is done but you know how it is with newbies it is actually putting together the puzzle and getting the smaller pieces into their places.

I did until recently struggle with putting data into arrays, extracting it later, creating scripts with several switch() actions but now it is easy. My latest file manager for Webbywarehouse is IMHO very robuhst. But I worked my tail off to get it. And certainly your programming would run circles around mine.

Not being proud help me to connect the dots if you don't mind. Here is how I would open directory and create the initial file (containing the hrefs that we want to parse).

<?php
function file_get_contents($filename)
{
$fp = @fopen($filename, "r");
$fpsize = filesize($fp);
if (!($fp))
{
return 0;
}
/*
This ensures maximum file size, if $fp = "" it was too big!!! Though not needed for this example I suppose. I put it in for uploading images and limitting their size or I would have HUGE bitmaps.
*/
$temp .= fread($fp, 131072);
if (!feof($fp))
{
$temp="";
}
fclose ($fp);
return $temp;
}
if ($getcontents)
{
$data=file_get_contents("$remote_file");
if ($data != "")
{
$rfilesize = filesize($temp);
$fp=fopen(" $local_file","wb");
fputs($fp,$data,strlen($data));
fclose($fp);
$rext = pathinfo($local_file);
$rr = $rext['extension'];
}
else
{
$toobig="1";
}
}
else
{
?>
<html>
<head>
<title>start</title>
</head>
<body>
<form name="getcontents" action="<?php echo $PHP_SELF; ?>">
<input name="remote_file" value="http://">
<input type="hidden" name="getcontents" value="go">
<input type="submit" value="Get the file!!!">
</form>
</body>
</html>
<?php
}
?>
<html>
<head>
<title>finish</title>
</head>
<body>
<?php
echo $data;
?>
</body>
</html>

This is taken from the actaul script that I use for my URL uploader for Webbywarehouse and does work pretty well I guess. Nothing to write home about though.

Anyway, once this file is read, and it is not too big, I don't see how I can then parse this with the script that you have generously shared. That is where I am lost. Oh, just a quick follow up, how is fsockopen() better, in a nut shell. Once I have mastered this then I can move on to a better method. I love this stuff you know.

Thank you for sticking with me on this one.

Last edited by Webbywarehouse : December 6th, 2002 at 02:04 AM.

December 6th, 2002, 02:17 AM

kicken

Wiser? Not exactly.

Join Date: May 2001

Location: Ft Myers, FL

Posts: 4,149

Time spent in forums: 1 Month 2 Weeks 6 Days 16 h 9 m 13 sec
Reputation Power: 546

With what you posted, there are a couple problems I see of hand that you will need to look at.

Quote:

function file_get_contents($filename)

Such a function like that already exists, so you would get fatal errors trying to use that. If you arn't now, it may be because you are using an older php version.

Quote:

$fpsize = filesize($fp);

Here there are a couple things. One, you don't use a handle from fopen() in filesize(), you will give it the path to the actual file, like filesize('myfile.txt');

Secondly, in this particular situation, it won't work at all. filesize() will only work on local files, not remote ones. That kind of information is usually not avaiable, and even when it is, php won't get it.

Quote:

$temp .= fread($fp, 131072);

This isn't really a problem, but since $temp doesn't already exist, you should just use $temp = fread() rather than $temp .= fread();

Ok, now with that said, about your questions.

Quote:

While the script that you provided does seem for me overwhelming, I am not proud nor shy that I am a newbie to PHP

Nothing wrong with being a newbie, everyone is a one point or another

. If by chance I offended you w/ the post about takeing the easy way, didn't mean to. Just wanted to keep it simple enough that hopefully there wouldn't be a lot of confusion.

Quote:

I don't see how I can then parse this with the script that you have generously shared.

Getting the file list is going to be the more complicated part. I used some regular expressions to get them which, if you don't know much about it, would be a really good thing to learn.

PHP Code:


		
			
 preg_match_all('/HREF="([^/?].[^"]*)"/', $con, $files);

That is the line in my script that actuall extracts the files to download from the listing ($con). The results of running the regex are put into the $files array for you to use. The regex itself is fairly simple:
all perl style regex's need a dilimiter, so that is what the /'s on both sides are for.
the HREF=" part will tell the regex to look for that string
the () around the next part will tell php to store the inner information into the $files array.
the [^\/?] (devshed removed the \) part is to tell php to match any character EXCEPT / or ?.
the . tells php to match any character.
the [^"]* will tell php to match any number of characters, so long as one of them is not a "
Finally, the last " finishes the HREF attribute value.

After php is done processing things, the $files array should contain two more arrays, one containing everything matched and one containing the stuff that we told it to keep using the ()'s in the regex. What we told it to keep is stored in $files[1] (you can do a print_r($files) to see how this is setup)

Quote:

foreach ($files[1] as $file){
$remotefile=$url.$file;
//Make sure url path has a / on the end.
......
//Print a done message.
echo "Done copying file $remotefile to $currentSavePathn";
}

This section loops though the captured filenames and will pull them off the server using fopen(). I added a couple checks in there to make sure paths end in a / or \, even though they probably arn't needed.

Finally, my script isn't really a ready to use thing that you can plugin to yours, it will need to be adapted, but all the more power to ya, good learning experience. Good luck w/ your project.

December 6th, 2002, 04:50 PM

Webbywarehouse

Contributing User

Join Date: Dec 2002

Posts: 53

Time spent in forums: 1 h 40 m 15 sec
Reputation Power: 6

Light at the end of the tunnel.

Thank you again, I see that there is light at the end of the tunnel. Your break down is very helpful and the time you spent is appreciated.

I have not as much time each day as I would like but over the next few days I will be able to use what you have shown me to create the basic script that I need. I am confident of that.

I will post another follow up at that point. Thank you again for your help. I really was stuck.

Take care.

#10

December 8th, 2002, 07:46 PM

Webbywarehouse

Contributing User

Join Date: Dec 2002

Posts: 53

Time spent in forums: 1 h 40 m 15 sec
Reputation Power: 6

GOT IT!!!!!!!!!!

kicken Thank you so much for all your help. I got my uploader working BUT I have another issue now.

I do READ the remote fil