Working with large files?

Dev Shed Forums Sponsor:

February 12th, 2013, 12:56 PM

paulh1983

Contributing User

Join Date: Dec 2004

Posts: 2,237

Time spent in forums: 2 Weeks 1 Day 11 h 3 m 46 sec
Reputation Power: 201

PHP-General - Working with large files?

So i have a large file, 14MB and i want to work on the content (emails on a separate line).

I have tried increasing the memory limit but it seems my host has set a maximum and i am unable to increase it as trying different values result in same amount of memory error i.e. tried to allocate 90MB~ blah blah..

so i have tried different ways:
1. file-get
2.fget to read one line at at ime
3.file("filename")
4. using form to input the email

but none of these work.. please let me know the best way short of splitting the file?

thanks

February 12th, 2013, 01:26 PM

gw1500se

Contributing User

Join Date: Jul 2003

Posts: 2,885

Time spent in forums: 1 Year 2 Weeks 3 Days 8 h 17 m 9 sec
Reputation Power: 581

"None of these work" is not much to go on. What did you try (post the code) and what didn't happen or what error did you get?

P.S. There is no limit per se in PHP. The total memory used by PHP is what is limited but that is changeable.

__________________
There are 10 kinds of people in the world. Those that understand binary and those that don't.

February 12th, 2013, 01:31 PM

ManiacDan

Likely to be eaten by a grue.

Dev Shed God 10th Plane (9500 - 9999 posts)

Join Date: Oct 2006

Location: Pennsylvania, USA

Posts: 9,811

Folding Points: 127430 Folding Title: Super Ultimate Folder - Level 1

Time spent in forums: 2 Months 3 Weeks 19 h 13 m 52 sec
Reputation Power: 6112

Show your fget code. fget (plus maybe fseek) is the only really correct solution.

__________________
HEY! YOU! Read the New User Guide and Forum Rules

"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin

"The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002

Think we're being rude? Maybe you asked a bad question or you're a Help Vampire. Trying to argue intelligently? Please read this.

February 12th, 2013, 03:15 PM

paulh1983

Contributing User

Join Date: Dec 2004

Posts: 2,237

Time spent in forums: 2 Weeks 1 Day 11 h 3 m 46 sec
Reputation Power: 201

sorry, here is the code.

Code:

$file = file_get_contents($_GET['file'].".csv");
    
    $emails = explode("\n",$file);
    
    $i = 0;
    $new_file = array();
    
    foreach( $emails as $email ) {
        $hashed_email = md5($email);
        $new_file[] = $email.",".$hashed_email;
    }
    
    $new_file = implode("\r\n",$new_file);
    file_put_contents($_GET['file']."-modified.csv",$new_file);

another way:

Code:

$handle = @fopen($_GET['file'].".csv", "r");
$emails = array();

if ($handle) {
    while (($buffer = fgets($handle, 4096)) !== false) {
        $emails[] = $buffer."<br/>";
    }
    if (!feof($handle)) {
        echo "Error: unexpected fgets() fail\n";
    }
    fclose($handle);

The error is the PHP's maximum limit reached error:

Fatal error: Allowed memory size of 94371840 bytes exhausted (tried to allocate 71 bytes) in .... on line 8

February 12th, 2013, 03:40 PM

ManiacDan

Likely to be eaten by a grue.

Join Date: Oct 2006

Location: Pennsylvania, USA

Posts: 9,811

Time spent in forums: 2 Months 3 Weeks 19 h 13 m 52 sec
Reputation Power: 6112

The purpose of using fgets is so that you don't hold the whole file in memory. Both of your examples...hold the whole file in memory.

Do your processing inside this loop:

PHP Code:


		
			
 $handle = @fopen($_GET['file'].".csv", "r");



if ($handle) {

    while (($buffer = fgets($handle, 4096)) !== false) {

        mail($buffer, 'no-reply@yoursite.com', 'You were in our file!', 'Congratulations, you were in the file we were processing, have a nice day!');

    }

    fclose($handle);

}

February 12th, 2013, 03:54 PM

paulh1983

Contributing User

Join Date: Dec 2004

Posts: 2,237

Time spent in forums: 2 Weeks 1 Day 11 h 3 m 46 sec
Reputation Power: 201

i dont see how your code is different than mine? but to give you the benefit, i tried the following and still got the fatal execution.. the thing is at HOME on my LAMP, i have increased the memory but still nothing is happening?

I have even tried to run the script on CLI which i thought was better? but still same error

Code:

<?php


    $handle = @fopen($_GET['file'].".csv", "r");

    if ($handle) {
        while (($buffer = fgets($handle, 4096)) !== false) {
           $hashed_email = md5($buffer);
            $new_file[] = $buffer.",".$hashed_email;
        }
        fclose($handle);
    }  
    
    $new_file = implode("\r\n",$new_file);
    file_put_contents($_GET['file']."-modified.csv",$new_file);

?>

February 12th, 2013, 04:08 PM

requinix

Still alive

Join Date: Mar 2007

Location: Washington, USA

Posts: 12,717

Folding Points: 417516 Folding Title: Super Ultimate Folder - Level 1

Time spent in forums: 5 Months 1 Week 4 Days 7 h 21 m 39 sec
Reputation Power: 8969

Send a message via Google Talk to requinix

The difference is that his does the action for each individual line in the file as they're being read in, while yours tries to build a giant array of everything and then sends one email with them all.

Your script should only use one loop and only fopen/fread/fwrite/fclose. No file_get_contents(), no file_put_contents(), no array that you build up in one loop then tear down in another loop.

__________________
Site rules and FAQ | Project Suggestions | mod_rewrite guide | PHP New User Guide and FAQ - PHP Stickies | Regex Resources

February 12th, 2013, 06:40 PM

E-Oreo

Lost in code

Join Date: Dec 2004

Posts: 7,944

Folding Points: 945 Folding Title: Novice Folder

Time spent in forums: 2 Months 10 h 17 m 56 sec
Reputation Power: 7053

Inside your while(fgets) loop you cannot have anything that appends; ie, you cannot have:

PHP Code:


		
			
 $array[] =

PHP Code:


		
			
 $string .=

Otherwise you will end up appending the entire contents of the source file into memory, which is why you are getting the memory error.

You need to overwrite all of the variables in the loop on every iteration of the loop.

In the followup code you posted you're still appending to $new_file inside the loop.

You could actually still use file_put_contents, but you have to use the FILE_APPEND flag, and you have to call it within the loop.

__________________
PHP FAQ
How to program a basic, secure login system using PHP
Connect with me on LinkedIn

Quote:

Originally Posted by Spad

Ah USB, the only rectangular connector where you have to make 3 attempts before you get it the right way around

February 13th, 2013, 03:47 AM

paulh1983

Contributing User

Join Date: Dec 2004

Posts: 2,237

Time spent in forums: 2 Weeks 1 Day 11 h 3 m 46 sec
Reputation Power: 201

cheers, now i understand.. i didnt realise that the error i was getting was due to me putting it all into an array and not because of fget...

thanks for explaining.

PHP-General - Working with large files?

Developer Shed Advertisers and Affiliates