#1
  1. No Profile Picture
    Contributing User
    Devshed Loyal (3000 - 3499 posts)

    Join Date
    Dec 2004
    Posts
    3,031
    Rep Power
    377

    Working with large files?


    So i have a large file, 14MB and i want to work on the content (emails on a separate line).

    I have tried increasing the memory limit but it seems my host has set a maximum and i am unable to increase it as trying different values result in same amount of memory error i.e. tried to allocate 90MB~ blah blah..

    so i have tried different ways:
    1. file-get
    2.fget to read one line at at ime
    3.file("filename")
    4. using form to input the email

    but none of these work.. please let me know the best way short of splitting the file?

    thanks
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Expert (3500 - 3999 posts)

    Join Date
    Jul 2003
    Posts
    3,617
    Rep Power
    595
    "None of these work" is not much to go on. What did you try (post the code) and what didn't happen or what error did you get?

    P.S. There is no limit per se in PHP. The total memory used by PHP is what is limited but that is changeable.
    There are 10 kinds of people in the world. Those that understand binary and those that don't.
  4. #3
  5. Sarcky
    Devshed Supreme Being (6500+ posts)

    Join Date
    Oct 2006
    Location
    Pennsylvania, USA
    Posts
    10,908
    Rep Power
    6352
    Show your fget code. fget (plus maybe fseek) is the only really correct solution.
    HEY! YOU! Read the New User Guide and Forum Rules

    "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin

    "The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002

    Think we're being rude? Maybe you asked a bad question or you're a Help Vampire. Trying to argue intelligently? Please read this.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Loyal (3000 - 3499 posts)

    Join Date
    Dec 2004
    Posts
    3,031
    Rep Power
    377
    sorry, here is the code.

    Code:
    $file = file_get_contents($_GET['file'].".csv");
        
        $emails = explode("\n",$file);
        
        $i = 0;
        $new_file = array();
        
        foreach( $emails as $email ) {
            $hashed_email = md5($email);
            $new_file[] = $email.",".$hashed_email;
        }
        
        $new_file = implode("\r\n",$new_file);
        file_put_contents($_GET['file']."-modified.csv",$new_file);
    another way:
    Code:
    $handle = @fopen($_GET['file'].".csv", "r");
    $emails = array();
    
    if ($handle) {
        while (($buffer = fgets($handle, 4096)) !== false) {
            $emails[] = $buffer."<br/>";
        }
        if (!feof($handle)) {
            echo "Error: unexpected fgets() fail\n";
        }
        fclose($handle);
    The error is the PHP's maximum limit reached error:

    Fatal error: Allowed memory size of 94371840 bytes exhausted (tried to allocate 71 bytes) in .... on line 8
  8. #5
  9. Sarcky
    Devshed Supreme Being (6500+ posts)

    Join Date
    Oct 2006
    Location
    Pennsylvania, USA
    Posts
    10,908
    Rep Power
    6352
    The purpose of using fgets is so that you don't hold the whole file in memory. Both of your examples...hold the whole file in memory.

    Do your processing inside this loop:

    PHP Code:
    $handle = @fopen($_GET['file'].".csv""r");

    if (
    $handle) {
        while ((
    $buffer fgets($handle4096)) !== false) {
            
    mail($buffer'no-reply@yoursite.com''You were in our file!''Congratulations, you were in the file we were processing, have a nice day!');
        }
        
    fclose($handle);

    HEY! YOU! Read the New User Guide and Forum Rules

    "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin

    "The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002

    Think we're being rude? Maybe you asked a bad question or you're a Help Vampire. Trying to argue intelligently? Please read this.
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Loyal (3000 - 3499 posts)

    Join Date
    Dec 2004
    Posts
    3,031
    Rep Power
    377
    i dont see how your code is different than mine? but to give you the benefit, i tried the following and still got the fatal execution.. the thing is at HOME on my LAMP, i have increased the memory but still nothing is happening?

    I have even tried to run the script on CLI which i thought was better? but still same error
    Code:
    <?php
    
    
        $handle = @fopen($_GET['file'].".csv", "r");
    
        if ($handle) {
            while (($buffer = fgets($handle, 4096)) !== false) {
               $hashed_email = md5($buffer);
                $new_file[] = $buffer.",".$hashed_email;
            }
            fclose($handle);
        }  
        
        $new_file = implode("\r\n",$new_file);
        file_put_contents($_GET['file']."-modified.csv",$new_file);
    
    ?>
  12. #7
  13. Jealous Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,302
    Rep Power
    9400
    The difference is that his does the action for each individual line in the file as they're being read in, while yours tries to build a giant array of everything and then sends one email with them all.

    Your script should only use one loop and only fopen/fread/fwrite/fclose. No file_get_contents(), no file_put_contents(), no array that you build up in one loop then tear down in another loop.
  14. #8
  15. No Profile Picture
    Lost in code
    Devshed Supreme Being (6500+ posts)

    Join Date
    Dec 2004
    Posts
    8,316
    Rep Power
    7171
    Inside your while(fgets) loop you cannot have anything that appends; ie, you cannot have:
    PHP Code:
    $array[] = 
    or
    PHP Code:
    $string .= 
    Otherwise you will end up appending the entire contents of the source file into memory, which is why you are getting the memory error.

    You need to overwrite all of the variables in the loop on every iteration of the loop.

    In the followup code you posted you're still appending to $new_file inside the loop.

    You could actually still use file_put_contents, but you have to use the FILE_APPEND flag, and you have to call it within the loop.
    PHP FAQ

    Originally Posted by Spad
    Ah USB, the only rectangular connector where you have to make 3 attempts before you get it the right way around
  16. #9
  17. No Profile Picture
    Contributing User
    Devshed Loyal (3000 - 3499 posts)

    Join Date
    Dec 2004
    Posts
    3,031
    Rep Power
    377
    cheers, now i understand.. i didnt realise that the error i was getting was due to me putting it all into an array and not because of fget...

    thanks for explaining.

IMN logo majestic logo threadwatch logo seochat tools logo