#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Posts
    3
    Rep Power
    0

    Memory overusage and functions : Allowed memory size exhausted


    I have a data processing script that I execute using a cron on a daily basis. The amount of data varies each day.

    The data process code is broken down into separate files which helps ( me, anyway ) manage and maintain the code better.

    The files are all run from a central script like this :

    Code:
    for($q=1;$q<=30;$q++){
    	include "/usr/www/users/xxx/function" . $q . ".php";
    }
    This is working very well for most the time, but every once in a while, I get an error :

    PHP Fatal error: Allowed memory size of 67108864 bytes exhausted

    So from my understanding, I have something that is chewing up the memory. I know that ultimately I would need to go through the code and find the cause, but in the mean-time, would the following be a solution :

    To convert each of the separate 'include' files into a function, and then call the function. This way I think that all the memory used in each function would be unset at the end of each function, and the chances of an accumulated memory overload would be severely reduced.
    There are only 2 variables that I need to carry over into each of the 'include' files.

    So would the following work :

    Code:
    $var1 = "Variable 1";
    $var2 = "Variable 2";
    
    for($q=1;$q<=30;$q++){
    	if($q == 1){
    		function MyFunction1(){
    			global $var1;
    			global $var2;
    			include "/usr/www/users/xxx/function" . $q . ".php";
    		}
    		MyFunction1();
    		// at this point, all variables created in MyFunction1 will be unset ?
    	}	
    
    	// repeat the above IF() block for all 30 'include' files	
    	
    }
  2. #2
  3. Did you steal it?
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    13,997
    Rep Power
    9397
    You might not need to make things that complicated.

    What do those function*.php files look like? Have you tried unset()ing variables once you're done with them?
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Posts
    3
    Rep Power
    0
    Originally Posted by requinix
    You might not need to make things that complicated.

    What do those function*.php files look like? Have you tried unset()ing variables once you're done with them?
    Many Thanks for the reply.

    Some of them are simple code blocks of only 50 lines with only 1 or 2 variables, while others are 100's of lines, and include other files that deal with e-mailing, managing xml files, creating pdf's, etc.

    My point is that there is so much code it is going to take me a while to narrow down where the leak is happening, especially as it is intermitant and only happens when I have a larger than usual amount of data to process.

    So my idea of a short term solution is to unset all the variables at the end of each section. I think that running each section in a function would achieve this without me having to work through all the code to make a list of what needs to be unset.

    I was also thinking that at the end of each function I could add :
    Code:
    echo "Memory Limit : " . ini_get("memory_limit") . "<br>";
    echo "Memory Usage : " . memory_get_usage() . "<br>";
    so that I can start getting an idea of where the usage is highest.

    Regards
    Dave
  6. #4
  7. Wiser? Not exactly.
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    May 2001
    Location
    Bonita Springs, FL
    Posts
    5,938
    Rep Power
    4033
    It might help. It might not. If you want to try it just make a function then pass in the file name to include, eg:
    Code:
    function RunFile($f){
        include $f;
    }
    
    for($q=1;$q<=30;$q++){
          RunFile("/usr/www/users/xxx/function" . $q . ".php");
    }
    You could also run each script as a separate process by using exec() them rather than including them.


    What you should try and do is save one of your datasets that triggers the error, then start debugging the scripts using that dataset. The error message will give you the file and line# that the error occurred on, use that as a starting point and start tracing backward through the code.
    Recycle your old CD's, don't just trash them



    If I helped you out, show some love with some reputation, or tip with Bitcoins to 1N645HfYf63UbcvxajLKiSKpYHAq2Zxud
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Posts
    3
    Rep Power
    0
    Hi Kicken

    Thanks for the suggestion - a much cleaner idea than I was thinking of, and the variables used in the function will always be unset after each call of the function.

    I did find the point in the error log where the code stopped, but I think that doesn't tell me what was causing the memory increase - it could have been any code prior to that point, and the point where it stopped was just the straw on the camels back.

    I will try your idea, and also echo the memory_get_usage() before the end of each function, so I can see which ones are affecting memory by large amounts.

    Regards
    Dave

IMN logo majestic logo threadwatch logo seochat tools logo