Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. A Change of Season
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2004
    Location
    Next Door
    Posts
    2,689
    Rep Power
    171

    How big an array can php handle?


    Hello;

    I have 2 related questions I'd appreciate if you giveme some direction:

    I wrote this code so I can enter products to a table where I can practice mysql indexing. There are now 12 million products so far. Each time I run the script I have to stop at around 3 hundred thousand otherwise my computer starts making noises and gets ready hot!!

    PHP Code:
    while($count<300000)
        {
            
    $count++;
            
    $code substr(str_shuffle("ABCDEFGHIJKLMNOPQRSTUVWXYZ"), 04);
            
    $class substr(str_shuffle("123456789"), 01);
            
    $category substr(str_shuffle("ABCDEFGHIJKLMNOPQRSTUVWXYZ"), 01);
            
    $price rand(10,250);
            
    $production_date $this->random_date('1950-12-11','2013-04-11' );
            
    $data[]=array('code'=>$code,'class'=>$class'category'=>$category'price'=>$price'production_date'=>$production_date);
        }
    $this->model_products->insert($data); 
    PHP Code:
    public function insert($data)
            {
                
    $this->db->insert_batch('products_table'$data);     
            } 

    Question 1:
    I want to add another column and that is a 2 parahraph text of product_description. What would be the best way to generate random text so I can add it to the $data array above?

    Question 2:
    How big an array can php store? Is it the right way to send such big arrays to model for database interactions? If not whats the alternative?

    Thank you
  2. #2
  3. Jealous Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,303
    Rep Power
    9400
    1. Lorem ipsum. Read from /dev/urandom and base64_encode(). It's not like everything has to be unique or has to be totally random.

    2a. As large as you have memory for, but memory usage won't rile up your computer like that. How's your CPU usage?
    2b. Batch inserts. Like you have, but I wouldn't be comfortable doing 300k at a time. More like 10k.
  4. #3
  5. A Change of Season
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2004
    Location
    Next Door
    Posts
    2,689
    Rep Power
    171
    Originally Posted by requinix
    1. Lorem ipsum. Read from /dev/urandom and base64_encode(). It's not like everything has to be unique or has to be totally random.
    1 - Not sure what you mean. Lipsum doesnt seem to have a php section.
    Originally Posted by requinix
    2a. As large as you have memory for, but memory usage won't rile up your computer like that. How's your CPU usage?
    2 - CPU usage is usually on ~ 20%. When I run this query, while it's loading it goes up to 55% and as soon as the query ends, it drops down to 20%
    Originally Posted by requinix
    2b. Batch inserts. Like you have, but I wouldn't be comfortable doing 300k at a time. More like 10k.
    3 - So now that I am inserting data for test, it is going to take a long long time to insert another 10 million!

    4 - What about someperiodic updater so i dont have to manually run the script?

    5 - Any specifi reason why 10k?

    Thanks
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Loyal (3000 - 3499 posts)

    Join Date
    Dec 2004
    Posts
    3,031
    Rep Power
    377
    1. i think he means that generic latin text people use on their website at first. Just get that and use that as product desc.

    4. how about creating a cron job that does that every 30 minutes or so?

    5. I have from my previous experience noticed that php should comfortably handle around 10K records, when i try to do more, it stops (probably because of execution time)
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Feb 2013
    Posts
    14
    Rep Power
    0
    loren ipsum is random text use to fill the post. agree with what requinix has replied.
  10. #6
  11. Jealous Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,303
    Rep Power
    9400
    1. Right, just the static text. Prepend something unique to distinguish them if you wish.

    3. Yeah. You're talking about millions and millions of records here. Taking a long time is acceptable.

    4. Why something periodic? Wrap the while loop in another for loop, or something, then run the script. sleep() some if you need to.

    5. Presumably you're building a massive INSERT statement. 300,000 sets of values at a time is crazy. At least to me.
  12. #7
  13. A Change of Season
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2004
    Location
    Next Door
    Posts
    2,689
    Rep Power
    171
    Hello;

    Originally Posted by requinix
    1. Right, just the static text. Prepend something unique to distinguish them if you wish.
    What about something like:
    PHP Code:
    protected function generate_text($words)
        {
            
    $string "";
            
    $count 0;
            while(
    $count<$words)
                {
                    
    $count++;
                    
    $word_length rand(2,12);
                    
    $string .= substr(str_shuffle("ABCDEFGHIJKLMNOPQRSTUVWXYZ"), 0$word_length);
                    if(
    $count==30 || $count==76  || $count==165)
                        {
                            
    $string.=". ";    
                        }
                    
    $string .= " ";
                    if(
    $count==63 || $count==140)
                        {
                            
    $string.="\r\n";    
                        }
                }
            
    $string .= ". ";
            return 
    $string;
        } 
    6 - How do you recommend me store line breaks in the database? Am I doing it right above?
    Originally Posted by requinix
    4. Why something periodic? Wrap the while loop in another for loop, or something, then run the script. sleep() some if you need to.
    Do you mean something like:
    PHP Code:
    public function add_product()
            {

                
    $this->load->model('model_products');
                
    $count=0;
                while(
    $count<300000)
                    {
                        
    $count++;
                        
    $code substr(str_shuffle("ABCDEFGHIJKLMNOPQRSTUVWXYZ"), 04);
                        
    $class substr(str_shuffle("123456789"), 01);
                        
    $category substr(str_shuffle("ABCDEFGHIJKLMNOPQRSTUVWXYZ"), 01);
                        
    $price rand(10,250);
                        
    $production_date $this->random_date('1950-12-11','2013-04-11' );
                        
    $words rand(70250);
                        
    $description $this->generate_text($words);
                        
    $data[]=array('code'=>$code,'class'=>$class'category'=>$category'price'=>$price'production_date'=>$production_date'description'=>$description);
                    if(
    $count 10000)
                        {
                            
    sleep(10);    
                        }
                    }
                
    $this->model_products->insert($data);
                
    $products $this->model_products->list_products();
                
    $data['products']=$products;
                
    $data['number_of_products']=number_format($this->number_of_products);
                
    $this->parser->parse('test'$data);
                
    $this->output->enable_profiler(TRUE);
                
        } 
  14. #8
  15. Jealous Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,303
    Rep Power
    9400
    1. Why does the text have to be so random?

    6. As raw data. So yes, with \r and \n characters.
  16. #9
  17. A Change of Season
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2004
    Location
    Next Door
    Posts
    2,689
    Rep Power
    171
    Originally Posted by requinix
    1. Why does the text have to be so random?
    I am preparing to do a job. They have data with millions of very dynamic records I am practising performance optimization
    Originally Posted by requinix
    6. As raw data. So yes, with \r and \n characters.
    I'm gonna use \r\n.

    Also Did I use sleep() properly? Thanks
  18. #10
  19. Jealous Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,303
    Rep Power
    9400
    1. Well alright, but you'll take a performance hit.

    6. Yes but the condition is wrong: $count % 10000 will be true for every $count except those divisible by 10k. You want
    PHP Code:
    if(($count 10000) == 0
  20. #11
  21. A Change of Season
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2004
    Location
    Next Door
    Posts
    2,689
    Rep Power
    171
    Originally Posted by requinix
    1. Well alright, but you'll take a performance hit.
    What do you mean? Things slow down because I am doing something wrong?
  22. #12
  23. Jealous Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,303
    Rep Power
    9400
    Generating random text will be slower than using simple static text. Which shouldn't be surprising.
  24. #13
  25. A Change of Season
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2004
    Location
    Next Door
    Posts
    2,689
    Rep Power
    171
    Originally Posted by requinix
    Generating random text will be slower than using simple static text. Which shouldn't be surprising.
    Of course I understand but, having all records same value for description won't help me learn indexing that column would it?

    Did I use sleep properly?
  26. #14
  27. Jealous Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,303
    Rep Power
    9400
    Yes but the condition is wrong: $count % 10000 will be true for every $count except those divisible by 10k. You want
    PHP Code:
    if(($count 10000) == 0
  28. #15
  29. A Change of Season
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2004
    Location
    Next Door
    Posts
    2,689
    Rep Power
    171
    Originally Posted by requinix
    Yes but the condition is wrong: $count % 10000 will be true for every $count except those divisible by 10k. You want
    PHP Code:
    if(($count 10000) == 0
    thanks
Page 1 of 2 12 Last
  • Jump to page:

IMN logo majestic logo threadwatch logo seochat tools logo