#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2010
    Posts
    22
    Rep Power
    0

    Script killed when loading big hash


    Hi all,
    I am trying to load hash in size of ~10gb.
    i am using this command:
    %hash = %{ retrieve("filename") };
    the procces killed..
    How can i solve this?
    i was trying toset unlimit when execute by external bash script:

    #!/bin/bash
    ulimit -s unlimited
    perl tmp.pl

    Any ideas?

    Thanks
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jun 2012
    Posts
    830
    Rep Power
    496
    Yoiu did not say what the retrieve function is doing.

    But, in general, if the file is too big to fit into memory, then no luck.

    See if you can do your task differently, without loading the full data into memory.
    Last edited by Laurent_R; June 20th, 2013 at 01:43 AM. Reason: fixed a typo.
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2010
    Posts
    22
    Rep Power
    0
    Originally Posted by Laurent_R
    Yoiu did not say what the retrieve function is doing.

    But, in general, if the file is too big to fit into memory, then no luck.

    See if you can do your task differently, without loading the full data into memory.
    What do you mean what retrieve doing?
    retrieve the sorable hash..
    i succeeded to retrieve by reference..
    $hashref = retrieve('file');

    But now, how can i search in the hash? this not working..
    if (exists $hashref{$key}) { print "found\n"; }
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Jun 2012
    Posts
    830
    Rep Power
    496
    I did not know that the retrieve you used was the retrieve function of Storable, that is why I asked.

    Since you ar egetting a hashref, try this:

    Perl Code:
    if (exists $hashref->{$key}) { print "found\n"; }
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2010
    Posts
    22
    Rep Power
    0
    OK!
    Thanks!
    compiling

IMN logo majestic logo threadwatch logo seochat tools logo