#1
  1. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2003
    Posts
    6
    Rep Power
    0

    Creating large arrays


    I wanted to create an array to store the data I have from an image file. Being that they are medical images they are quite large (anywhere from 35mb to 600 mb)

    I tried doing the following but it just crashes

    for i in range(0, arraySize):
    myArray.append(0)

    I need to create an array just filled with zeros the proper size so that I can run the array through a filter that will put in the actual values.

    Doing the above crashes if I make the arraySize too large. (ie. over 1,000,000) I need arrays at least 18 million large.

    The strange thing is this worked when I did myArray.fromfile() and just streamed the entire binary into the array.

    Is this a limit of python and I need to figure out how to do it some other way or am I programming it wrong? Thank you for any help.

    (PS: I have 2 gigs of memory so it's not filling that up)
  2. #2
  3. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    Seems to work for me, I'm running the test now and its running through nicely, its very slow but then it is a very high number! Anyway, just wondering why if it worked with fromfile() why your not using it? It seems pretty pointless to rewrite code, plus fromfile is probably faster .

    Have fun,
    Mark.
  4. #3
  5. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69

    Max Iteration


    Oh ok, test just finished at 199999! I dont think it's an array size problem, if you can store 18000000 entries with the fromfile() function then it is most likly to do with the loop. Maybe you should try a while loop, i'm not sure whether there is a max number of iterations on this but it's worth a try, or maybe you could try several loops... 18 in fact but thats a really bad way of doing things!

    Hope this helps,
    Mark.
  6. #4
  7. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2003
    Posts
    6
    Rep Power
    0
    I can't use fromfile because I create the values on the fly. Thus I want to be able to insert them into the array at different indices. Does anyone know of any other way to initialize an array other than appending items?

    (fromfile does work much faster but that's not an option)

    ** This is how to do it for anyone that might have to do this in the future

    you want to do the following

    a = array('h', myArray = [0] * arraySize)


    Where of course the 0 is whatever you want to initialize the array to.

    here h is a signed int (2 bytes)
    Last edited by shochu; July 11th, 2003 at 09:23 AM.
  8. #5
  9. No Profile Picture
    Tattooed Python-Lovin' Freak-Boy
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2001
    Location
    orange county, CA
    Posts
    16
    Rep Power
    0
    Code:
    arraysize = 18000000
    mylist = range(arraysize)
    that will give you an array of 18,000,000 elements (just numbers) that you can iterate through, etc.

IMN logo majestic logo threadwatch logo seochat tools logo