Forums: » Register « |  Free Tools |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support |
 User Name: Password: Remember me

New Free Tools on Dev Shed!
We're Excited to announce that Dev Shed now has 70 free tools on the site. To learn more, click here!

 Dev Shed Forums Sponsor:
#1
September 3rd, 2003, 09:23 PM
 theperfectsoup
Contributing User

Join Date: Jul 2003
Posts: 35
Time spent in forums: < 1 sec
Reputation Power: 11
Unused memory unrecoverable?

So I'm writing a program in Python, and comparing the memory usage of arrays to hash tables... And when testing arrays, I write the following two pieces of code.

The first:

Code:
```from random import randint, seed, sample
from bisect import insort_left
from time import time

array = []
seed(5)

# record when operation begins
t_beg = time()
# for 10 iterations
for i in range(10):
# maintaining order, insert 2000 random [x, y] pairs into the array
for j in xrange(2000):
insort_left(array, [randint(0, 0xFFFFFFF), randint(0, 0xFFFFFFF)])
# now delete 500 randomly chosen pairs
rem = sample(xrange(len(array)), 500)
rem.sort()
rem.reverse()
for j in rem:
del array[j]
# record when operation ends
t_end = time()

# print statistics
print "took " + str(t_end - t_beg) + " seconds"
print "size is " + str(len(array))```

The secod piece of code is just like the first, but I delete 12000 randomly chosen elements of the array at the end:

Code:
```from random import randint, seed, sample
from bisect import insort_left
from time import time

array = []
seed(5)

# record when operation begins
t_beg = time()
# for 10 iterations
for i in range(10):
# maintaining order, insert 2000 random [x, y] pairs into the array
for j in xrange(2000):
insort_left(array, [randint(0, 0xFFFFFFF), randint(0, 0xFFFFFFF)])
# now delete 500 randomly chosen pairs
rem = sample(xrange(len(array)), 500)
rem.sort()
rem.reverse()
for j in rem:
del array[j]
# record when operation ends
t_end = time()

# 10*(2000-500) = 15000 entries exist; delete 12000 (80%) at random
rem = sample(xrange(len(array)), 12000)
rem.sort()
rem.reverse()
for j in rem:
del array[j]

# print statistics
print "took " + str(t_end - t_beg) + " seconds"
print "size is " + str(len(array))```

Now after deleting 12000 elements of the array, you'd think that memory would be recovered, right? Well through the pseudo-scientific method of bringing up task manager in Windows XP and watching the memory usage of python.exe in the Processes tab, it seems the memory isn't reclaimed -- both code segments occupy approximately the same amount of memory after execution (1.5 MB). For the second piece of code, I even tried importing the garbage collector module and running gc.collect(), but to no avail. What gives?

Thanks,
- theperfectsoup

#2
September 4th, 2003, 07:42 AM
 netytan
Hello World :)

Join Date: Mar 2003
Location: Hull, UK
Posts: 2,537
Time spent in forums: 1 Week 2 Days 18 h 17 m 47 sec
Reputation Power: 68
Oh ok, i'll have to test this myself when I get home.. just a few questions - always seem to be asking Q's - what version of Python are you using? Does isenabled() return true? also, when the program has finished the memory is returned anyway right?

Have fun,
Mark.

#3
September 4th, 2003, 03:32 PM
 theperfectsoup
Contributing User

Join Date: Jul 2003
Posts: 35
Time spent in forums: < 1 sec
Reputation Power: 11
I'm using Python version 2.3. The call gc.isenabled() returns 1 (True). And when I quit Python, the memory is all recovered.

After experimenting a little more, here are some other interesting facts I've found...

1. After you delete the 12,000 elements of the array, if you then allocate 12,000 elements (bringing the total back up to 15,000), then memory usage does NOT increase. So when you delete those 12,000 elements, the memory is put aside for future allocated elements to use, but it is not released by the Python interpreter.

2. If you use the Array module, which basically creates underlying arrays in C, allocate 15,000 elements, and then delete 12,000 you can clearly see the memory being freed via task manager. This is the behavior I want, but the problem is I need to store 80-bit numbers and Array only typically allows 32-bit (64-bit for floating point). I've thought about splitting up the 80-bit values across three Array variables -- one for the highest 32-bits, the next for the middle 32-bits, and the last for the lowest 16-bits, but because of the way I'm storing data (including multidimensional arrays, which Array does not allow), that falls through. Oh well.

- theperfectsoup

#4
September 4th, 2003, 07:05 PM
 netytan
Hello World :)

Join Date: Mar 2003
Location: Hull, UK
Posts: 2,537
Time spent in forums: 1 Week 2 Days 18 h 17 m 47 sec
Reputation Power: 68
I surpose in a way it makes a strange kinda sence for the interpriter to try hold onto the memory it's used untill it's done - not that thats what it does - that way it doesn't have to play tag with other programs, once it has it it doesn't have to let it go and get it back again , Just one point of view

Strange problem your having there, what exactly are you trying to do that requires putting numbers that long in a list? sounds interesting though..by 32 bit you mean a number with 32 trailing 0's?

Have fun,
Mark.

 Viewing: Dev Shed Forums > Programming Languages > Python Programming > Unused memory unrecoverable?

Developer Shed Advertisers and Affiliates

 Thread Tools Search this Thread Search this Thread: Advanced Search Display Modes Rate This Thread Linear Mode Rate This Thread: 5 : Excellent 4 : Good 3 : Average 2 : Bad 1 : Terrible

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are On [IMG] code is On HTML code is Off
 View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox Forum Jump Please select one User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home -------------------- Programming Languages    PHP Development        PHP FAQs and Stickies    Perl Programming        Perl FAQs and Stickies    C Programming        C Programming FAQs and Stickies    Java Help        Java FAQs    Python Programming        Python Programming FAQs    Ruby Programming        Ruby Programming FAQs    Game Development        Game Development FAQs Programming Languages - More    ASP Programming        ASP Programming FAQs    .Net Development        .Net Development FAQs    Visual Basic Programming        Visual Basic Programming FAQs    Software Design        Software Design FAQs    ColdFusion Development        ColdFusion Development FAQs    Delphi Programming        Delphi Programming FAQs    Regex Programming        Regex Programming FAQs    XML Programming        XML Programming FAQs    Other Programming Languages        Other Programming Languages FAQs Web Design    HTML Programming        HTML Programming FAQs    JavaScript Development        JavaScript Development FAQs    CSS Help        CSS Help FAQs    Flash Help        Flash Help FAQs    Photoshop Help        Photoshop Help FAQs    Web Design Help        Web Design Help FAQs    Website Critiques        Website Critiques FAQs    Search Engine Optimization        Search Engine Optimization FAQs Mobile Programming    Mobile Programming        Mobile Programming FAQs    iPhone SDK Development        iPhone SDK Development FAQs    Android Development        Android Development FAQs    BlackBerry Development        BlackBerry Development FAQs Web Site Management    Business Help        Business Help FAQs    Development Software        Development Software FAQs    Scripts        Scripts FAQs Databases    Database Management        Database Management FAQs    DB2 Development        DB2 Development FAQs    MySQL Help        MySQL Help FAQs    PostgreSQL Help        PostgreSQL Help FAQs    Firebird SQL Development        Firebird SQL Development FAQs    MS SQL Development        MS SQL Development FAQs    Oracle Development        Oracle Development FAQs    LDAP Programming        LDAP Programming FAQs System Administration    Mail Server Help        Mail Server Help FAQs    Apache Development        Apache Development FAQs    Security and Cryptography        Security and Cryptography FAQs    Antivirus Protection        Antivirus Protection FAQs    DNS        DNS FAQs    IIS        IIS FAQs    Networking Help        Networking Help FAQs    FTP Help        FTP Help FAQs Operating Systems    BSD Help        BSD Help FAQs    Linux Help        Linux Help FAQs    UNIX Help        UNIX Help FAQs    Windows Help        Windows Help FAQs    Mac Help        Mac Help FAQs Web Hosting    Web Hosting        Web Hosting FAQs    Free Web Hosting        Free Web Hosting FAQs    Web Hosting Requests        Web Hosting Requests FAQs    Web Hosting Offers        Web Hosting Offers FAQs Computer Hardware    Computer Hardware    CPUs        CPUs FAQs    Cooling        Cooling FAQs    Embedded Programming        Embedded Programming FAQs    Motherboards        Motherboards FAQs    Multimedia Hardware        Multimedia Hardware FAQs Other    Dev Shed Lounge        Dev Shed Lounge FAQs    Development Articles        Development Articles FAQs    Beginner Programming        Beginner Programming FAQs    Hire A Programmer        Hire A Programmer FAQs    Project Help Wanted        Project Help Wanted FAQs Latest News Updated Hourly    Technology News    Business News    Science News Forum Information    Forum Rules/Guidelines        Forum Rules/Guidelines FAQs    Forum Announcements        Forum Announcements FAQs    Dev Shed Gaming Center        Go to the Dev Shed Battle Arena        Go to the Dev Shed Arcade Games        Go to the Legend of the Green Dragon    Suggestions & Feedback        Suggestions & Feedback FAQs

 Forums: » Register « |  Free Tools |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support |