Python Programming
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me

The Shed is going Social! Join us on FaceBook and Twitter and chime in on the conversation.

Go Back   Dev Shed ForumsProgramming LanguagesPython Programming

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old September 3rd, 2003, 08:23 PM
theperfectsoup theperfectsoup is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jul 2003
Posts: 35 theperfectsoup User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 10
Exclamation Unused memory unrecoverable?

So I'm writing a program in Python, and comparing the memory usage of arrays to hash tables... And when testing arrays, I write the following two pieces of code.

The first:

Code:
from random import randint, seed, sample
from bisect import insort_left
from time import time

array = []
seed(5)

# record when operation begins
t_beg = time()
# for 10 iterations
for i in range(10):
	# maintaining order, insert 2000 random [x, y] pairs into the array
	for j in xrange(2000):
		insort_left(array, [randint(0, 0xFFFFFFF), randint(0, 0xFFFFFFF)])
	# now delete 500 randomly chosen pairs
	rem = sample(xrange(len(array)), 500)
	rem.sort()
	rem.reverse()
	for j in rem:
		del array[j]
# record when operation ends
t_end = time()

# print statistics
print "took " + str(t_end - t_beg) + " seconds"
print "size is " + str(len(array))


The secod piece of code is just like the first, but I delete 12000 randomly chosen elements of the array at the end:

Code:
from random import randint, seed, sample
from bisect import insort_left
from time import time

array = []
seed(5)

# record when operation begins
t_beg = time()
# for 10 iterations
for i in range(10):
	# maintaining order, insert 2000 random [x, y] pairs into the array
	for j in xrange(2000):
		insort_left(array, [randint(0, 0xFFFFFFF), randint(0, 0xFFFFFFF)])
	# now delete 500 randomly chosen pairs
	rem = sample(xrange(len(array)), 500)
	rem.sort()
	rem.reverse()
	for j in rem:
		del array[j]
# record when operation ends
t_end = time()

# 10*(2000-500) = 15000 entries exist; delete 12000 (80%) at random
rem = sample(xrange(len(array)), 12000)
rem.sort()
rem.reverse()
for j in rem:
	del array[j]

# print statistics
print "took " + str(t_end - t_beg) + " seconds"
print "size is " + str(len(array))


Now after deleting 12000 elements of the array, you'd think that memory would be recovered, right? Well through the pseudo-scientific method of bringing up task manager in Windows XP and watching the memory usage of python.exe in the Processes tab, it seems the memory isn't reclaimed -- both code segments occupy approximately the same amount of memory after execution (1.5 MB). For the second piece of code, I even tried importing the garbage collector module and running gc.collect(), but to no avail. What gives?

Thanks,
- theperfectsoup

Reply With Quote
  #2  
Old September 4th, 2003, 06:42 AM
netytan's Avatar
netytan netytan is offline
Hello World :)
Dev Shed Frequenter (2500 - 2999 posts)
 
Join Date: Mar 2003
Location: Hull, UK
Posts: 2,537 netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 1 Week 2 Days 18 h 17 m 47 sec
Reputation Power: 68
Send a message via ICQ to netytan Send a message via AIM to netytan Send a message via MSN to netytan Send a message via Yahoo to netytan
Oh ok, i'll have to test this myself when I get home.. just a few questions - always seem to be asking Q's - what version of Python are you using? Does isenabled() return true? also, when the program has finished the memory is returned anyway right?

Have fun,
Mark.

Reply With Quote
  #3  
Old September 4th, 2003, 02:32 PM
theperfectsoup theperfectsoup is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jul 2003
Posts: 35 theperfectsoup User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 10
I'm using Python version 2.3. The call gc.isenabled() returns 1 (True). And when I quit Python, the memory is all recovered.

After experimenting a little more, here are some other interesting facts I've found...

1. After you delete the 12,000 elements of the array, if you then allocate 12,000 elements (bringing the total back up to 15,000), then memory usage does NOT increase. So when you delete those 12,000 elements, the memory is put aside for future allocated elements to use, but it is not released by the Python interpreter.

2. If you use the Array module, which basically creates underlying arrays in C, allocate 15,000 elements, and then delete 12,000 you can clearly see the memory being freed via task manager. This is the behavior I want, but the problem is I need to store 80-bit numbers and Array only typically allows 32-bit (64-bit for floating point). I've thought about splitting up the 80-bit values across three Array variables -- one for the highest 32-bits, the next for the middle 32-bits, and the last for the lowest 16-bits, but because of the way I'm storing data (including multidimensional arrays, which Array does not allow), that falls through. Oh well.

- theperfectsoup

Reply With Quote
  #4  
Old September 4th, 2003, 06:05 PM
netytan's Avatar
netytan netytan is offline
Hello World :)
Dev Shed Frequenter (2500 - 2999 posts)
 
Join Date: Mar 2003
Location: Hull, UK
Posts: 2,537 netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level)netytan User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 1 Week 2 Days 18 h 17 m 47 sec
Reputation Power: 68
Send a message via ICQ to netytan Send a message via AIM to netytan Send a message via MSN to netytan Send a message via Yahoo to netytan
I surpose in a way it makes a strange kinda sence for the interpriter to try hold onto the memory it's used untill it's done - not that thats what it does - that way it doesn't have to play tag with other programs, once it has it it doesn't have to let it go and get it back again , Just one point of view

Strange problem your having there, what exactly are you trying to do that requires putting numbers that long in a list? sounds interesting though..by 32 bit you mean a number with 32 trailing 0's?

Have fun,
Mark.

Reply With Quote
Reply

Viewing: Dev Shed ForumsProgramming LanguagesPython Programming > Unused memory unrecoverable?

Developer Shed Advertisers and Affiliates



Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 


Powered by: vBulletin Version 3.0.5
Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.

© 2003-2013 by Developer Shed. All rights reserved. DS Cluster - Follow our Sitemap