#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Location
    Europe
    Posts
    21
    Rep Power
    0

    Huge calculation


    I couldn't find a proper title, anyway my problem is the following: i have made a program that creates a random sequence of 1 and 0 and then pass it to string and then to a file. But if i give a sequence bigger than 1.000.000 the program just sits there without doing anything. Is there anyway to overcome that? i mean is it a buffer over flow, too big string or what...? and secondly is there any way to display the process of the program? like: loading 20% for example, i can do it, but i cant make it to stay in the same position...and just change the 20% to 30% or 50%...
  2. #2
  3. No Profile Picture
    Super Moderator
    Devshed Specialist (4000 - 4499 posts)

    Join Date
    Jul 2003
    Posts
    4,009
    Rep Power
    2791
    Which version of Python are you using? Can you post the code please.
    [PHP] | [Perl] | [Python] | [Java] != [JavaScript] | [XML] | [C] | [C++] | [LUA] | [MySQL] | [FirebirdSQL] | [PostgreSQL] | [HTML] | [XHTML] | [CSS]

    W3Fools - A W3Schools Intervention.
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Location
    Europe
    Posts
    21
    Rep Power
    0
    Originally Posted by Winters
    Which version of Python are you using? Can you post the code please.
    i have managed to make it go up to 100.000.000 by writing directly to file but still i cant go over that... i know it is silly trying to go over 100 million but i want to try it out.. anyway, here is the code
    Code:
    #random digit generator 1.2 12-11-2012 12:43
    #version 2.7.3
    
    import random, os, sys
    
    def _get_error_type_1():
    	print "\nplease use letters! try again!"
    	
    def _get_error_type_2():
    	print "\nplease use integers! try again!"
    	
    def _get_exit_msg():
    	print "\nthe program will now terminate!"
    
    while True:
    	save_to_file = 0
    	choise = raw_input("\ndo ou want to save te results to a file?(y or n): ")
    	
    	try:
    		test = str(choise)
    	except ValueError:
    		_get_error_type_1()
    		continue
    		
    	if ((choise == 'y') or (choise == 'Y')):
    		save_to_file = 1
    	elif ((choise == 'n') or (choise == 'N')):
    		_get_exit_msg()
    		sys.exit()
    	else:
    		print "\nworng input please choose either y or n"
    		continue
    		
    
    	break
    	
    i = 0
    	
    while True:
    	i = i + 1
    	try:
    		text_length = int(raw_input("\nhow big do you want your text to be?: "))
    	except ValueError:
    		_get_error_type_2()
    		continue
    	
    	if save_to_file == 1:
    		fp = open("generated_text.txt", "a+", 255)
    
            if save_to_file == 1:
    		if i == 1:
    			fp.write("\n\n--New Session--\n\n")
    		else:
    			fp.write("\n\n--Sub-session--\n\n")
    		
    	while text_length > 0:
    		digit_num = random.randint(0, 1)
    		digit_str = str(digit_num)
    		fp.write(digit_str)
    		if text_length % 100 == 1:
    			fp.write("\n")
    		text_length -= 1
    	
    	while True:
    	
    		choise = raw_input("\ndo you want to continue?(y or n): ")
    	
    		try:
    			test = str(choise)
    		except ValueError:
    			_get_error_type_1()
    			continue
    			
    		if ((choise == 'y') or (choise == 'Y')):
    			break
    		elif ((choise == 'n') or (choise == 'N')):
    			_get_exit_msg()
                            fp.close()
    			sys.exit()
  6. #4
  7. No Profile Picture
    Super Moderator
    Devshed Specialist (4000 - 4499 posts)

    Join Date
    Jul 2003
    Posts
    4,009
    Rep Power
    2791
    That would be somewhere around 750MB in size. How much RAM do you have available? Is your operating system 32- or 64-bit?

    "Choise" is spelt "Choice."
    [PHP] | [Perl] | [Python] | [Java] != [JavaScript] | [XML] | [C] | [C++] | [LUA] | [MySQL] | [FirebirdSQL] | [PostgreSQL] | [HTML] | [XHTML] | [CSS]

    W3Fools - A W3Schools Intervention.
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Location
    Europe
    Posts
    21
    Rep Power
    0
    Originally Posted by Winters
    That would be somewhere around 750MB in size. How much RAM do you have available? Is your operating system 32- or 64-bit?

    "Choise" is spelt "Choice."
    64 bit 8 giga ram, anyway i can deal with the limited amount of text, what about the loading indicator? is it possible to print something, than delete is and print it again? so the 30% becomes 31% in the same position? and one more thing multiprocessing implementation would make it run faster?


    P.S. you are right for the choice i feel a bit embarrassed but english are not my mother language and when i write code grammar is the least of my worries so...
  10. #6
  11. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,963
    Rep Power
    481
    I'd think the code performance would be much enhanced if you generated random numbers from 0 to 255 inclusive random.randrange(256) using that to index into a precomputed list of strings having the text you'd like in the output. For instance I'd compute 2 bits at a time with:

    '00 01 10 11'.split()[random.randrange(4)]


    You open your output file for append. Now, maybe you delete that file every now and then, or maybe you've let it grow to the maximum size set by your operating system.
    [code]Code tags[/code] are essential for python code and Makefiles!
  12. #7
  13. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Location
    Europe
    Posts
    21
    Rep Power
    0
    Originally Posted by b49P23TIvg
    I'd think the code performance would be much enhanced if you generated random numbers from 0 to 255 inclusive random.randrange(256) using that to index into a precomputed list of strings having the text you'd like in the output. For instance I'd compute 2 bits at a time with:

    '00 01 10 11'.split()[random.randrange(4)]


    You open your output file for append. Now, maybe you delete that file every now and then, or maybe you've let it grow to the maximum size set by your operating system.
    wow thx, i would have never think of that! but, should i assume that a code like this: '000 001 010 etc 111'.split()[random.randrange(8)] would be even faster and still as random as the above? or not?
  14. #8
  15. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,963
    Rep Power
    481
    Reread my post, you'll see that I recommended a byte at a time---that's the 256. On my system I'd use 16 bits at a time. The 2 bit example was to demonstrate the concept.
    [code]Code tags[/code] are essential for python code and Makefiles!
  16. #9
  17. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Location
    Europe
    Posts
    21
    Rep Power
    0
    Originally Posted by b49P23TIvg
    Reread my post, you'll see that I recommended a byte at a time---that's the 256. On my system I'd use 16 bits at a time. The 2 bit example was to demonstrate the concept.
    now i am confused, you mean to translate the decimal random into its binary form and then write it?

    EDIT: suppose i get 188 from the rand function what then?
  18. #10
  19. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,963
    Rep Power
    481

    If I were to write this for myself I'd write this


    Code:
    import sys
    import random
    rr = random.randrange
    wr = sys.stdout.write
    
    bitRep8 = []
    append = bitRep8.append
    bitRep2='00 01 10 11'.split()
    for s0 in bitRep2:
        for s1 in bitRep2:
            high_nibble = s0+s1
            for s2 in bitRep2:
                prefix = high_nibble+s2
                for s3 in bitRep2:
                    append(prefix+s3)
    
    assert 256 == len(bitRep8)
    assert 8*128 == ''.join(bitRep8).count('0')
    
    for i in range(4):                        #  4 lines
        for i in range(72//8):                # 72 characters long
            wr(bitRep8[rr(256)])
        wr('\n')
    [code]Code tags[/code] are essential for python code and Makefiles!
  20. #11
  21. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Location
    Europe
    Posts
    21
    Rep Power
    0
    Originally Posted by b49P23TIvg
    Code:
    import sys
    import random
    rr = random.randrange
    wr = sys.stdout.write
    
    bitRep8 = []
    append = bitRep8.append
    bitRep2='00 01 10 11'.split()
    for s0 in bitRep2:
        for s1 in bitRep2:
            high_nibble = s0+s1
            for s2 in bitRep2:
                prefix = high_nibble+s2
                for s3 in bitRep2:
                    append(prefix+s3)
    
    assert 256 == len(bitRep8)
    assert 8*128 == ''.join(bitRep8).count('0')
    
    for i in range(4):                        #  4 lines
        for i in range(72//8):                # 72 characters long
            wr(bitRep8[rr(256)])
        wr('\n')
    thx very much, i will study the code very carefully i really appreciate it, its good practise to see what a more experienced person would do

IMN logo majestic logo threadwatch logo seochat tools logo