#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    76
    Rep Power
    2

    Smoother way of doing this with generators?


    So, I was working with they array module, something that I'm not exactly big on, and I came upon this silly little trick:

    Code:
    from array import array
    test=array('H')
    list(test.append(ord(v)) for v in 'string')
    print(test)
    You'll notice test now has all the byte values for each character in "string".
    I thought that was actually a really neat, concise way of doing it.
    However, since I used the list to execute all the generator expressions, it does create a blank list in which it puts all the "None" values returned from the function.
    Although it's really no biggie as Python garbage-collects it anyway, can someone tell me if there's a quicker way to just run through all the values of a generator without resorting to this crazy list-creation function?
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    114
    Rep Power
    3
    The itertools docs include a recipe for a consume() function to do this, but it's essentially doing the same thing you are, under the hood. It uses collections.deque(it, maxlen=0) instead of using list(), which I suppose wastes less memory.

    You can always just do, "for item in it: pass", of course.
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2013
    Location
    Usually Japan when not on contract
    Posts
    240
    Rep Power
    11
    Originally Posted by Mr909
    ...can someone tell me if there's a quicker way to just run through all the values of a generator...
    This is an advance form of swimming against the current. Generators are a deliberate design that trades execution time in exchange for lazy lists. The point of a generator is to have an iterable for which you don't need to run through all the values (particularly since it very well may be an infinite list...).

    The better question isn't "how to run through all values of a generator" but "why do I find myself needing to run through all values of a generator?" Answering the second question leads to design decisions which obviate the first -- unless you're working around a blackbox API.
  6. #4
  7. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    76
    Rep Power
    2
    Originally Posted by zxq9
    This is an advance form of swimming against the current. Generators are a deliberate design that trades execution time in exchange for lazy lists. The point of a generator is to have an iterable for which you don't need to run through all the values (particularly since it very well may be an infinite list...).

    The better question isn't "how to run through all values of a generator" but "why do I find myself needing to run through all values of a generator?" Answering the second question leads to design decisions which obviate the first -- unless you're working around a blackbox API.
    Okay, then teach me. Teach me another way to do what I did in the original trick in one line or less.

    I mean, I totally get what you're saying, but to be honest I can't even tell you what proper usage for generators look like because I usually just use it to quickly, efficiently format lists. I only really USE finite generators, and they work for me because they give me the results of an iterable without blocking out an entire 'for-in' loop just to do the same thing, more verbosely.

    Originally Posted by Nyktos
    The itertools docs include a recipe for a consume() function to do this, but it's essentially doing the same thing you are, under the hood. It uses collections.deque(it, maxlen=0) instead of using list(), which I suppose wastes less memory.

    You can always just do, "for item in it: pass", of course.
    Thank you.
  8. #5
  9. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2013
    Location
    Usually Japan when not on contract
    Posts
    240
    Rep Power
    11
    Originally Posted by Mr909
    Teach me another way to do what I did in the original trick in one line or less.
    "One line" would be:
    python Code:
    # Python 3
    test.frombytes('string')
     
    # Python2
    test.fromstring('string')
    # or
    test.fromunicode(u'string')

    These do in the background what you are trying to do above, but without going to the extra trouble of assembling a list result that will be discarded.

    "less" would be:
    python Code:
    # Line below deleted to avoid coercing data objects
    # to non-portable machine-level representations
    # without explanation

    I mean, I totally get what you're saying, but to be honest I can't even tell you what proper usage for generators look like because I usually just use it to quickly, efficiently format lists.
    Without knowing anything about your use case I can't give you practical programming advice (why do you need to do this? why are you using array? etc.). You've posed an X Y problem.

    But the more general question of how/why to use generators VS lists is a good question.

    The short answer is that lists require all memory resources involved to be allocated through the entire life of the list plus e(n) processing time to initialize (an up-front cost). Generators, on the other hand, require a tiny, constant amount of memory and a trivial amount of time to initialize regardless the size of the set they represent, but are more expensive to access than lists.

    The longer answer goes something like...
    A list must contain all of its members for the duration of its life. A generator only need be aware of its last value and how that relates to the next value it should yield -- it generates answers as it goes, it doesn't recall them from memory. Generation is more expensive than recall, but it permits you to do things like ask the computer "tell me the next number after 314" instead of "remember every number from X to Y and tell me the next one". The generator approach lets you create lists of arbitrary size without worrying about how much memory that will suck up.

    Consider the python2 example:
    python Code:
    # A list with 1 billion members... ouch!
    for x in range(0, 1000000000):
      foo()
     
    # A generator containing only its instance variables
    for x in xrange(0, 1000000000):
      foo()

    Pretty vast difference. In Python3 xrange() has been renamed range() and the old list version has been dropped entirely. If you want to iterate over a list you can still do that, just generate it on your own.

    The generator approach also lets you define a dynamic range of values -- a range that may change in the middle of some other processing. Some folks consider this black magic -- and writing code like is certainly a way to outsmart yourself if you're not careful -- but it is useful in some cases. If the domains you are dealing with are large then generators are often the only practical solution to such a problem, because there may not be enough processing time or memory in the world to generate a list for every variation of a dynamic domain as it evolves.

    But in almost every case what you need is a list, not a generator. In the example above where you used 'string' as your target you're using a special form of tuple (a string), not a generator anyway, so this whole point is moot -- hence my reference to the array methods fromstring() and frombyte() (there is a from*() method for nearly any valid C array type on common architectures).
  10. #6
  11. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    76
    Rep Power
    2
    Well, then the "consume" recipe should be the ideal solution for method calls through a generator, non? I mean, seriously, there is no point (like in my initial example) generating a list of "None" just to perform all the method calls.
  12. #7
  13. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    May 2013
    Location
    Usually Japan when not on contract
    Posts
    240
    Rep Power
    11
    The point, you missed it.

    But that's ok.

    Only after a very long journey may the path you want to travel meet the one you should be traveling.
  14. #8
  15. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2012
    Posts
    76
    Rep Power
    2
    Well, I see you taught me some new use cases and some new things I might be able to do. The new fromstring method. But as to the original question, which was "how to batch-process method calls", this remains a good way. And your solution with array.fromstring remains a good solution for that specific problem.

    I don't see where our paths branch here. You solved the specific, the other poster created a new general solution I can use for other things.

IMN logo majestic logo threadwatch logo seochat tools logo