python filter calculated values from large lists to create multiple smaller lists

I want to find an optimal method to process very large ordered lists of integers eg.

biglist = [45, 34, 2, 78, 7, 9, 10, 33, 78, 51, 99, 24, 88, ... N] where N > 1m

to create multiple small lists of fixed length S (~=200) by reading each biglist element, apply different operations on the element, and if meets a conditional criteria add the element or a value to each small list until S is reached eg.

x_smallist = []
y_smallist = []
z_smallist = []
count = 0
for i 开发者_Python百科in biglist:
    b = i / 5
    a = b * 2
    c = a^3 + b
    if b > 7 and b < 69:
        x_smallist.append(i)
        y_smallist.append(a)
        z_smallist.append(b)
        count += 1
    if count > S:
        break

The example and function is for illustration only. As biglist is large and each element is read and operated on until S is reached, and the process is repeated thousands of times, I want to avoid the for-loop. How can this be achieved with a list comprehension (or map or filter)?

I think the following should do what you want, this works by having a generator yield a tuple with a new element for each list, and then using zip to create the three lists you want. An iterator for biglist is created at the beginning so that each time through the loop you will pick up where you left off, and islice is used so that the generator stops at S elements.

itr = iter(biglist)
while True:
    lists = itertools.islice(((i, i/5*2, i/5) for i in itr if 7 < i/5 < 69), S)
    x_smallist, y_smallist, z_smallist = zip(*lists)
    if len(x_smallist) == 0:
        break       # reached the end of biglist
    # do stuff with your small lists

Does biglist really have to be a list? If you can create them with a generator, you can save the memory, and perhaps save some time.

S = 200
import itertools
biglist = itertools.islice(itertools.ifilter(lambda x: 7 < x/5 < 69, biglist),S)

or if you want multiple chunks just apply ifilter and then loop on the result.