reduce time for a long for-loop in python

a other stupid question from my side ;) I have some issues with the following snippet with len(x)=len(y)=7'700'000:

from numpy import *

for k in range(len(x)):
    if x[k] == xmax:
        xind = -1
    else:
        xind = int(floor((x[k]-xmin)/xdelta))
    if y[k] == ymax:
        yind = -1
    else:
        yind = int(floor((y[k]-ymin)/ydelta))

    arr = append(arr,grid[xind,yind])

All variables are floats or integers except arr and grid. arr is a 1D-array and grid is a 2D-array.

My problem is that it takes a long time to run through the loop (several minutes). Can anyone explain me, why this takes such a long time? Have anyone a suggestion? Even if I try to exchange range() through arange()then I save only some second.

Thanks.

1st EDIT Sorry. Forgot to tell that I'm importing numpy

2nd EDIT

I have some points in a 2D-grid. Each cell of the grid have a value stored. I have to find out which position the point have and apply the value to a new array. That's my problem and my idea.

p.s.: look at the picture if you want to understand it better. the values of the cell are represented with different c开发者_JAVA百科olors.

How about something like:

import numpy as np
xind = np.floor((x-xmin)/xdelta).astype(int)
yind = np.floor((y-ymin)/ydelta).astype(int)

xind[np.argmax(x)] = -1
yind[np.argmax(y)] = -1

arr = grid[xind,yind]

Note: if you're using numpy don't treat the arrays like python lists if you want to do things efficiently.

for x_item, y_item in zip(x, y):
    # do stuff.

There's also izip for if you don't want to generate a giant extra list.

I cannot see an obvious problem, beside the size of the data. Is your computer able to hold everything in memory? If not, you are probably "jumping around" in swapped memory, which will always be slow. If the complete data is in memory, give psyco a try. It might speed up your calculation a lot.

I suspect the problem might be in the way you're storing the results:

arr = append(arr,grid[xind,yind])

The docs for append say it returns:

A copy of arr with values appended to axis. Note that append does not occur in-place: a new array is allocated and filled.

This means you'll be deallocating and allocating a larger and larger array every iteration. I suggest allocating an array of the correct size up-front, then populating it with data in each iteration. e.g.:

arr = empty(len(x))

for k in range(len(x)):
    ...
    arr[k] = grid[xind,yind]

x's lenght is 7 millions? I think that's why! THe iterations ocurrs 7 millions times,

probably you shoud make another kind of loop. It's really necesary looping over 7 m times?