l[:] performance problem

Recently I was debugging a code like following

def getList():
    #query db and return a list
    total_list = Model.objects.all()
    result = list()
    for item in total_list:
        if item.attr1:
            result.append(item)
    return result


# in main code    
org_list = getList()
list = orgList[:]#this line cause cpu problems.

if len(org_list)>0 and org_list[0].is_special:
    my_item = org_list[0]

for i in list:
    print_item(i)

doSomethingelse(list[0])

In order to simplify the code I change most of it but the main part is here.

In getList method we query db and get 20~30 rows. Then we create a python list from it and return it. in main method we get org_list variable from getList method and slice it with orgList[:]

and loop over this list and call spesific elements on it 开发者_开发百科like list[0]

The problem here is that this code runs on a really busy server and unfortunaletty it uses most of the cpu and eventually locks our servers. Problem here is the line that we slice list varibale with list[:] if we dont do that and just use org_list variable instead our servers does not have a problem. Does anybosy have any idea why that might happen. is slicing uses alot of cpu or when we use a sliced list. does it uses alot of cpu?

The code that you are showing would run in the 0.1 microseconds that it would take to raise an exception:

org_list = getList()
list = orgList[:]#this line cause cpu problems.

orgList should be org_list. Show the minimal code that actually reproduces the problem.

Also, that kills the list built-in function. Don't do that.

Update Another thought: A common response to "My Django app runs slowly!" is "turn off the debug flag" ... evidently it doesn't free up memory in debug mode.

Update2 about """I found out that when you slice a list. it actually works as a view to that list and just call original list methods to get actual item.""" Where did you get that idea? That can't be the case with a plain old list! Have you redefined list somewhere?

In your getList function:

(1) put in print type(list)

(2) replace result = list() with result = [] and see whether the problem goes away.

list = org_list[:] makes a copy of org_list. Usually you see something like this when you need to modify the list but you want to keep the original around as well.

The code you are showing me doesn't seem like it actually modifies org_list, and if doSomethingelse(list[0]) doesn't modify list, I don't see why it's actually being copied in the first place. Even if it does modify list, as long as org_list isn't needed after those few lines of code, you could probably get away with just using org_list and not doing the slice copy.

I would like to know:

• what is the type of total_list ? Please do print type(total_list)

• what are the types of the elements of total_list ?

• have you an idea of the sizes of these elements ?

• what is the number of elements in the list returned by getList() ? 20-30 ?

Remarks:

-giving the keyword "list" as a name for a list isn't a good idea (in list = orgList[:] )

-your getList function can be written :

def getList():
    #query db and return a list
    total_list = Model.objects.all()
    return [ item for item in total_list if item.attr1 ]

unfortunately actual code is property of the company I worked for so I cannot give it exactly like it is but here is some part of it (variable names are changed,but argument of mainTree method was really named list.)

    mainObj = get_object_or_404(MainObj, code_name=my_code_name)
    if (not mainObj.is_now()) and (not request.user.is_staff):
        return HttpResponseRedirect("/")

    main_tree = mainObj.main_tree()
    objects = main_tree[:] # !!!
    if objects.__len__() == 1 and not objects[0].has_special_page:
        my_code_name = objects[0].code_name

...
...
...

mainObj.main_tree() is the following

    def main_tree(self):
        def append_obj_recursive(list, obj):
            list.append(obj)
            for c in self.obj_set.filter(parent_obj=obj):
                append_obj_recursive(list, c)
            return list

        root_obj = self.get_or_create_root_obj();
        top_level_objs = self.obj_set.filter(parent_obj=root_obj).order_by("weight")
        objs = []

        for tlc in top_level_objs:
            append_obj_recursive(objs, tlc)
        return objs

This is a really weird problem. slicing should not cause such a bizare problem. If I can find an answer I will post it here too.