Loading db.ListProperty() with AppEngine bulkloader

I'm trying to populate a db.ListProperty() model field using bulkloader.

I'm using an import transform function as follows:

def parse_array(fn):
    def wrapper(value):
        return [fn(seg) for seg in re.split("\\,", value) if not seg=='']
   开发者_运维知识库return wrapper

Configured as follows:

import_transform: lib.bulkloader_helpers.parse_array(int)

Some of the arrays are empty, and these are causing a problem. When I upload:

BadValueError: May not use the empty list as a property value; property xxx is [].

Uh oh. Okay, let's change the import transform function:

def parse_array(fn):
    def wrapper(value):
        args[fn(seg) for seg in re.split("\\,", value) if not seg=='']
        if args==[]:
            return None
        else:
            return args 
    return wrapper

Now the empty lists load just fine. However when the app tries to load the model:

BadValueError: Property xxx is required

Nor can I set db.ListProperty(required=False):

google.appengine.ext.db.ConfigurationError: required must be True.

Anyone suggest a way out ?

Thanks,

Justin

This is a known bug (Issue 3646) with the App Engine development environment.

A workaround is to change (as of version 1.9.6) line 1530 of google.appengine.api.datastore_types.py which reads:

  if not values:

To:

  if not values_type is list and not values:

Afterwards you can properly insert

[]

for the empty arrays and it should work.

Google haven't fixed this yet (Feb 2013, API version 1.7.4), so here's a workaround that doesn't involve patching AppEngine code:

First use your updated import_transform function, that returns None instead of an empty list. Then add a post_import_function for the entity to your bulkloader.yaml (docs) that checks list properties for None and deletes them from the entity, thereby setting them back to the empty list:

def post_import(input_dict, instance, bulkload_state_copy):
    if instance["list_prop_name"] is None:
        del instance["list_prop_name"]
    return instance

Just omit the required argument altogether. If it's supplied, it must be set to True, but not supplying it makes it False.