Using java google app engine bulkloader.yaml to upload data with export_transform on __key__ changes ID/NAME coloumn from id (long) to name (string)

I think my question is similar to: BulkLoader -export_transform OR https://stackoverflow.com/questions/3220911/gae-datastore-export-transform

Basically, I'm using the bulkloader to backup and restore entities that I've created. To test I'm using the "Game" kind and outputting it to a csv file called game.csv.

This is the process I go through:

1. Download the game kind to game.csv using:

   appcfg.py download_data --config_file=bulkloader.yaml --kind=Game --filename=game.csv --application=MyAppId --url=http://MyAppId.appspot.com/remote_api --rps_limit=500 --bandwidth_limit=2500000 --batch_size=100

2. Delete all game entities. I then checked the my app's admin portal datastore viewer tab and I see that there are no more entities in my data store.

3. Upload the game kind using the game.csv created using (command is the same as download_data but with upload_data):

   appcfg.py upload_data --config_file=bulkloader.yaml --kind=Game --filename=game.csv --application=MyAppId --url=http://MyAppId.appspot.com/remote_api --rps_limit=500 --bandwidth_limit=2500000 --batch_size=100

4. Run a servlet that retrieves an entity by 'name' (this is the property shown in Game.java below).

The following error occurs:

Uncaught exception from servlet
java.lang.IllegalStateException: Loaded Entity has name but com.example.app.model.Game has no String @Id
    at com.googlecode.objectify.impl.ConcreteEntityMetadata.setKey(ConcreteEntityMetadata.java:343)
    at com.googlecode.objectify.impl.ConcreteEntityMetadata.toObject(ConcreteEntityMetadata.java:210)
    at com.googlecode.objectify.impl.QueryImpl$ToObjectIterator.translate(QueryImpl.java:640)
    at com.googlecode.objectify.impl.QueryImpl$ToObjectIterator.translate(QueryImpl.java:629)
    at com.googlecode.objectify.util.TranslatingIterator.next(TranslatingIterator.java:35)
    at com.googlecode.objectify.impl.QueryImpl.list(QueryImpl.java:470)

I suspect it has something to do with my bulkloader.yaml file not being configure correct for the __key__ property. So I've posted it below:

- import: google.appengine.ext.bulkload.transform
- import: google.appengine.ext.bulkload.bulkloader_wizard
- import: google.appengine.ext.db
- import: google.appengine.api.datastore
- import: google.appengine.api.users

transformers:

- kind: Game
  connector: csv
  connector_options:
    # TODO: Add connector options here--these are specific to each connector.
  property_map:
    - property: __key__
      external_name: key
      export_transform: transform.key_id_or_name_as_string

    - property: __scatter__
      #external_name: __scatter__
      # Type: ShortBlob Stats: 56 properties of this type in this kind.

    - property: genre
      external_name: genre
      # Type: String Stats: 6639 properties of thi开发者_如何学Gos type in this kind.

    - property: name
      external_name: name
      # Type: String Stats: 6639 properties of this type in this kind.

    - property: releasedate
      external_name: releasedate
      # Type: Date/Time Stats: 6548 properties of this type in this kind.
      import_transform: transform.import_date_time('%Y-%m-%dT%H:%M:%S')
      export_transform: transform.export_date_time('%Y-%m-%dT%H:%M:%S')

Prior to the deletion and upload of data, the "Game" kind (within the datastore viewer tab of the app's admin portal) shows the ID/NAME column with hyperlinks that look like ID=12345, ID=67890 ... for each entitiy row.

After the upload the datastire viewer shows the ID/NAME column with hyperlinks that look like NAME=12345, NAME=67890 ... for each entitiy row.

Possible cause?

I've actually posted my question on, https://groups.google.com/forum/?hl=en#!topic/objectify-appengine/FFuB2Onfnzc but we don't know the syntax to avoid transforming the __key__ property to string when uploading back the data.

Jeff (creator of Objectify3.0) says:

The error message says that the data in the datastore as a String name key, but your Game entity has a numeric Long @Id. I don't really know the syntax for the bulk loader, but the most suspect line is this one:

export_transform: transform.key_id_or_name_as_string

It looks like you are converting all numeric ids to strings here, which would be your problem. Leave them as numbers.

I've tried removing the export_transform: transform.key_id_or_name_as_string but when I do the download the csv file is empty.

Maybe a solution is to avoid using the --config_file parameter and bulkloader.yaml file and just let the bulkloader download/upload the data without any transformations?

Extra information incase it might be relevant

I'm using the objectify3.0.jar to manipulate my GAE datastore. So my Game kind looks like this:

public class Game {
    @Id private Long id; //This is my key property, auto generated by objectify  
    private String name;
    private String genre; 
    private Date releasedate;

    //ommitting getters and setters 
}

I did some experiements and I believe I have the solution.

I actually got the idea from another stackoverflow post I asked: Using Java Google App Engine bulkloader to download entire datastore to one csv file

The fix is to avoid using the --config_file and bulkloader.yaml.

I used the following to download every kind to a single csv file:

appcfg.py download_data --filename=backup.csv --application=MyAppId --url=http://MyAppId.appspot.com/remote_api --rps_limit=500 --bandwidth_limit=2500000 --batch_size=100

I used the following to upload the single csv file back to the datastore:

appcfg.py upload_data --filename=backup.csv --application=MyAppId --url=http://MyAppId.appspot.com/remote_api --rps_limit=500 --bandwidth_limit=2500000 --batch_size=100

They are the same commands but just download_data and upload_data swapped around.

The idea is just to let appcfg download and upload all entities (not being kind specific) i.e. not using any export or import transformations.

To avoid conversion from Id (Integer) to Name in the key at import time, have a look at the following post:

Bulkloader: How to define the transform.create_foreign_key() as int?