开发者

Google App Engine Bulk download

I was downloading data(more than 1 GB) from datastore using bulk download. Suddenly, my internet stopped working and download process stopped in middle. I want to resume from where it stopped. When I try, I get the following error

File "/Users/FYP/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore_types.py", line 156, in ValidateString
    (name, value, typename(value)))
BadArgumentError: kind should be a string; received 3 (a int):
[INFO    ] [WorkerThread-2] Backing off due to errors: 1.0 seconds
开发者_StackOverflow中文版[INFO    ] An error occurred. Shutting down...
[ERROR   ] Error in WorkerThread-0: kind should be a string; received 3 (a int):

This is my code to download the data

appcfg.py download_data --config_file=bulkloader.yaml --batch_size=200 --filename=final80_2.csv 
--kind=TasksTime1 --url=http://abc.appspot.com/_ah/remote_api --rps_limit=40 
--db_filename=bulkloader-progress-20110429.141103 --result_db_filename=bulkloader-results-20110429.141103

How to solve the problem ?


Just re-run the same command. Per the docs:

If the transfer is interrupted, you can resume the transfer from where it left off using the --db_filename=... argument. The value is the name of the progress file created by the tool, which is either a name you provided with the --db_filename argument when you started the transfer, or a default name that includes a timestamp. This assumes you have sqlite3 installed, and did not disable the progress file with --db_filename=skip.

You specified a db_filename on the first execution, so if you re-run the command, it should find the existing file and pick up where it left off.


I've had the same issue:

Error in WorkerThread-0: kind should be a string; received 3 (a int)

Removing all *.sql3 and *.csv files stopped this error from occurring. Obviously, yes, the resuming is broken, but if you want to be able to bulk download again you'll have to disable it.

As a matter of course, we normally wrap the bulkloader commands in a function to clean these files beforehand.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜