开发者

App Engine Bulk Data Upload URLError: "Connection reset by peer"

I'm trying to upload some records to my local data store using appcfg.py

Only a small number of records are actually inserted and I get the following output (with a ton of errors):

       $ appcfg.py upload_data --config_file=bulkloader.yaml --filename=/output.csv --kind=AutoCompleteIndex --url=http://localhost:8084/remote_api .
        Application: theapp1; version: dev.
        Uploading data records.
        [INFO    ] Logging to bulkloader-log-20100808.170223
        [INFO    ] Throttling transfers:
        [INFO    ] Bandwidth: 250000 bytes/second
        [INFO    ] HTTP connections: 8/second
        [INFO    ] Entities inserted/fetched/modified: 20/second
        [INFO    ] Batch Size: 10
        [INFO    ] Opening database: bulkloader-progress-20100808.170223.sql3
        Please enter login credentials for localhost
        Email: kjuyg
        Password for kjuyg: 
        [INFO    ] Connecting to localhost:8084/remote_api
        [INFO    ] Starting import; maximum 10 entities per post
        .......................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/ap开发者_运维问答pengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .............[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put

.... etc .....
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C..[INFO    ] An error occurred. Shutting down...
        ..........
        [INFO    ] 19580 entites total, 0 previously transferred
        [INFO    ] 18570 entities (3593152 bytes) transferred in 225.1 seconds
        [INFO    ] Some entities not successfully transferred

Any idea what the problem could be?


I had the exact same problem on my Mac. I solved it by creating a proxy, by putting the following in my httpd-vhost.conf file (in etc/apache2/extra/):

<Location /remote_api>
  ProxyPass http://localhost:8888/remote_api
</Location>

provided your test server is running on port 8888.

Then I run:

sudo apachectl restart

to restart the apache server.

Now this command ("Survey" is one of the data items):

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/appcfg.py upload_data /
--kind=Survey --file=Survey.csv /
--url=http://localhost/remote_api --email=test@example.com /
--config_file=config.yml 

gives this output:

[INFO    ] Logging to bulkloader-log-20121114.100837
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Password for test@example.com: 
[INFO    ] Opening database: bulkloader-progress-20121114.100837.sql3
[INFO    ] Connecting to localhost/remote_api
[INFO    ] Starting import; maximum 10 entities per post
....
[INFO    ] 36 entities total, 0 previously transferred
[INFO    ] 36 entities (15402 bytes) transferred in 1.7 seconds
[INFO    ] All entities successfully transferred

As it should be.


Look at your server logs. Are you getting a heap size exceeded or OutOfMemmory exception? This will happen on the local dev server if your upload is large because the dev server keeps the datastore in main memory. The server will hang for a bit and then you'll get a connection reset. I've had the dev server run out of memory with as little as 10MB of datastore data, since it uses a lot of extra RAM to organize it and store metadata. You can increase the size of the heap using the -Xmx switch.

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜