开发者

GAE mapper producing 'No quota, aborting' errors

I am trying to setup a mapper job on Google app engine using the mapper framework here (java version): http://code.google.com/p/appengine-mapreduce/

I kicking the job off via code like such:

Configuration conf = new Configuration(false);
conf.setClass("mapreduce.map.class", MyMapper.class, Mapper.class);
conf.setClass("mapreduce.inputformat.class", DatastoreInputFormat.class, InputFormat.class);
conf.set(DatastoreInputFormat.ENTITY_KIND_KEY, "Organization");

// Queue up the mapper request.
String configString = ConfigurationXmlUtil.convertConfigurationToXml(conf);
Queue queue = GaeQueueFactory.getQueue(QUEUE_NAME);
queue.add(
        Builder.url("/mapre开发者_开发问答duce/start")
        .param("configuration", configString));

I get the following error in the logs on both the dev server and prod server:

com.google.appengine.tools.mapreduce.MapReduceServlet processMapper: No quota. Aborting!

There is no additional stack trace. This appears about a dozen or so times each time I try kick a job off.


I think you don't have enough quota to process your Mapper job with the predefined default values.

Try to lower these configuration parameters:

mapreduce.mapper.inputprocessingrate
The aggregate number of entities processed per second by all mappers.
Used to prevent large amounts of quota being used up in a short time period.
Default 1000

mapreduce.mapper.shardcount
The number of concurrent workers to use.
This also determines the number of shards to split the input into
Default = 8


I have figured out the issue and am recording this here for anyone else having this problem. What confused me the most about this is that it was working several weeks ago and seems to have stopped sometime recently. Because the app isnt being used in a production environment, no one noticed.

What changed was that I was sending map requests to a custom 'mapper' task queue; not the default task queue. Because I have several mapper jobs, I setup a different queue for all mapper jobs to use:

<queue>
    <name>mapper</name>
    <rate>5/s</rate>
    <bucket-size>10</bucket-size>
</queue>

When I switched the code back to using the default queue, everything worked as expected. I have filed a bug with the mapper team here: http://code.google.com/p/appengine-mapreduce/issues/detail?id=73

0

上一篇:

下一篇:

精彩评论

暂无评论...
验证码 换一张
取 消

最新问答

问答排行榜