MongoDB setShardVersion failed, mongodump only

I am use sharding (over replica set) and try to dump the database. Sharding enabled for mycms-prod.fs.chunks by key files_id. Additional info: http://groups.google.com/group/mongodb-user/browse_thread/thread/a8f05cbf495d6487 I have readed this instruction: http://www.mongodb.org/display/DOCS/Backing+Up+Sharded+Cluster (for Small clusters).

$ /opt/mongodb/bin/mongodump -h localhost:30000 -d mycms-prod 
.... 
Other collections that are not currently in sharding dumps ok. 
       mycms-prod.tracking_daystat to dump/mycms-prod/ 
tracking_daystat.bson 
                370 objects 
.... 
       mycms-prod.fs.chunks to dump/mycms-prod/fs.chunks.bson 
assertion: 11010 count fails:{ assertion: "setShardVersion failed 
host[server1.domain.com:28000] { errmsg: "not master...", 
assertionCode: 10429, errmsg: "db assertion failure", ok: 0 } 

And in mongos.log:

######################### 
Tue Apr 12 01:20:14 [mongosMain] connection accepted from 
127.0.0.1:42975 #27 
Tue Apr 12 01:20:15 [conn27]      setShardVersion failed 
host[server1.domain.com:28000] { errmsg: "not master", ok: 0.0 } 
Tue Apr 12 01:20:15 [conn27] Assertion: 10429:setShardVersion failed 
host[server1.domain.com:28000] { errmsg: "not master", ok: 0.0 } 
0x51f4a9 0x69b163 0x69acf2 0x69acf2 0x69acf2 0x576ba6 0x5774b6 
0x575630 0x575b31 0x65f661 0x57bdcc 0x631062 0x66432c 0x6761c7 
0x57ea3c 0x69ec30 0x3a9be0673d 0x3a9b6d40cd 
 /opt/mongodb/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x129) [0x51f4a9] 
 /opt/mongodb/bin/mongos [0x69b163] 
 /opt/mongodb/bin/mongos [0x69acf2] 
 /opt/mongodb/bin/mongos [0x69acf2] 
 /opt/mongodb/bin/mongos [0x69acf2] 
 /opt/mongodb/bin/ 
mongos(_ZN5boost6detail8function17function_invoker4IPFbRN5mongo12DBClientBa seERKSsbiEbS5_S7_biE6invokeERNS1_15function_bufferES5_S7_bi 
+0x16) [0x576ba6] 
 /opt/mongodb/bin/ 
mongos(_ZN5mongo17ClientConnections13checkVersionsERKSs+0x1c6) 
[0x5774b6] 
 /opt/mongodb/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x2d0) 
[0x575630] 
 /opt/mongodb/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs 
+0xa1) [0x575b31] 
 /opt/mongodb/bin/ 
mongos(_ZN5mongo15dbgrid_pub_cmds8CountCmd3runERKSsRNS_7BSONObjERSsRNS_14BS ONObjBuilderEb 
+0x9e1) [0x65f661] 
 /opt/mongodb/bin/ 
mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONOb jBuilderE 
+0x67c) [0x57bdcc] 
 /opt/mongodb/bin/ 
mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262) 
[0x631062] 
 /opt/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x29c) [0x66432c] 
 /opt/mongodb/bin/ 
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21Abstract MessagingPortEPNS_9LastErrorE 
+0x77) [0x6761c7] 
 /opt/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE 
+0x34c) [0x57ea3c] 
 /opt/mongodb/bin/mongos(thread_proxy+0x80) [0x69ec30] 
 /lib64/libpthread.so.0 [0x3a9be0673d] 
 /lib64/libc.so.6(clone+0x6d) [0x3a9b6d40cd] 
Tue Apr 12 01:20:15 [conn27] end connection 127.0.0.1:42975 
######################### 

Then I'm goint to the server1.domain.com.

[moskrc@server9 db]$ /opt/mongodb/bin/mongo server1.domain.com:28000 
MongoDB shell version: 1.8.1 
connecting to: server1.domain.com:28000/test 
rs1:SECONDARY> use mycms-prod 
switched to db mycms-prod 
rs1:SECONDARY> db.fs.chunks.count() 
Tue Apr 12 01:22:23 uncaught exception: count failed: { "errmsg" : 
"not master", "ok" : 0 } 
开发者_开发技巧rs1:SECONDARY> 

I am think that mongos should not contact to this computer. Since it is SECONDARY to the replica set.

This is a bug?

All the components of version 1.8.1.

There is a nuance. This database is restored with the help of mongorestore.

I thought may be something wrong with the dump. I just created a new database shard-test, copied into it 500 files (in gridfs) and activated sharding.

Wait until all data is distributed across servers. And triedto make a dump the database . Surprisingly, it worked out! From different mongos works too. What does this mean?

This is console output: 
        { "_id" : "shard-test", "partitioned" : true, "primary" : "rs2" } 
                shard-test.fs.chunks chunks: 
                                rs1     3 
                                rs3     3 
                                rs2     5 
                        { "files_id" : { $minKey : 1 } } -->> { "files_id" : 
ObjectId("4da48f64d8b9bb5239000000") } on : rs1 { "t" : 2000, "i" : 
0 } 
                        { "files_id" : ObjectId("4da48f64d8b9bb5239000000") } -->> 
{ "files_id" : ObjectId("4da49002d8b9bb527400005d") } on : rs3 { "t" : 
3000, "i" : 0 } 
                        { "files_id" : ObjectId("4da49002d8b9bb527400005d") } -->> 
{ "files_id" : ObjectId("4da49006d8b9bb5274000132") } on : rs1 { "t" : 
4000, "i" : 0 } 
                        { "files_id" : ObjectId("4da49006d8b9bb5274000132") } -->> 
{ "files_id" : ObjectId("4da49009d8b9bb527400028e") } on : rs3 { "t" : 
5000, "i" : 0 } 
                        { "files_id" : ObjectId("4da49009d8b9bb527400028e") } -->> 
{ "files_id" : ObjectId("4da4900ed8b9bb52740003d9") } on : rs1 { "t" : 
6000, "i" : 0 } 
                        { "files_id" : ObjectId("4da4900ed8b9bb52740003d9") } -->> 
{ "files_id" : ObjectId("4da4902ad8b9bb5274000530") } on : rs3 { "t" : 
7000, "i" : 0 } 
                        { "files_id" : ObjectId("4da4902ad8b9bb5274000530") } -->> 
{ "files_id" : ObjectId("4da49032d8b9bb52740005e1") } on : rs2 { "t" : 
7000, "i" : 1 } 
                        { "files_id" : ObjectId("4da49032d8b9bb52740005e1") } -->> 
{ "files_id" : ObjectId("4da49039d8b9bb5274000697") } on : rs2 { "t" : 
2000, "i" : 2 } 
                        { "files_id" : ObjectId("4da49039d8b9bb5274000697") } -->> 
{ "files_id" : ObjectId("4da4906ed8b9bb5274000749") } on : rs2 { "t" : 
3000, "i" : 2 } 
                        { "files_id" : ObjectId("4da4906ed8b9bb5274000749") } -->> 
{ "files_id" : ObjectId("4da490a1d8b9bb52be000007") } on : rs2 { "t" : 
7000, "i" : 2 } 
                        { "files_id" : ObjectId("4da490a1d8b9bb52be000007") } -->> 
{ "files_id" : { $maxKey : 1 } } on : rs2 { "t" : 7000, "i" : 3 } 
> bye 

(env)[moskrc@server2 tmp]$ /opt/mongodb/bin/mongodump -h localhost: 
30000 -d shard-test 
connected to: localhost:30000 
DATABASE: shard-test     to     dump/shard-test 
        shard-test.system.indexes to dump/shard-test/system.indexes.bson 
                 4 objects 
        shard-test.fs.chunks to dump/shard-test/fs.chunks.bson 
                600/1496        40% 
                700/1496        46% 
                900/1496        60% 
                1100/1496       73% 
                1400/1496       93% 
                 1496 objects 
        shard-test.fs.files to dump/shard-test/fs.files.bson 
                 804 objects 

Noticed one more nuance.

If I do so:

[moskrc@server9 mycms-prod]$ /opt/mongodb/bin/mongodump -h localhost: 
30000 -d mycms-prod 
connected to: localhost:30000 
DATABASE: mycms-prod     to     dump/mycms-prod 
        mycms-prod.cms_comment to dump/mycms-prod/cms_comment.bson 
                 16 objects 
        mycms-prod.system.indexes to dump/mycms-prod/system.indexes.bson 
                 67 objects 
        mycms-prod.cms_pdfcontent to dump/mycms-prod/cms_pdfcontent.bson 
                 18 objects 
        mycms-prod.djangoratings_vote to dump/mycms-prod/ 
djangoratings_vote.bson 
                 25 objects 
        mycms-prod.auth_permission to dump/mycms-prod/auth_permission.bson 
                 192 objects 
        mycms-prod.tracking_pagevisit to dump/mycms-prod/ 
tracking_pagevisit.bson 
assertion: 11010 count fails:{ assertion: "setShardVersion failed 
host[server2.domain.com:28000] { errmsg: "not maste...", 
assertionCode: 10429, errmsg: "db assertion failure", ok: 0 } 

Error has occured on tracking_pagevisit collection.

Then... Let's try to dump this collection separately.

[moskrc@server9 mycms-prod]$ /opt/mongodb/bin/mongodump -h localhost: 
30000 -d mycms-prod -c tracking_pagevisit 
connected to: localhost:30000 
DATABASE: mycms-prod     to     dump/mycms-prod 
        mycms-prod.tracking_pagevisit to dump/mycms-prod/ 
tracking_pagevisit.bson 
                 14158 objects 

This work out!!! What's happening?

My system:

CentOS 5.5 

Kernel: Linux server9.domain.com 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux 

I have 6 mongos. Actually used always only 2. For 2 applications. Each application has its own mongos.

Total I have 9 servers. On each run mongod with params: shardsvr = true replSet = rs1 (rs2 and rs3). Three replicas. Each replica consists of a 3-mongod. And three config servers (server4.domain.com: 28001,server6.domain.com:28001,server1.domain.com:28001).

Mongos params bind_ip = 127.0.0.1,123.456.789.12 port = 30000 fork = true configdb = server4.domain.com:28001,server6.domain.com:28001,server1.domain.com: 28001

I restarted used mongos. That helped. Databases are now identical. But the dump is still not working. I wrote the above error.

Thanks.

It's was mongodb bug. See here: https://jira.mongodb.org/browse/SERVER-2961