Testing MongoDB clustering with Morphia

I have been trying out MongoDB in a replica-set configuration to see how it scales/performs/copes.

I have been using Morphia (a POJO mapping layer on top of Mongo's Java drivers) to persist 10,000 simple random documents to a single collection. I have annotated my POJO (MyData in the snippet below) with the @Entity(concern="REPLICAS_SAFE") annotation, in the hope that the data sent to the database will be safely persisted.

My POJO was made up of the ObjectId field (Mongo's primary key type), a String of random characters of random length (maximum 20 characters) and a long generated using Random.nextLong().

My code is the following:

for (int i=0;i<10000;i++) {
    final MyData data = new MyData();

    boolean written = false;
    do {
        try {
        ds.save(data); //ds is of type DataStore
        written=true;
        } catch (Exception e) {
            continue;
        }
    }
    while (!wr开发者_Go百科itten);
}

I set up a four-node replica-set cluster, ran the above program, and then started metaphorically pulling out cables to see what happened.

The desired result was the program running until it had successfully persisted all documents to the database.

The actual result, after a few goes, was one of:

• Java reporting it had committed 10k entries but the database only having <10k
• Java reporting it had committed <10k and the database reporting either the same value or even less
• Everything working fine

In one case, nodes that were brought back up were unable to actually catch up with the PRIMARY node and had to be started from scratch with a deleted database. This was despite increasing the opfile parameter to 2 gigs, which I would have thought would have been enough to replay 10,000 rows of very simple data.

Other things you should know:

1. All of this is running on a single piece of hardware (2 gig Pentium D!) with the cluster running on two 32-bit Ubuntu Server VirtualBox instances with 128 megs of ram each and the Java client running inside the Windows XP host. Two mongod processes ran on each virtual machine, plus an arbiter was running on one virtual machine as well.
2. The clocks on the two virtualised machines were off by a few seconds (I need to install VirtualBox Guest Additions to fix this), but not a large amount - 10gen say the time shouldn't be an issue for clustering, but I thought I'd mention it.

I am aware of the 2 gig limitation with Mongo on a 32-bit machine, the fact that other people have had disappearing records, and I'm aware that the machine I'm doing these tests on is not exactly in the Top 500 (which is why the data I chose to persist was small) but when my tests worked, they worked very well.

Are the problems I had proof that Mongo isn't ready for primetime yet, or am I doing something inherently wrong?

I'm using 1.6.5.

Any insight, hints, tips, pointers, explanation or criticism greatly appreciated!

ps: I'm not trolling - I really like the idea of NoSQL for the kinds of data it is good for, so I really want it to work, but so far I'm not having much luck!

MongoDB is definitely being used "in primetime" in lots of places right now. So it's worth taking a look at what else may be going on here.

So a couple of starter questions here:

1. How does "new MyData()" work? Is it possible that you're hammering existing IDs?
2. Are your replica sets "up" throughout the process? You're just "continuing" on error, so I'm not really sure how errors are being handled. Is Morphia correctly bubbling errors?

I really appreciate that you've gone through and written a kind of "test case", but I think you need to dig one step deeper with the case. Would you be able to try the following two things?

1. Set the _id on MyData to i. This way you can see where in the process you're dying.
2. Do a console.write or equivalent every time that you get an error. See if you can't figure out where the data has actually gone.
3. By the same measure do a console.write on every successful save.

If you do these steps, you'll get a log of what's going on and you'll be able to see what is or isn't saved and compare that to the data in the DB.

I understand that this is all a little tedious, but I think you have one of two problems and doing these steps will help clear this up.

Either 1. Morphia isn't reporting errors correctly (not being handled correctly) 2. You're finding an actual issue with replica sets 3. You're being caught by the "eventual consistency".

Either way, with more details, we should be able to drill into the problem.