MongoDb : Keeping an array on fixed length inside an object , with fifo policy and some other conditions?

I have some scripts which update, mongoDb records which look like this :

{ "_id" : "c12345", "arr" : [
        {
                "i" : 270099850,
                "a" : 772,

        },
        {
                "i" : 286855630,
                "a" : 622,

        }
] }

The scripts append elements in the "arr" array of the object,using "pushAll" which works fine and is very fast.

My requirement: 1. Keep modifying these objects, but process them once the size of arr exceeds 1000.

1. When arr exceeds 1000,I choose some important records, discard some less important ones, and discard some old ones, and reduce the size of arr to 500 .

Current implementation: 1. Script A takes some data from somewhere and finds the object in another collection using "_id" field, and appends th开发者_运维问答at data into "arr" array.

1. The same script when finds the element,checks for size of "arr", if less than 1000, it does a normal append to arr, else proceeds to processing of PHP object retreived through find,modifies it, and updates the mongo record using "SET".

Current bottlenecks: 1. I want the updating script to run very fast. Upserts are fast, however the find and modifying operations are slower for each record.

Ideas in mind: 1. Instead of processing EXCEEDED items within the scripts, set a bool flag in the object, and process it using a seperate Data Cleaner script. ( but this also requires me to FIND the object before doing UPSERT ).

1. always maintain a COUNT variable in the object,which stores current length of "arr", and use it in Data cleaner script which cleans all the objects fetched through a mongodb query "count" > 1000. ( As mongodb does not allow $size operator to have Ranges, and only equal condition currently, I need to have my own COUNT counter)

Any other clean and efficient ideas you can suggest ? Thanks .

In version 2.3.2 of mongo a new feature has been added. There is now a $slice that can be used to keep an array to a fixed size.

E.g.:

t.update( {_id:7}, { $push: { x: { $each: [ {a:{b:3}} ], $slice:-2, $sort: {'a.b':1} } } } )

There's no easy way to do this, however, this is a good idea:

1. Instead of processing EXCEEDED items within the scripts, set a bool flag in the object, and process it using a seperate Data Cleaner script.

Running a separate script definitely makes sense for this.

MongoDB does not have a method for "fixed-length" arrays. But it definitely does not have a method for doing something like this:

choose some important records, discard some less important ones, and discard some old ones

The only exception I would make is the "bool" flag. You probably want just a straight counter. If you can index on this counter then it should be fast to find those arrays that are "too big".