MongoImport Dates Occurring Before The Epoch

I am writing a utility at work which converts our relational DB at work to a complex JSON object and dumps to files grouped by subject. I then would like to import these files into MongoDB collections using the mongoimport tool.

Our data includ开发者_StackOverflowes timestamps which represent dates occurring before the epoch, the appropriate JSON representation of which yields negative numbers. While MongoDB itself will handle these fine, the import tools JSON parser uses unsigned long long variables and fails.

If you use Mongo's special JSON date representation format ({"key": { "$date": "value_in_ticks" } }), the import tool will throw an error on those documents and skip the import. You can also use the JavaScript date notation ({"key": new Date(value_in_ticks) }) which will be successfully imported but parsed as an unsigned value creating a garbage date.

The special date format fails because of an assertion checking for reserved words. This code is reached because the presence of the negative sign at the beginning of the value causes the special date parsing to exit and return to normal document parsing.

The code to parse JSON dates explicitly calls the boost library uint_parser. There exists a signed version of this function and an issue on their JIRA tracker already exists to utilize it (on which I commented that I would attempt).

Short of diving into the code immediately to try and update this to be signed, is there an alternate route that I can take to load these dates for now?

I want to run this nightly via cron for a few months for testing so I would prefer it be very easy. These dates exist in many different parts of documents in many different collections so the solution should be generalized.

A little late to the party, but I have just come up against the same issue.

My workaround was to import the dates as strings (e.g. "1950-01-01"), and to script the conversion using Ruby on Rails with Mongoid:

Dates.each do |d|
  d.mydate = d.mydate.to_date
  d.save
end

Hopefully you can adapt this to whatever language/framework you are using.

This Python snippet works for me.

import time, struct

def bson_datetime(adatetime):
    try:
        ret = int(1000*(time.mktime(adatetime.timetuple()) + 3600))
        if ret < 0:
            ret = struct.unpack('Q', struct.pack('q', ret))[0]
        return {'$date': ret}
    except ValueError:
        return None

I.e.

import datetime
print bson_datetime(datetime.datetime(1950, 12, 30, 0, 0))

yields {"abc" : {"$date" : 18446743473920751616}}.

Step 1: go to groups.google.com/group/mongodb-user and post the issue "mongoimport does not support dates before the epoch". Response times on the groups tend to be very good.

Step 2: think of running dates in a universally accepted format like "1964-04-25 13:23:12"

It will take a little bit more space in MongoDB because you'll be storing string. However it should be easy to interpret for anyone pulling out the data.