MongoDB and PyMongo - Searching for documents from a given date and time

I have been working on this for a while but I suspect I don't understand the right way of doing it.

I am trying to find all the data in a collection from a given date and time.

The date and time are stored in ISO Date format. For example: ISODate("2015-01-07T23:39:000Z")

The following command works in the MongoDB shell:

db.sensors.find({ "Date" : { "$gte" : ISODate("2014-12-31T02:15:00Z") }})

However, I can not get this to work in Python. It looks like MongoDB is expecting the full ISODate format: ISODate("2014-01-07T23:39:000Z"). If I just just give it the date without ISODate("") it won't return anything.

I created a variable and escaped the quote. This printed perfectly but when I tried to use it in the creation of the cursor it included the escape characters as well!

So, I tried to escape the quotes when creating the cursor object. I'm having no luck doing it this way but I don't think I'm going down the right track. It seems unlikely that every time someone needs to search for records for a specific date and time they would need to jump through these hoops. Can you tell me what I'm doing wrong here? Below, I've included a few examples of what I've tried. Below this, I've shown the output provided when I profile MongoDB to see what Python is passing through to it. At the end, I show that I previously got this working using PHP but for various reasons, I'm no longer using PHP.

date1 = datetime.datetime.utcnow()-datetime.timedelta(minutes=15)
date1 = date1.strftime("%Y-%m-%dT%H:%M:%S.000Z")
for cursor in sensors.find({'Date':{"$gte":'ISODate("'date1')"'}}):

First line is defining the date and time minus 15 minutes. Second line is converting it into the right format. Third line is creating the cursor and adding the ISODate("") component.

Next example:

date1 = datetime.datetime.utcnow()-datetime.timedelta(minutes=15)
date1 = date1.strftime("%Y-%m-%dT%H:%M:%S.000Z")
for cursor in  sensors.find({'Date':{"$gte":date1}}):

First and second lines are doing the same. The third line is creating the cursor but not actually adding the ISODate component.

Here's another example of the third line. Again, as you can see I was trying to escape the quotes.

date1 = 'ISODate(\"' + date1 + '\")'

Information on enabling and using profiling in MongoDB. http://docs.mongodb.org/manual/tutorial/manage-the-database-profiler/#DatabaseProfiler-EnablingProfiling

db.setProfilingLevel(2) // Turn on verbose profiling. 
db.setProfilingLevel(0)  // Turn off profiling. 
db.system.profile.find()  // Show the contents of the profile collection.

I saw that someone else has been retrieving data from two given date and time slots but I couldn't follow the code. Retrieve data from mongodb based on date condition in pymongo

Here is a blog post I wrote when I finally got this working in PHP. http://www.digitaldarragh.com/2014/02/27/using-date-ranges-in-mongodb-and-php/

Any suggestions very welcome.

Sorry for the long question but I thought it was better to explain it well and to show that I've spent time looking into this before resorting to asking on a forum.

Solution

ISODate is just a MongoDB shell construct for representing MongoDB date and time data. That data is stored in the collection docs as a BSON Date data type, which is a 64-bit integer count of the number of milliseconds since the Unix epoch.

When querying against these BSON Date fields, you use the native datetime data type for the language, not a string. So in your pymongo case, you leave date1 as a datetime and it's just:

date1 = datetime.datetime.utcnow()-datetime.timedelta(minutes=15)
for cursor in sensors.find({'Date':{"$gte": date1}}):
    ...