So I received data that was stored in mongoDB then exported as a JSON via the bson.json_util.dumps() function. When I try to reconvert it back into datetime objects, none of the usual parsers work. Here's a sample:
{
u'_group': 0,
u'_range': u'',
u'canon': 0,
u'comment_id': 0,
u'created': {u'$date': 1491468607000L},
u'description': u''
}
and all the dates look like this:
[{u'$date': 1491457629000L}, {u'$date': 1491458993000L}, {u'$date': 1491457072000L}, {u'$date': 1491457035000L}, {u'$date': 1491457330000L}, {u'$date': 1491458323000L}, {u'$date': 1491458147000L}, {u'$date': 1491458277000L}, {u'$date': 1491459839000L}, {u'$date': 1491466340000L}, {u'$date': 1491463804000L}, {u'$date': 1491464304000L}, {u'$date': 1491465145000L}, {u'$date': 1492073749000L}, {u'$date': 1492073750000L}, {u'$date': 1492075415000L}, {u'$date': 1492155813000L}, {u'$date': 1492608582000L}, {u'$date': 1492671698000L}, {u'$date': 1493001047000L}, {u'$date': 1493714117000L}]
My tests:
y = {u'$date': 1491457629000L}
>>> y['$date']
1491457629000L
And various failed attempts:
print(
datetime.datetime.fromtimestamp(
y['$date']
).strftime('%Y-%m-%d %H:%M:%S')
)
ValueError: timestamp out of range for platform localtime()/gmtime() function
from dateutil import parser
parser.parse(str(y['$date']))
OverflowError: Python int too large to convert to C long
utc_time = datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=y['$date'])
OverflowError: date value out of range
It seems like any unix timestamp converter should be able to read this. Why isn't it working? I also only need the year and month, so timezone specificity is outside the scope of my demands.
Note (to the downvoter): This is not a duplicate of converting epoch time with milliseconds to datetime because the solution here was to use a different JSON parser that mirrors the way it was converted out of mongodb, instead of trying to reformat it with an unrelated timestamp parser. I had already researched the other link and found it didn't solve my problem.
This worked. I don't know why this particular timestamp format won't convert with other tools,
{u'$date': 1491457629000L}
but reversing the json_util.dumps() method with its own corresponding json_util.loads() method works. It have to do that on the requested data instead of using standard .json() method in python's requests module:
from bson import json_util
import requests
raw = requests.get('someurlhere')
data = json_util.loads(raw.text)
{...
u'created': datetime.datetime(2017, 4, 6, 5, 47, 9, tzinfo=<bson.tz_util.FixedOffset object at 0x0000000002FAAC18>),
...}
That yields a datetime object and I can work with that.