Search code examples
pythonredditpraw

Getting more than 100 search results with PRAW?

I'm using the following code to obtain reddit search results with PRAW 4.4.0:

params = {'sort':'new', 'time_filter':'year'}
return reddit.subreddit(subreddit).search('', **params)

I'd like to scrape an indefinite amount of posts from the subreddit, for a period of up to a year. Reddit's search functionality (and correspondingly, their API) achieves this with the 'after' parameter. However, the above search function doesn't accept 'after' as a parameter. Is there a way to use PRAW's .search() to obtain more than 100 search results?

Solution

  • Yes, by sending parameter limit=None will increase that to 1000, but will not guarantee any timeframe and no way to grab more that 1000. However you can use cloudsearch syntax. It is described in detail in reddit wiki https://www.reddit.com/wiki/search#wiki_cloudsearch_syntax and is pretty powerful search enhancer.

    To support it with some code, example usage like this case can be achieved in this way:

    import datetime
params = {'sort':'new', 'limit':None, 'syntax':'cloudsearch'}
time_now = datetime.datetime.now()
return reddit.subreddit(subreddit).search('timestamp:{0}..{1}'.format(
    int((time_now - datetime.timedelta(days=365)).timestamp()),
    int(time_now.timestamp())),
    **params)

    This has limit of 1000 results per query, but due to specified timeframe you can query multiple times for different timeframes. I.e. grab 1000 submissions, get utc_time from oldest one and send that time as first parameter for timestamp, which will give you results starting at the point in time that your last query stopped.