Search code examples
matlabresttwitter

Using additional request parameters in Twitter Premium Search API

I am using the Twitter API from Matlab, specifically by means of the twitter class from the Datafeed Toolbox.

I have essentially followed the example code from the official documentation. I created a Twitter app in my Twitter developer page, and obtained its API keys and access tokens. With those I can use the Twitter Standard search API from Matlab:

c = twitter(consumerkey,consumersecret,accesstoken,accesstokensecret);
% The variables 'consumerkey' etc are defined as character vectors
s = search(c,tweetquery,'count',100); % this works

Now I want to use the Premium search API. This has two endpoints for accessing Tweets:

  • 30-day endpoint: provides Tweets from the previous 30 days.
  • Full-archive endpoint: provides complete and instant access to Tweets dating all the way back to the first Tweet in March 2006.

In addition, the Premium API has two tiers of access:

  • Free Sandbox access that enables initial testing and development.
  • Paid Premium access that provides increased access.

The link above specifies the restrictions associated to sandbox as compared with paid access.

I am trying to use the full-archive endpoint with sandbox access. For that I had to create a developer environment on Twitter, which I named dev.

The search method in Matlab's twitter class (which worked for the Standard access, as described above) doesn't seem to work with the Premium access. But I noticed that search actually calls getdata, and the latter does work for Premium access as follows. First, the Premium access URL needs to be defined:

c.URL = 'https://api.twitter.com/1.1/tweets/search/fullarchive/dev.json';

and then the following syntax works:

s = getdata(c,c.URL,'query','Jimi Hendrix'); % this works

I have also been able to add operators within the query string, for example to specify a range of geographical positions or to restrict the search to tweets that contain images:

s = getdata(c,c.URL,'query','place:"Palo Alto"'); % this works
s = getdata(c,c.URL,'query','Robert Smith bounding_box:[-0.2 51.4 0.1 51.6]') % this works

However—and this is my question—, I haven't been able to use additional request parameters defined in the Twitter API to refine the search, such as fromDate, toDate or maxResults:

s = getdata(c,c.URL,'query','John Frusciante', 'fromDate', '201708130000') % doesn't work
s = getdata(c,c.URL,'query','Rob Scallon', ...
    'fromDate', '201708130000', 'toDate', '201708150000') % doesn't work
s = getdata(c,c.URL,'query','Michael Lemmo', 'maxResults', '20') % doesn't work

All of the above return an HTTP/1.1 422 Unprocessable Entity error.

Is my syntax not correct? Maybe the fromDate etc parameters have to be part of the query string? Or maybe the sandbox tier of the Premium search doesn't support those parameters?

For context, I don't really know what all those terms like endpoint, tier, developer environment and token mean, but still I'd like to make this work.

Solution

  • Going by the description at https://developer.twitter.com/en/docs/tweets/search/api-reference/premium-search#DataParameters, what you call 'addition request parameters' are defined for requests of type POST /search/:product. These are HTTP POST requests, can you try using postdata (https://in.mathworks.com/help/datafeed/twitter.postdata.html) instead of getdata. Their usage is almost identical.