I have a list of approximately 1.500 twitter accounts (that may or may not have tweeted) for which I want to retrieve the last (max 100 tweets) every ~20 minutes. Considering the rate limits of Twitter API v.2, what is the most efficient way of doing this without hitting the rate limits (https://developer.twitter.com/en/docs/twitter-api/rate-limits)?
As far as I understand, there is no way of getting tweets from multiple users at the same time using https://api.twitter.com/2/users/<twitter id>/tweets and iterating through the 1.500 accounts to get the last tweets will make you hit the rate limit of ~900 requests per 15 minutes.
Is there a bulk request that can do this? Is adding them all to a Twitter list and get the latest tweets from there the only real option here?
I am needing this for a Node.js application but the issue is more about how to solve it at a Twitter API level.
The Twitter search API is publicly available at /2/tweets/search/all. You can also use /2/tweets/search/recent.
Using this, you can search from tweets from multiple accounts at once using their OR operator:
(from:twitter OR from:elonmusk)
Returns:
{
"data": [
{
"id": "1540059169771978754",
"text": "we would know"
},
{
"id": "1540058653155278849",
"text": "ratios build character"
},
{
"id": "1539759270501023744",
"text": "RT @NASA: The landmark law #TitleIX opened up a universe of possibility for women, including Janet Petro, the 1st woman director of @NASAKe…"
},
// ...
Note, this has a more strict rate limit, and you will have a limit of how many characters you can use in your search (probably 512).
You can add extra fields like author_id from tweet.fields, if you need them.
If you cannot get by with this, then you may be able to combine API endpoints, since rate limits are applied per-endpoint. For example, search half via the searching endpoint, and the other half via the individual user endpoints.
If this still doesn't work, you're right (from everything that I've found), you will need to either: