Get tweets from topics section with Twitter API

Question

I want to get tweets from a specific topic from the topics section. But I couldn't find this on the twitter api page or the tweepy website. how can I get this tweet with tweepy or twitter api? Below picture shows the topics I followed. I need to select one topic from here and save these tweets for the project.

Twitter topics example

Answer 1

In order to build something like this, you would use Tweet annotations, which are a part of Twitter API v2 (link to the official documentation). These annotations are used inside Twitter to power the Topics features, but they may not always be a direct mapping.

Note that not all annotations are available via the API, as mentioned in this previous answer.

Let's take your example - you're looking at the Topics page showing what you are following. I also follow Augmented Reality, so I clicked through that link to a timeline of Tweets in that topic, and chose the an individual Tweet I saw, Tweet ID 1582925073509281792. I tried to pick one without a lot of links and hashtags to keep the response shorter, otherwise there would be a lot more entity values returned.

Then I used the API to get that Tweet and the attached annotations (this is the basic HTTP version not a Tweepy or Python version):

GET /2/tweets/1582925073509281792?tweet.fields=context_annotations,entities

The result:

{
  "data": {
    "entities": {
      "mentions": [
        {
          "start": 0,
          "end": 8,
          "username": "tipatat",
          "id": "94606087"
        }
      ]
    },
    "id": "1582925073509281792",
    "edit_history_tweet_ids": [
      "1582925073509281792"
    ],
    "text": "@tipatat AR is very much a part of the larger system we think of as the Metaverse…",
    "context_annotations": [
      {
        "domain": {
          "id": "46",
          "name": "Business Taxonomy",
          "description": "Categories within Brand Verticals that narrow down the scope of Brands"
        },
        "entity": {
          "id": "1557697333571112960",
          "name": "Technology Business",
          "description": "Brands, companies, advertisers and every non-person handle with the profit intent related to softwares, apps, communication equipments, hardwares"
        }
      },
      {
        "domain": {
          "id": "30",
          "name": "Entities [Entity Service]",
          "description": "Entity Service top level domain, every item that is in Entity Service should be in this domain"
        },
        "entity": {
          "id": "848920371311001600",
          "name": "Technology",
          "description": "Technology and computing"
        }
      },
      {
        "domain": {
          "id": "131",
          "name": "Unified Twitter Taxonomy",
          "description": "A taxonomy of user interests. "
        },
        "entity": {
          "id": "848920371311001600",
          "name": "Technology",
          "description": "Technology and computing"
        }
      },
      {
        "domain": {
          "id": "131",
          "name": "Unified Twitter Taxonomy",
          "description": "A taxonomy of user interests. "
        },
        "entity": {
          "id": "1427745203700469767",
          "name": "Metaverse"
        }
      },
      {
        "domain": {
          "id": "165",
          "name": "Technology",
          "description": "for individual and types of technology, e.g., food technology, 3D printing"
        },
        "entity": {
          "id": "848920371311001600",
          "name": "Technology",
          "description": "Technology and computing"
        }
      }
    ]
  }
}

That's still long set of responses, but we can start to find things we might be interested in from here. Each annotation has a top-level domain and then an individual id within that domain. If we look down the list we can see that domain 131 with entity id 1427745203700469767 seems to be about "Metaverse". Now we have a way to search for Tweets that Twitter has identified as related to "Metaverse"!

If we run a search using context of 131.1427745203700469767 (this is domain_id.entity_id) then we can get back some related Tweets. I'll add in the text augmented reality to narrow things down. Again I've just used the HTTP format here, you can modify this for the library of your choice.

GET /2/tweets/search/recent?query=context:131.1427745203700469767 augmented reality

That call returns 10 Tweets (by default) on the topic of Metaverse, in this case.

You will need to experiment with identifying usable context ids - not every topic is directly reflected by contexts in the API, and this will not exactly match what you might see on a particular tab in the Twitter app itself.

Here's a link to a useful post about how to do something similar: How to search for Tweets about various 'Topics' using the Twitter API v2. I also recommend using Twitter's API tools and API Playground to test this out.