Search code examples
c#-4.0twitterlinq-to-twitter

how to work with pagination using since_id and max_id in linqtotwitter v2.1?


I had used linqtotwitter for fetching tweets from twitter.
However i can pull tweets from only one shot at a time if i try to pull tweets using sinceid and maxid it returning empty and i'm also executing with range dates in query. with the below code i'm only able to get only 100 tweets.what i'm doing wrong
Thanks

My code

var auth = new SingleUserAuthorizer
        {
            Credentials = new InMemoryCredentials
            {
                ConsumerKey = "xxxxx",
                ConsumerSecret = "xxxxx",
                OAuthToken = "xxxxx",
                AccessToken = "xxxxxxx"
            }
        };
        var twitterCtx = new TwitterContext(auth);
        var ownTweets = new List<Status>();
        ulong sinceId = 0;
        ulong maxID = 0;
        int lastStatusCount = 0;
        var dateFrom = DateTime.Now.AddDays(-20);
        bool flag = true;
        var statusResponse = new List<Status>(); 

        statusResponse = (from tweet in twitterCtx.Status
                              where tweet.Type == StatusType.User
                                    && tweet.ScreenName == screenname
                                    && tweet.Count == 100
                                    && (tweet.CreatedAt >= dateFrom && tweet.CreatedAt <= DateTime.Now)
                              select tweet).ToList();

        if (statusResponse.Count > 0)
        {
            maxID = statusResponse.Min(status => ulong.Parse(status.StatusID)) - 1;
            ownTweets.AddRange(statusResponse);
            sinceId = Convert.ToUInt64(ownTweets.Last().StatusID);
        }
        do
        {
          int rateLimitStatus = twitterCtx.RateLimitRemaining;
          if (rateLimitStatus != 0)
          {
              statusResponse = (from tweet in twitterCtx.Status
                            where tweet.Type == StatusType.User
                                  && tweet.ScreenName == screenname
                                  && tweet.SinceID == sinceId && tweet.MaxID == maxID
                                  && tweet.Count == 100
                                  && (tweet.CreatedAt >= dateFrom && tweet.CreatedAt <= DateTime.Now)
                            select tweet).ToList();


                        lastStatusCount = statusResponse.Count;

                        if (lastStatusCount != 0)
                        {
                            maxID = statusResponse.Min(status => ulong.Parse(status.StatusID)) - 1;

                            ownTweets.AddRange(statusResponse);
                        }
                        else
                        {
                            flag = false;
                        }
            }
            else
            {
                flag = false;
            }

        }
        while (flag);

Solution

  • You can fix this by changing how you initialize SinceID, which represents the oldest tweet to retrieve. Remember, you're working backwards through the list, until you reach SinceID.

    1. Create a field for SinceID in your data store, associated with the user represented by screenname (or UserID).
    2. Initialize that field to 1. Don't set it to 0, because Tweet ID #0 doesn't exist and Twitter will return an error.
    3. In your code, initialize sinceID with the value from the data store for that user, which will be 1 the first time through.
    4. Remove any other changes to sinceID in your code, including the one inside the if (statusResponse.Count > 0) block.
    5. When your algorithm is complete, use that line you just removed from the if block to get what the next sinceID should be.
    6. Save that value of sinceID in your data store for this user so that you can read it again the next time you run this algorithm for that user.

    For more info, please visit my blog post Working with Timelines with LINQ to Twitter.