Fetching all tweets using twitter4j around a given location

Question

I want to fetch all the tweets from twitter API using Twitter4j. Here is my code to fetch tweets using geoCode but I am unable to fetch all tweets, it is just showing me around 100 tweets. Is there any way I can fetch them all. I think there is a rate limit on the amount of tweets i can receive but i am not sure. Could anyone help ?

import java.util.List;

import twitter4j.*;
import twitter4j.auth.AccessToken;

public class Main {
public static void main(String[] args) {

    final Twitter twitter = new TwitterFactory().getInstance();
    final  AccessToken accessToken = new AccessToken("XXX", "YYY");
    twitter.setOAuthConsumer("AAA", "BBB");
    twitter.setOAuthAccessToken(accessToken);

    try {
        Query query = new Query();
        query.geoCode(new GeoLocation(37.781157,-122.398720),1000.0,"mi");
        QueryResult result;
        System.out.println("Searching...");
        int Count=0;

        do {
            result = twitter.search(query);
            List<Status> tweets = result.getTweets();
            for (Status tweet : tweets) {
                if(tweet.getGeoLocation()!=null)
                    System.out.println(tweet.getGeoLocation());
            }
            try {
                Thread.sleep(500);
            } catch (InterruptedException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
        while ((query = result.nextQuery()) != null);
        System.out.println(Count);
        System.exit(0);
    } catch (TwitterException te) {
        te.printStackTrace();
        System.out.println("Failed to search tweets: " + te.getMessage());
        System.exit(-1);
    }
}

}

Answer 1

You have some problems with how are you approaching your solution.

Your code is fine
You are asking Twitter to get some tweets in the last few days in the given location, and that is what Twitter is giving to you (at the current moment around 400 tweets), so... Why are there so few tweets? Why other sites have so many tweets? The main problem is the search api.

The Search Api
You need to know that the search API is focused on relevance and not completeness, so you only get some tweets from the last days (about a week), and as the documentation states:

If you want to match for completeness you should consider using a Streaming API instead

That lead us to the next step...

The Streaming Api
In the streaming api you can't search for tweets, you get them in real time. If you want a month of localized tweets you will need to set the bound on the streaming api and let it run for a month. On Twitter4j you need to set the localization filter and then run the streaming, something like this:

    ConfigurationBuilder cb = new ConfigurationBuilder();
    cb.setDebugEnabled(true).setOAuthConsumerKey("---")
            .setOAuthConsumerSecret("---")
            .setOAuthAccessToken("---")
            .setOAuthAccessTokenSecret("---");

    TwitterStream twitterStream = new TwitterStreamFactory(cb.build())
            .getInstance();
    StatusListener listener = new StatusListener() {

        @Override
        public void onStatus(Status status) {
                //here you do whatever you want with the tweet
            System.out.println(status.getText());

        }

        @Override
        public void onException(Exception ex) {
            ex.printStackTrace();
        }

        @Override
        public void onDeletionNotice(StatusDeletionNotice arg0) {
                  // TODO Auto-generated method stub

        }

        @Override
        public void onScrubGeo(long arg0, long arg1) {

        }

        @Override
        public void onStallWarning(StallWarning arg0) {
            // TODO Auto-generated method stub
            System.out.println(arg0);
        }

        @Override
        public void onTrackLimitationNotice(int arg0) {
            // TODO Auto-generated method stub
            System.out.println(arg0);
        }

    };

    twitterStream.addListener(listener);
    FilterQuery filterQuery = new FilterQuery();
    double[][] locations = {{-74,40}, {-73,41}}; //those are the boundary from New York City
    filterQuery.locations(locations);
    twitterStream.filter(filterQuery);
    twitterStream.filter(filterQuery);

If you are getting too many tweets the Streaming Api will cut them, so try to not set the bounds box to big.
But... if I want tweets from the past and I want them right now?
This is where Twitter get some extra money and you could get them with Gnip