So, I'm trying to the friends of all of someone's followers. Not recursively digging, just that single level.
I'm able to pull the followers just fine (and it looks like it can handle more than 75k followers, so I'm 90% sure I have the rate limits fine there).
But my code to pull the friends is throwing authentication errors after it returns 15 of the followers friends.
In other words, in this bit of code:
for (var i = 0; i < followerList.Count; i++)
{
await followersFriends(followerList[i]);
}
I'm getting this error when i == 15:
LinqToTwitter.TwitterQueryException HResult=0x80131500 Message={"request":"/1.1/friends/ids.json","error":"Not authorized."} - Please visit the LINQ to Twitter FAQ (at the HelpLink) for help on resolving this error.
It does appear to completely get 15 lists, it's the 16th that throws it. I have the feeling that this is not a coincidence that that's the same as the rate limit, albeit an odd error if it is a rate limit.
I'll throw a check to see if we're on a multiple of 15 and do a 15 minute pause, but, on the chance that that doesn't work, does anyone have a theory as to what else could be going on here?
ETA: I did try redoing my authentication (Doing application based) on each run through followersFriends. Didn't help.
ETA2:
This is the new calling code:
for (var i = 0; i < followerList.Count; i++)
{
if (i % 15 == 0 && i != 0) {
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - i = " + i + " and user = " + followerList[i]);
PauseThatRefreshes("friends");
RunMe();
}
await followersFriends(followerList[i]);
}
And here's the code being called:
private async Task followersFriends(ulong uid)
{
var twitterCtx2 = new TwitterContext(m_foo);
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Start Friend Get for " + uid.ToString() + " ***");
long cursor = -1;
do
{
var friendList = await (from friend in twitterCtx2.Friendship where friend.Type == FriendshipType.FriendIDs && friend.UserID == uid.ToString() && friend.Cursor == cursor select friend).SingleOrDefaultAsync();
if (twitterCtx2.RateLimitRemaining <= 2)
{
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Pausing A ***");
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at Pause A: " +
twitterCtx2.RateLimitRemaining.ToString());
PauseThatRefreshes("friends");
}
if (friendList != null &&
friendList.IDInfo != null &&
friendList.IDInfo.IDs != null) {
cursor = friendList.CursorMovement.Next;
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining: " + twitterCtx2.RateLimitRemaining.ToString());
friendList.IDInfo.IDs.ForEach(id => Output(uid.ToString(), id.ToString()));
}
if (twitterCtx2.RateLimitRemaining <= 2)
{
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Pause B ***");
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at Pause B: " + twitterCtx2.RateLimitRemaining.ToString());
PauseThatRefreshes("friends");
}
} while (cursor != 0);
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Stop Friend Get for " + uid.ToString() + " ***");
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at stop: " + twitterCtx2.RateLimitRemaining.ToString());
return;
}
ETA just for clarification, the code that is calling the followersFriends. Yes, I know, I didn't change my button name =D This is more about collecting the data and I prefer a form as opposed to a command line... I've no idea why. Anyways, neither here nor there:
private async void button2_Click(object sender, EventArgs e)
{
var twitterCtx = new TwitterContext(m_foo);
string[] header = { "Account,Follows" };
File.WriteAllLines(@"C:\temp\followersFriends.txt", header);
//List to store ids of followers of main account
List<ulong> followerList = new List<ulong>();
// get main accounts followers and put into an array
long cursor = -1;
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Start Follower Get ***");
do
{
var followers =
await
(from follower in twitterCtx.Friendship
where follower.Type == FriendshipType.FollowerIDs &&
follower.ScreenName == "[[redacted]]" &&
follower.Cursor == cursor
select follower)
.SingleOrDefaultAsync();
if (followers != null &&
followers.IDInfo != null &&
followers.IDInfo.IDs != null
&&
followers.CursorMovement != null)
{
cursor = followers.CursorMovement.Next;
followers.IDInfo.IDs.ForEach(id =>
followerList.Add(id));
}
if (twitterCtx.RateLimitRemaining <= 2)
PauseThatRefreshes("followers");
} while (cursor != 0);
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Done Follower Get ***");
for (var i = 0; i < followerList.Count; i++)
{
if (i % 15 == 0 && i != 0)
{
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - i = " + i
+ " and user = " + followerList[i]);
PauseThatRefreshes("friends");
RunMe();
}
await followersFriends(followerList[i]);
}
}
As indicated in your notes, a TwitterQueryException with a "Not Authorized" message throws when encountering the 1963joeym account. Inspecting this account, I can see that they've protected it. The concept of "protected" means that no one can see a person's account or related activity unless that person "authorizes" your account. Since they haven't authorized the account that is attempting to query their friends list, you receive the HTTP 401 Unauthorized, from Twitter. Thus, the reason for the "Not Authorized" message.
Although User entities have a bool Protected property, it isn't practical to check that in your case because you're only working with a list of IDs and don't have the full User instance available. In this situation, the only thing you can do is to handle the exception and recover from there. There are two places to handle this exception: (1) When querying the original friends list and (2) When querying the friend's friends list, shown below:
var allFollowerIds = new List<ulong>();
Friendship followers = null;
long cursor = -1;
do
{
try
{
followers =
await
(from follower in twitterCtx.Friendship
where follower.Type == FriendshipType.FollowerIDs &&
follower.UserID == "1039951639" &&// "15411837" &&
follower.Cursor == cursor &&
follower.Count == 500
select follower)
.SingleOrDefaultAsync();
}
catch (TwitterQueryException tqExUnauthorized) when (tqExUnauthorized.StatusCode == HttpStatusCode.Unauthorized)
{
Console.WriteLine("This user hasn't given your account permission to view their account.");
}
catch (TwitterQueryException tqe)
{
Console.WriteLine(tqe.ToString());
break;
}
if (followers != null &&
followers.IDInfo != null &&
followers.IDInfo.IDs != null)
{
cursor = followers.CursorMovement.Next;
allFollowerIds.AddRange(followers.IDInfo.IDs);
}
} while (cursor != 0);
cursor = -1;
foreach (var uid in allFollowerIds)
{
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Start Friend Get for " + uid.ToString() + " ***");
do
{
Friendship friendList = null;
try
{
friendList = await (from friend in twitterCtx.Friendship where friend.Type == FriendshipType.FriendIDs && friend.UserID == uid.ToString() && friend.Cursor == cursor select friend).SingleOrDefaultAsync();
}
catch (TwitterQueryException tqExUnauthorized) when (tqExUnauthorized.StatusCode == HttpStatusCode.Unauthorized)
{
Console.WriteLine("This user hasn't given your account permission to view their account.");
}
if (twitterCtx.RateLimitRemaining <= 2)
{
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Pausing A ***");
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at Pause A: " +
twitterCtx.RateLimitRemaining.ToString());
//PauseThatRefreshes("friends");
}
if (friendList != null &&
friendList.IDInfo != null &&
friendList.IDInfo.IDs != null)
{
cursor = friendList.CursorMovement.Next;
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining: " + twitterCtx.RateLimitRemaining.ToString());
//friendList.IDInfo.IDs.ForEach(id => Output(uid.ToString(), id.ToString()));
}
if (twitterCtx.RateLimitRemaining <= 2)
{
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Pause B ***");
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at Pause B: " + twitterCtx.RateLimitRemaining.ToString());
//PauseThatRefreshes("friends");
}
} while (cursor != 0);
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - *** Stop Friend Get for " + uid.ToString() + " ***");
Console.WriteLine(DateTime.Now.ToString("h:mm:ss tt") + " - Friends Rate Limit Remaining at stop: " + twitterCtx.RateLimitRemaining.ToString());
}
Notice that I used a try/catch with a when clause, specifically for the 401 Unauthorized scenario, repeated below:
catch (TwitterQueryException tqExUnauthorized) when (tqExUnauthorized.StatusCode == HttpStatusCode.Unauthorized)
{
Console.WriteLine("This user hasn't given your account permission to view their account.");
}
For the friends list, this lets you handle just the Unauthorized portion and still protect for any other TwitterQueryException that doesn't fit that criteria. For the friend's friends list, you just need to handle only the Unauthorized exception and let the query continue to iterate on remaining friends.