My question is about the possibility of scraping data from particular websites. At the moment, my algorithm is converting HTML to text, then checking for flagged words that are contained in a file, and summing up the number of flags.
My problem lies in the inability to "scroll" down while scraping the website. As you can see, it's checking the number of flags on a twitter account, but it is limited to 50sh latest tweets. I hope I made myself clear.
ps: I gave twitter as an example, and I'm not looking for something specific for twitter, but something more robust.
I will greatly appreciate any hints.
Sample output:
DHS & Other Agencies: 0 instances
Domestic security: 1 instances
HAZMAT & Nuclear: 0 instances
Health Concern + H1N1: 0 instances
Infrastructure Security: 2 instances
Southwest Border Violence: 1 instances
Terrorism: 0 instances
Weather/Disaster/Emergency: 2 instances
Cyber Security: 0 instances
TOTAL FLAGS: 6 instances
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.io.*;
import java.util.LinkedList;
import java.util.List;
public class Main {
static List<String> dhsAndOtherAgencies = new LinkedList<>();
static List<String> domesticSecurity = new LinkedList<>();
static List<String> hazmatNuclear = new LinkedList<>();
static List<String> healthConcern = new LinkedList<>();
static List<String> infrastructureSecurity = new LinkedList<>();
static List<String> southwestBorderViolence = new LinkedList<>();
static List<String> terrorism = new LinkedList<>();
static List<String> weatherDisasterEmergency = new LinkedList<>();
static List<String> cyberSecutiry = new LinkedList<>();
static String stream;
public static void main(String[] args) throws IOException {
createLists();
createStream();
Raport raport = generateReport();
System.out.println(raport);
}
static int flagStream(List<String> list, String stream){
int counter = 0;
for (String flag: list){
if(stream.contains(flag)) {
System.out.println(flag);
counter++;
}
}
return counter;
}
static Raport generateReport(){
return new Raport(
flagStream(dhsAndOtherAgencies,stream),
flagStream(domesticSecurity,stream),
flagStream(hazmatNuclear,stream),
flagStream(healthConcern,stream),
flagStream(infrastructureSecurity,stream),
flagStream(southwestBorderViolence,stream),
flagStream(terrorism,stream),
flagStream(weatherDisasterEmergency,stream),
flagStream(cyberSecutiry,stream)
);
}
static void createStream() throws IOException {
Document doc = Jsoup.connect("https://twitter.com/realDonaldTrump").userAgent("mozilla/17.0").get();
stream = doc.text();
}
static void createLists() throws IOException {
BufferedReader read = new BufferedReader(new FileReader("clearListAllCases.txt"));
String input;
int hashCounter = 0;
while((input=read.readLine())!=null){
if(input.charAt(0)=='#'){
hashCounter++;
continue;
}
switch (hashCounter){
case 1:
dhsAndOtherAgencies.add(input);
break;
case 2:
domesticSecurity.add(input);
break;
case 3:
hazmatNuclear.add(input);
break;
case 4:
healthConcern.add(input);
break;
case 5:
infrastructureSecurity.add(input);
break;
case 6:
southwestBorderViolence.add(input);
break;
case 7:
terrorism.add(input);
break;
case 8:
weatherDisasterEmergency.add(input);
break;
case 9:
cyberSecutiry.add(input);
break;
}
}
}
}
class Raport {
int a,b,c,d,e,f,g,h,i;
int totalFlags;
Raport(int a, int b, int c, int d, int e, int f, int g, int h, int i){
this.a = a;
this.b = b;
this.c = c;
this.d = d;
this.e = e;
this.f = f;
this.g = g;
this.h = h;
this.i = i;
totalFlags = a+b+c+d+e+f+g+h+i;
}
public String toString(){
return "DHS & Other Agencies:\t\t\t"+a+" instances\n"+
"Domestic security:\t\t\t\t"+b+" instances\n"+
"HAZMAT & Nuclear:\t\t\t\t"+c+" instances\n"+
"Health Concern + H1N1:\t\t\t"+d+" instances\n"+
"Infrastructure Security:\t\t"+e+" instances\n"+
"Southwest Border Violence:\t\t"+f+" instances\n"+
"Terrorism:\t\t\t\t\t\t"+g+" instances\n"+
"Weather/Disaster/Emergency:\t\t"+h+" instances\n"+
"Cyber Security:\t\t\t\t\t"+i+" instances\n"+
"TOTAL FLAGS:\t\t\t\t\t"+totalFlags+" instances";
}
}
I would suggest opening the browser's developer tab for trying to find out which url/endpoint the website uses for fetching new items for the infinite scroll, as JSoup does not execute Javascript itself. Then you can call the endpoint with JSoup and parse the results.
In case it does not work, It would be probably better to move to HtmlUnit or Selenium as both of them are full-featured web browser APIs which you can control with Java.