I am extremely novice when it comes to web scraping.
I am trying to do some exploratory analytics on some public base salary information where I work. They have a website, but it is awful trying to get any information out of it (almost like they did it on purpose...). Reading some intro to web scraping tutorials in R, I think this is an embedded table; the formatting looks like it is from Tableau.
After reading this post, I also think it is dynamic since the table isn't generated right away when the webpage is opened. When I inspect the webpage it has a form method="post" and that seems to make things harder from the little bit of reading I have done.
Due to this post, I think I need to use the package httr for a post request. But trying to read through this is making my head spin. https://f.briatte.org/r/scraping-form-results-with-httr
Now I am just realizing "All Campuses" the default is fine; I can filter later in R. So I just need to push the "Search" with httr.
Is there any way to scrape this information? I am most fluent in R if it is possible to do it there.
The website I am trying to scrape: https://www.cusys.edu/budget/cusalaries/
You can consider using RSelenium. Here is an example :
library(RSelenium)
url <- "https://www.cusys.edu/budget/cusalaries/"
shell('docker run -d -p 4445:4444 selenium/standalone-firefox')
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "firefox")
remDr$open()
remDr$navigate(url)
web_Obj <- remDr$findElement("xpath", '/html/body/table/tbody/tr/td/div[1]/div/form/table/tbody/tr/td/table/tbody/tr[2]/td/input') #search button
web_Obj$clickElement()
web_Obj_Table <- remDr$findElement("xpath", '/html/body/table/tbody/tr/td/div[1]/div/div/form/div/div/div/table[1]')
text_Table <- web_Obj_Table$getElementText()[[1]]
strsplit(text_Table, "\n")
[[1]]
[1] "DENVER & ANSCHUTZ ASA-Ctr Intrntl Bus/Ed/Rsch ACADEMIC & STUDENT AFFAIRS R21-122954 Director-Faculty Administrative Faculty 2.5 $60,000.00"
[2] "DENVER & ANSCHUTZ ASA-Ctr Intrntl Bus/Ed/Rsch ACADEMIC & STUDENT AFFAIRS R21-111214 Business Services Asst Dir Univ Staff - Business Services 100 $72,000.00"
[3] "DENVER & ANSCHUTZ ASA-Ctr Intrntl Bus/Ed/Rsch ACADEMIC & STUDENT AFFAIRS R21-114090 Business Services Senior Prof Univ Staff - Business Services 100 $54,000.00"
[4] "DENVER & ANSCHUTZ AVC OARS-NVTI ACADEMIC & STUDENT AFFAIRS R21-123858 Lecturer Other Faculty 50 $0.00"
[5] "DENVER & ANSCHUTZ AVC OARS-OO OMBUDSMAN OFFICE ACADEMIC & STUDENT AFFAIRS R21-121705 Academic Services Assoc Dir Univ Staff - Academic Services 100 $95,822.76"
[6] "DENVER & ANSCHUTZ AVC OARS-OO OMBUDSMAN OFFICE ACADEMIC & STUDENT AFFAIRS R21-109738 Academic Services Assoc Dir Univ Staff - Academic Services 100 $85,285.08"
[7] "DENVER & ANSCHUTZ AVC OARS-OO OMBUDSMAN OFFICE ACADEMIC & STUDENT AFFAIRS R21-125075 Academic Services Director Univ Staff - Academic Services 100 $108,289.68"
[8] "DENVER & ANSCHUTZ AVCAP-I/R & EFFECTIVENESS ACADEMIC & STUDENT AFFAIRS R21-100138 Business Services Assoc Dir Univ Staff - Business Services 100 $85,000.00"
[9] "DENVER & ANSCHUTZ AVCAP-I/R & EFFECTIVENESS ACADEMIC & STUDENT AFFAIRS R21-119011 Business Services Professional Univ Staff - Business Services 100 $67,000.00"
[10] "DENVER & ANSCHUTZ AVCAP-I/R & EFFECTIVENESS ACADEMIC & STUDENT AFFAIRS R21-103367 Business Services Professional Univ Staff - Business Services 100 $65,000.00"
[11] "DENVER & ANSCHUTZ AVCAP-I/R & EFFECTIVENESS ACADEMIC & STUDENT AFFAIRS R21-120088 Business Services Senior Prof Univ Staff - Business Services 100 $82,081.92"
[12] "DENVER & ANSCHUTZ AVCAP-I/R & EFFECTIVENESS ACADEMIC & STUDENT AFFAIRS R21-119368 Business Services Senior Prof Univ Staff - Business Services 100 $80,480.88"
[13] "DENVER & ANSCHUTZ AVCAP-I/R & EFFECTIVENESS ACADEMIC & STUDENT AFFAIRS R21-110281 Business Services Senior Prof Univ Staff - Business Services 100 $75,000.00"
[14] "DENVER & ANSCHUTZ AVCAP-I/R & EFFECTIVENESS ACADEMIC & STUDENT AFFAIRS R21-124082 Asst Vice Chancellor Univ Staff - Executive 100 $157,000.00"
[15] "DENVER & ANSCHUTZ AVCAP-I/R & EFFECTIVENESS ACADEMIC & STUDENT AFFAIRS R21-122661 Executive Director Univ Staff - Executive 100 $140,000.00"
[16] "DENVER & ANSCHUTZ AVCAP-I/R & EFFECTIVENESS ACADEMIC & STUDENT AFFAIRS R21-119303 IT Principal Prof Univ Staff - Information Tech. 50 $46,278.96"
[17] "DENVER & ANSCHUTZ AVCASA-CH Campus Housing ACADEMIC & STUDENT AFFAIRS R21-108478 Business Services Senior Prof Univ Staff - Business Services 100 $55,000.00"
[18] "DENVER & ANSCHUTZ AVCASA-CH Campus Housing ACADEMIC & STUDENT AFFAIRS R21-103171 Executive Director Univ Staff - Executive 100 $102,000.00"
[19] "DENVER & ANSCHUTZ AVCASA-CH Campus Housing ACADEMIC & STUDENT AFFAIRS R21-110700 Student Service Assoc Director Univ Staff - Student Services 100 $75,000.00"
[20] "DENVER & ANSCHUTZ AVCASA-CH Campus Housing ACADEMIC & STUDENT AFFAIRS R21-109706 Student Service Assoc Director Univ Staff - Student Services 100 $80,000.00"
[21] "DENVER & ANSCHUTZ AVCASA-CH Campus Housing ACADEMIC & STUDENT AFFAIRS R21-103169 Student Services Senior Prof Univ Staff - Student Services 100 $42,729.12"
[22] "DENVER & ANSCHUTZ AVCASA-CH Campus Housing ACADEMIC & STUDENT AFFAIRS R21-102105 Student Services Senior Prof Univ Staff - Student Services 100 $42,729.00"
[23] "DENVER & ANSCHUTZ AVCASA-CH Campus Housing ACADEMIC & STUDENT AFFAIRS R21-101263 Student Services Senior Prof Univ Staff - Student Services 100 $42,729.00"
[24] "DENVER & ANSCHUTZ AVCASA-CH Campus Housing ACADEMIC & STUDENT AFFAIRS R21-101116 Student Services Senior Prof Univ Staff - Student Services 100 $42,729.00"
[25] "DENVER & ANSCHUTZ AVCASA-CH Campus Housing ACADEMIC & STUDENT AFFAIRS R21-109348 Finance/Acctg Senior Prof Univ Staff- Finance & Acct. 100 $52,800.00"
[26] "DENVER & ANSCHUTZ AVCASA-DO Diversity Office ACADEMIC & STUDENT AFFAIRS R21-100015 Executive Assistant Univ Staff- Prof. Support Sr 100 $72,000.00"
[27] "DENVER & ANSCHUTZ AVCASA-DO Educatnl Opportunity ACADEMIC & STUDENT AFFAIRS R21-121508 Technician III Class Staff-Professional Srvce 100 $60,684.00"
[28] "DENVER & ANSCHUTZ AVCASA-DO Educatnl Opportunity ACADEMIC & STUDENT AFFAIRS R21-122605 Student Services Prgm Director Univ Staff - Student Services 100 $64,503.96"
[29] "DENVER & ANSCHUTZ AVCASA-DO Educatnl Opportunity ACADEMIC & STUDENT AFFAIRS R21-116274 Student Services Prgm Director Univ Staff - Student Services 100 $65,585.04"
[30] "DENVER & ANSCHUTZ AVCASA-DOS Dean of Students ACADEMIC & STUDENT AFFAIRS R21-108225 Business Services Senior Prof Univ Staff - Business Services 100 $48,136.68"
[31] "DENVER & ANSCHUTZ AVCASA-DOS Dean of Students ACADEMIC & STUDENT AFFAIRS R21-112560 Student Service Assoc Director Univ Staff - Student Services 100 $75,000.00"
[32] "DENVER & ANSCHUTZ AVCASA-DOS Dean of Students ACADEMIC & STUDENT AFFAIRS R21-114054 Student Services Director Univ Staff - Student Services 100 $90,385.00"
[33] "DENVER & ANSCHUTZ AVCASA-DOS Dean of Students ACADEMIC & STUDENT AFFAIRS R21-112128 Student Services Prgm Director Univ Staff - Student Services 100 $70,009.92"
[34] "DENVER & ANSCHUTZ AVCASA-DOS Dean of Students ACADEMIC & STUDENT AFFAIRS R21-106671 Student Services Prgm Manager Univ Staff - Student Services 100 $57,500.04"
[35] "DENVER & ANSCHUTZ AVCASA-DOS Dean of Students ACADEMIC & STUDENT AFFAIRS R21-115935 Student Services Professional Univ Staff - Student Services 100 $50,000.00"
[36] "DENVER & ANSCHUTZ AVCASA-DOS Dean of Students ACADEMIC & STUDENT AFFAIRS R21-102357 Student Services Professional Univ Staff - Student Services 100 $50,000.00"
[37] "DENVER & ANSCHUTZ AVCASA-DOS Dean of Students ACADEMIC & STUDENT AFFAIRS R21-100443 Student Services Professional Univ Staff - Student Services 100 $50,000.00"
[38] "DENVER & ANSCHUTZ AVCASA-DOS PhoenixCtr at Aurar ACADEMIC & STUDENT AFFAIRS R21-109956 Student Services Prgm Director Univ Staff - Student Services 100 $70,000.00"
[39] "DENVER & ANSCHUTZ AVCASA-DOS PhoenixCtr at Aurar ACADEMIC & STUDENT AFFAIRS R21-108692 Student Services Prgm Manager Univ Staff - Student Services 100 $54,550.00"
[40] "DENVER & ANSCHUTZ AVCASA-DOS PhoenixCtr at Aurar ACADEMIC & STUDENT AFFAIRS R21-105423 Student Services Professional Univ Staff - Student Services 100 $50,000.00"
[41] "DENVER & ANSCHUTZ AVCASA-DOS PhoenixCtr at Aurar ACADEMIC & STUDENT AFFAIRS R21-104981 Student Services Professional Univ Staff - Student Services 100 $50,000.00"
[42] "DENVER & ANSCHUTZ AVCASA-DOS Stdnt&Comm Counseln ACADEMIC & STUDENT AFFAIRS R21-122385 Academic Services Professional Univ Staff - Academic Services 100 $60,534.00"
[43] "DENVER & ANSCHUTZ AVCASA-DOS Stdnt&Comm Counseln ACADEMIC & STUDENT AFFAIRS R21-100159 Student Services Asst Director Univ Staff - Student Services 100 $69,178.92"
[44] "DENVER & ANSCHUTZ AVCASA-DOS Stdnt&Comm Counseln ACADEMIC & STUDENT AFFAIRS R21-117580 Student Services Director Univ Staff - Student Services 100 $95,693.04"
[45] "DENVER & ANSCHUTZ AVCASA-DOS Stdnt&Comm Counseln ACADEMIC & STUDENT AFFAIRS R21-111326 Student Services Professional Univ Staff - Student Services 100 $60,088.08"
[46] "DENVER & ANSCHUTZ AVCASA-DOS Stdnt&Comm Counseln ACADEMIC & STUDENT AFFAIRS R21-102554 Student Services Professional Univ Staff - Student Services 100 $56,000.00"
[47] "DENVER & ANSCHUTZ AVCASA-DOS Stdnt&Comm Counseln ACADEMIC & STUDENT AFFAIRS R21-102485 Student Services Professional Univ Staff - Student Services 100 $56,000.00"
[48] "DENVER & ANSCHUTZ AVCASA-DOS Stdnt&Comm Counseln ACADEMIC & STUDENT AFFAIRS R21-122396 Student Services Senior Prof Univ Staff - Student Services 50 $31,239.00"
[49] "DENVER & ANSCHUTZ AVCASA-DOS Stdnt&Comm Counseln ACADEMIC & STUDENT AFFAIRS R21-115308 Student Services Senior Prof Univ Staff - Student Services 100 $63,500.00"
[50] "DENVER & ANSCHUTZ AVCASA-EM Admissions ACADEMIC & STUDENT AFFAIRS R21-116557 Admin Assistant II Class Staff-Admin Support 100 $38,484.00"