I am trying to pull a csv table from a webpage into a pandas dataframe within a Jupyter Notebook, a process I have successfully implemented several times for other websites in the past, but I am running into a challenge with this one.
The table I am trying to pull is only available as a JavaScript response to a button click instead of being available at a specific URL.
Here is the URL: https://ciffc.net/national // The table I am trying to scrape is contained within the Export (CSV) button in red.
I have tried implementing a solution using pandas.read_csv, but I can't seem to point to the button in question.
Of note, I am using Jupyter Notebooks within the ArcGIS Online environment.
Thank you!
The json for example for page 1 seems to be available over the stable URL https://api.ciffc.net//v1/wildfires?page=1.
So if you don't mind doing some extra formatting on the DataFrame you can use:
import pandas
import requests
df = pandas.DataFrame.from_dict(
requests.get('https://api.ciffc.net//v1/wildfires?page=1').json()["rows"])
Which contains:
| field_agency_fire_id | field_agency_code | field_agency_data_timezone | field_situation_report_date | field_status_date | field_stage_of_control_status | field_system_fire_cause | field_response_type | field_fire_size | field_latitude | field_longitude | full_count | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | SS007-24 | nt | MDT | 2024-05-12T22:21:00.000Z | 2024-06-14T20:00:10.000Z | OC | N | MON | 1 | 60.5853 | -116.902 | 1515 |
| 1 | FS005-24 | nt | MDT | 2024-05-13T17:40:00.000Z | 2024-06-14T20:00:10.000Z | UC | N | FUL | 22 | 61.52 | -121.393 | 1515 |
| 2 | SS015-24 | nt | MDT | 2024-06-01T21:57:00.000Z | 2024-06-14T20:00:10.000Z | OC | N | FUL | 0.2 | 60.0809 | -117.185 | 1515 |
| 3 | SS008-24 | nt | MDT | 2024-05-23T23:24:00.000Z | 2024-06-14T20:00:10.000Z | OC | N | MON | 8 | 60.2382 | -118.13 | 1515 |
| 4 | SS003-24 | nt | MDT | 2024-05-05T19:45:00.000Z | 2024-06-14T20:00:10.000Z | OUT | N | FUL | 0.01 | 60.6161 | -116.073 | 1515 |
| 5 | FS006-24 | nt | MDT | 2024-05-13T21:47:00.000Z | 2024-06-14T20:00:10.000Z | OC | N | MON | 2930 | 61.2465 | -119.392 | 1515 |
| 6 | ZF004-24 | nt | MDT | 2024-05-25T17:55:00.000Z | 2024-06-14T20:00:10.000Z | OUT | H | FUL | 0 | 62.4502 | -114.401 | 1515 |
| 7 | ZF001-24 | nt | MDT | 2024-02-05T15:00:00.000Z | 2024-06-14T20:00:10.000Z | OUT | H | FUL | 0.05 | 62.3217 | -114.252 | 1515 |
| 8 | SS012-24 | nt | MDT | 2024-05-25T18:00:00.000Z | 2024-06-14T20:00:10.000Z | OUT | H | FUL | 0.1 | 61.1722 | -113.669 | 1515 |
| 9 | SS009-24 | nt | MDT | 2024-05-24T16:08:00.000Z | 2024-06-14T20:00:10.000Z | OC | N | MOD | 5681 | 60 | -117 | 1515 |
| 10 | SS005-24 | nt | MDT | 2024-05-10T20:10:00.000Z | 2024-06-14T20:00:10.000Z | OUT | H | FUL | 0.5 | 60.5885 | -116.176 | 1515 |
| 11 | SS004-24 | nt | MDT | 2024-05-05T20:45:00.000Z | 2024-06-14T20:00:10.000Z | BH | N | FUL | 3 | 60.3924 | -117.896 | 1515 |
| 12 | SS002-24 | nt | MDT | 2024-04-23T06:00:00.000Z | 2024-06-14T20:00:10.000Z | OUT | N | FUL | 0.5 | 60.3258 | -112.282 | 1515 |
| 13 | SS001-24 | nt | MDT | 2024-04-23T06:00:00.000Z | 2024-06-14T20:00:10.000Z | OUT | N | FUL | 2 | 60.119 | -112.281 | 1515 |
| 14 | ZF007-24 | nt | MDT | 2024-06-06T22:37:00.000Z | 2024-06-14T20:00:10.000Z | OUT | H | FUL | 0 | 62.5128 | -113.694 | 1515 |
| 15 | ZF005-24 | nt | MDT | 2024-05-26T13:00:00.000Z | 2024-06-14T20:00:10.000Z | OUT | H | FUL | 0 | 62.5185 | -114.319 | 1515 |
| 16 | ZF003-24 | nt | MDT | 2024-05-23T23:30:00.000Z | 2024-06-14T20:00:10.000Z | OUT | H | FUL | 5 | 63.4575 | -116.098 | 1515 |
| 17 | ZF002-24 | nt | MDT | 2024-05-19T23:45:00.000Z | 2024-06-14T20:00:10.000Z | OUT | H | FUL | 0.01 | 62.4923 | -113.521 | 1515 |
| 18 | SS014-24 | nt | MDT | 2024-05-29T23:55:00.000Z | 2024-06-14T20:00:10.000Z | OUT | N | FUL | 2.5 | 60.2638 | -116.334 | 1515 |
| 19 | SS013-24 | nt | MDT | 2024-05-27T20:18:00.000Z | 2024-06-14T20:00:10.000Z | OUT | N | FUL | 0.5 | 60.3 | -116.65 | 1515 |
| 20 | SS011-24 | nt | MDT | 2024-05-24T18:40:00.000Z | 2024-06-14T20:00:10.000Z | OUT | N | FUL | 1 | 60.9028 | -112.421 | 1515 |
| 21 | SS010-24 | nt | MDT | 2024-05-24T18:22:00.000Z | 2024-06-14T20:00:10.000Z | OC | N | MOD | 28 | 60.9536 | -112.39 | 1515 |
| 22 | ZF006-24 | nt | MDT | 2024-05-31T21:49:00.000Z | 2024-06-14T20:00:10.000Z | OC | N | MON | 145 | 62.5954 | -119.058 | 1515 |
| 23 | SS006-24 | nt | MDT | 2024-05-10T21:49:00.000Z | 2024-06-14T20:00:10.000Z | OUT | N | MON | 1 | 60.6623 | -115.978 | 1515 |
| 24 | FS004-24 | nt | MDT | 2024-05-10T18:20:00.000Z | 2024-06-14T20:00:10.000Z | OC | N | MOD | 11894 | 61.1473 | -119.376 | 1515 |