Search code examples
pythonhtmlweb-scraping

Obtaining a list of donations from a GoFundMe campaign

It's a long story as to why I need this information, but I'm looking to obtain a list of the donations on a GoFundMe campaign. I would have thought this would have been possible through an API, but much to my distress GoFundMe doesn't seem to have an API. This led me down a rabbit hole of web scraping the campaign page.

This gave me some of the information I needed (The total amount of money raised and the goal/target), but for some reason whenever I go to scrape the section at the bottom of the website that contains the list of donations and their comments, or the /donations page attached to every campaign, somewhere along the way an element comes out as blank. For the sake of example, with the /donations page, this happens as soon as I scrape down to <div id="portal">.

This is my scraping code:

headers = {'Accept-Language': 'en-UK,en;q=0.5'}
page = get(URL + "/donations", headers=headers)
soup = bs(page.content, "html.parser")

results = soup.find(id="__next")
results = results.find("div", class_="page-modal-layout_pageModalLayout__O_yxc")
results = soup.find(id="portal")

But the results variable after this step is <div id="portal"></div>, whereas I was expecting it to contain a whole multitude of different classes, as it does here in the inspect pane on Google Chrome:

Screen Capture of Compressed Google Chrome Inspect

I've uploaded a screen capture here because it's easier to see the tree of "div"s, and I don't fully understand what I'm looking at as I'm not the best at HTML.

My first assumption as to why I can't access this data would be because it's being accessed from some sort of external database, and that database is locked down and requires some sort of authentication that I clearly don't have to access the data. The problem I have with that initial assumption is that I've found something that does almost what I want it to do: On this blog page they mention an stream overlay called "GoFundMe Alerts by LX", which does indeed function, and receives alerts every time a donation goes through, but I cannot for the life of me figure out how they did it given GoFundMe's lack of API.

Digging around in the inspect view for one of these overlays showed me that they were receiving a lot of data about those donations, one of them looks like this:

{donation_id: 1102799627, amount: 5, is_offline: false, is_anonymous: true, ...}

This implied to me that there was some sort of API at play, but unfortunately that was where my search was terminated because I couldn't find any further leads. I may be going down completely the wrong path to try to find how to get this information, and basically what I'm asking is: How do I get GoFundMe donations sent to me live? Is it even possible?

Solution

  • There's comments API you can use, for example:

    import requests


comments_url = "https://gateway.gofundme.com/web-gateway/v1/feed/24hr-stream-2024-reformation-fundraising/comments"

params = {"limit": 10, "offset": 0}
data = requests.get(comments_url, params=params).json()

for c in data["references"]["contents"]:
    donation = c.get("donation", {})
    print(c["name"], donation.get("amount", 0))

    Prints:

    Andrew Knight A B KNIGHT 5.0

    You can check this URL in periodic intervals and send an alert on every change.