Search code examples
phpweb-scrapingautocomplete

scraping data from search query with PHP


I'm using PHP, also a complete noob at it.

So I have this URL that takes queries and returns a data file with some matches.

Say www.example.com/search?q=$query where $query is whatever the search term is.

However when you go to the URL in your browser it downloads a text file with the information in it.

I want to run through the whole alphabet and every combination of letters up to 10 letters and extract the data from all the returned files and store it in a database.

I'm not exactly sure how to go about opening each URL from the PHP script. Is there a better way to do this than downloading each file and extracting the info then deleting the file?

What I have below doesn't work at all.

$alphabet = "abcdefghijklmnopqrstuvwxyz";
for ($i=0; $i<=25; $i++){
    $query = $alphabet[$i];
    $url = "www.example.com/search?q=$query";
    $html = fopen($url);
    $stringify = (string)$html;
    echo $stringify;
}

Solution

  • You can't just open a file from an URL. You need to use an HTTP client library in order to get the files. This link may come in handy: HTTP Client Library for PHP


    EDIT: based on alex's comment, you may want to check allow_url_fopen in your php.ini file.