Search code examples
htmlhttprusthyper

Download and get HTML code from a website

I need to download HTML code from some web page. What is the best way to approach this task? As I understand there are very few working web frameworks for Rust right now and hyper is the one most people use? But after searching it's documentation I couldn't find a way. The closest I got is this

extern crate hyper;

use hyper::Client;

fn main() {
    let client = Client::new();
    let res = client.get("http://www.bloomberg.com/")
        .send()
        .unwrap();

    println!("{:?}", res);
}

But it returns Response, which doesn't seem to contain any code from HTML body.

Solution

  • Note: this answer is outdated!

    I don't have the time to update this with every hyper release. But please see my answer to a very related question: How can I download a website's content into a string?

    It's a bit hidden: The Response type implements the trait Read. One method of Read is read_to_string which reads everything into the a String. That's a simple way you can get the body.

    extern crate hyper;

use hyper::Client;
use std::io::Read;

fn main() {
    let client = Client::new();
    let mut res = client.get("http://www.bloomberg.com/")
        .send()
        .unwrap();
    let mut body = String::new();
    res.read_to_string(&mut body).expect("failed to read into string");
    println!("{}", body);
}

    Currently Rustdoc (the HTML documentation of Rust) is a little bit misleading because Rust beginners think that trait implementations don't add any important functionality. This is not true, so better look out for it. However, the hyper documentation could be better...