how to simply open an url and read the data from a webpage with D? (I prefer phobos over tango, if needing to use standard lib functionality)
curl is in the standard library. You can fetch a url pretty easily like this:
import std.net.curl;
string content = get("d-lang.appspot.com/testUrl2");
http://dlang.org/phobos/std_net_curl.html#get
If you need to parse html, I wrote a dom library that is pretty good at it. https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff
grab dom.d and characterencodings.d then you can:
import arsd.dom;
auto document = new Document();
document.parseGarbage(content); // content is from above, the html string
writeln(document.title); // the <title> contents
auto paragraph = document.querySelector("p");
if(paragraph is null)
writeln("no paragraphs in this document");
else
writeln("the first paragraph is: ", paragraph.innerText);
and so on. If you've used javascript dom api, this is pretty similar (though expanded in a lot of ways too).