Search code examples
f#type-providersf#-datafsharp.data.typeproviders

How to reuse the web client for HtmlProvider?


I load multiple pages using the same type provider, HtmlProvider:

type Article = HtmlProvider<"https://en.wiktionary.org/wiki/giraffe">

let sheepArticle = Article.Load "https://en.wiktionary.org/wiki/sheep"
let koalaArticle = Article.Load "https://en.wiktionary.org/wiki/koala"
let pandaArticle = Article.Load "https://en.wiktionary.org/wiki/panda"
...

Is it anyhow possible to configure the provider so that the web client underneath is reused there?

I haven't found nothing on that in the docs. I do a lot of similar calls, so that would be a significant optimization.


Solution

  • There is no way to configure the HtmlProvider to do this automatically behind the scenes, but you can easily create your own WebClient to download the pages and then use the Parse method of the provided type (rather than using Load to download and parse it):

    type Article = HtmlProvider<"https://en.wiktionary.org/wiki/giraffe">
    
    let wc = new WebClient()
    let sheepArticle = Article.Parse(wc.DownloadString("https://en.wiktionary.org/wiki/sheep"))
    let koalaArticle = Article.Parse(wc.DownloadString("https://en.wiktionary.org/wiki/koala"))
    let pandaArticle = Article.Parse(wc.DownloadString("https://en.wiktionary.org/wiki/panda"))