System.Net.Http.HttpClient returns error "ERR: Missing UA30"

Question

The following code gives me the rather strange error ERR: Missing UA30 in the response content. I can't find the error anywhere on the Internet. It only happens to some particular websites. There is another version of the code testing serveral ics URLs simultaneously on this repo. Since F# and C# share the same framework, I suspect this would happen to C# as well but I haven't tested it as my C# is not as fluent.

open System.Net.Http

let url = "https://portal.macleans.school.nz/index.php/ics/school.ics"
(* other working urls:
    "https://gist.githubusercontent.com/DeMarko/6142417/raw/1cd301a5917141524b712f92c2e955e86a1add19/sample.ics"
    "https://www.kayaposoft.com/enrico/ics/v2.0?country=gbr&fromDate=01-01-2023&toDate=31-12-2023&region=eng"
    "https://www.phpclasses.org/browse/download/1/file/63438/name/example.ics"
    *)


let getAsync (client: HttpClient) (url: string) =
    async {
        let! content = client.GetStringAsync url |> Async.AwaitTask
        let len = content.Length
        printf "%d" len

        match len with
        | len when len < 50 -> printfn " content: %s" content
        | _ -> printfn ""
    }

let client = new HttpClient()

getAsync client url
|> Async.RunSynchronously

All what I've tried have been documented in the GitHub repo mentioned above.

I am expecting no error as curl command is able to download the ics file without problems -

curl https://portal.macleans.school.nz/index.php/ics/school.ics -o output.ics

Answer 1

You can try to set user agent setting header in your HTTP request. for example :

open System.Net.Http

let url = "https://portal.macleans.school.nz/index.php/ics/school.ics"

let getAsync (client: HttpClient) (url: string) =
    async {
        let request = new HttpRequestMessage(HttpMethod.Get, url)
        request.Headers.Add("User-Agent", "NonEmptyUA")

        let! response = client.SendAsync(request) |> Async.AwaitTask
        let content = response.Content.ReadAsStringAsync() |> Async.AwaitTask
        printfn "Response content length: %d" (content.Length)

        if content.Length < 50 then
            printfn "Response content: %s" content
    }

let client = new HttpClient()

getAsync client url
|> Async.RunSynchronously