Powershell: The invoke-webrequest is not working for me for a particular site. Tried many things like setting TLS settings etc. But still fails. One unique thing is it works for first request, post then it fails for 15mins but such is not the case when browsing the site via browser. Can anyone help me please
Invoke-WebRequest 'https://www1.nseindia.com/products/content/equities/equities/eq_turnapr2020.htm'
For the next request it waits forever then goes timeout. But in browser that is not the case(in IE, chrome)
It was because of website's anti-scrape mechanism. To Bypass that we need to make it feel like a user request via browser. For this current problem with this website., solution was to include headers in the Invoke-WebRequest. You can take copy of request headers via Chrome Developer Tools > Network tab > (right click url) > Copy > copy as powershell. Then paste that & take the hashtable of headers.(you can then remove cookies section).
