Search code examples
scrapyuser-agent

Getting latest chrome user agent for Scrapy in python or other wise

Recently I have started to use Scrapy on a regular basis to analyze sites which demand the latest browser (user agent) for their content to show up. Now, this may seem like an old time problem, yet up-to-date the issue is quite open. Why?

There is no simple API or Package to generate/download the latest version user agents (in any OS/platform).

A number of packages try to resolve this:

  1. shadow-useragent - but it relies on voluntary server which is for some reason inactive as of now.
  2. latest-user-agents - but it also uses a hosted json file + it also lists old user agents, and it doesn't have any documentation.

Lastly there is this web: www.whatismybrowser.com which is very helpful but can't be easily automated...

Any clue how to resove that?

Solution

  • An old answer, but I was actually also looking for this feature.

    Your 2nd option latest-user-agents works as it gets the JSON file from a daily updated source from the same author.

    The reason the README.md contains old user-agents is that the repository is not updated, so this is just an example. I tried it today, and it has all the latest user-agents what I could see.