I am a frontend developer building a metadata scraper and hosting it on Cloudflare Worker to bypass CORS for my application. My main app sends a get request with the URL attached to the worker server, which subsequently uses fetch provided by Cloudflare API to get HTML data similar to the given worker example.
The issue I am facing is the HTML response for some sites (like youtube) is changing (particularly the language) based on the location from where I am sending the initial request from. I am getting the response in English while a coworker is getting the response in German, even when we are in the same country although different cities. Strangely, when he changes his IP and switches to a different network, he is able to get the correct response in English.
I suspect what is happening is that Workers is changing the location (from among its various locations specified here) from where the second request to fetch HTML data is coming from based on the initial request IP address location.
Is there any way I can manually specify which Cloudflare host to send the fetch HTML request from to bypass this issue? Or any other alternative solution would be welcome. I would be grateful.
You can force YouTube to display in English by adding the following to the end of the URL: ?hl=en&persist_hl=1 (You can set a different language by replacing en with the desired language code.) There are probably other ways to change languages by manipulating YouTube settings.
Otherwise, YouTube tries to guess your preferred language based on the geo-location of the IP address making the request. In the case of a request through Workers, that IP address would actually be one belonging to Cloudflare.
Cloudflare Workers usually runs your Worker code in the closest Cloudflare location to the end user's browser. If you and your coworker are in different cities, it is very likely that your requests will go to different Cloudflare locations, since Cloudflare has locations in hundreds of cities around the world. As a result, the Worker will execute in different places. There is currently no way to tell Cloudflare to run your Worker in a particular place. It always runs in the Cloudflare colo which received the request.
For some reason, YouTube is deciding that one Cloudflare location prefers English while the other prefers German. It's possible YouTube's mapping of IP addresses to geographical locations is not correct for these Cloudflare IPs. When your coworker "changes his IP", I assume you mean he is using a VPN. The VPN is probably taking his traffic to a different city and therefore he ends up hitting a different Cloudflare location, which YouTube apparently decides wants English.