Since few days I'm not able to consume any information from Youtube with my Powershell Script, at first the cookies must be accepted in order to see the Videos. Any ideas how to accept Youtube Cookies with Powershell? There is already a question, but it does not concern Powershell Accept cookies consent from Youtube
$HTML=Invoke-WebRequest -Uri https://www.youtube.com/user/YouTube/videos
After some testing I got it work.
But keep in mind that the current process could fail in future if Google change the process or the form data of the response.
After our first call to youtube, we get redirected to consent.google.com and receive the following form content:
<form action="https://consent.youtube.com/s" method="POST" style="display:inline;">
<input type="hidden" name="gl" value="DE">
<input type="hidden" name="m" value="0">
<input type="hidden" name="pc" value="yt">
<input type="hidden" name="continue" value="https://www.youtube.com/user/YouTube/videos">
<input type="hidden" name="ca" value="r">
<input type="hidden" name="x" value="8">
<input type="hidden" name="v" value="cb.20210329-17-p2.de+FX+873">
<input type="hidden" name="t" value="ADw3F8i44JCpypLjx8SOx3tbsrxxS7ug:1617806186191">
<input type="hidden" name="hl" value="de">
<input type="hidden" name="src" value="1">
<input type="hidden" name="uxe" value="2398321372">
<input type="submit" value="Ich stimme zu" class="button" aria-label="In die Verwendung von Cookies und anderen Daten zu den beschriebenen Zwecken einwilligen" />
</form>
I assume that only the two properties continue and v are important, but I just use all properties in my POST process like my Browser did.
v will be the right portion of our final CONSENT cookie value. It will have a YES+ prefix.
e.g cb.20210329-17-p2.de+FX+873 from v becomes YES+cb.20210329-17-p2.de+FX+873 in the cookie CONSENT
Unfortunately, our call with Invoke-WebRequest does not provide us any predefined form property. The property (Invoke-WebRequest abc).Form is just NULL.
Therefore we have to parse the particular form data from the response content, build a key=value body and POST the body to the URL mentioned in the action attribute.
Please find the rest of the process as comments in the code.
This is the clean code without verbose output. Find the same code with verbose output below.
$youtubeUrl = 'https://www.youtube.com/user/YouTube/videos'
$consentDomain = 'consent.youtube.com'
$webUserAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.4976.0 Safari/537.36 Edg/102.0.1227.0'
# at first we disable the annoying (at least for this process) and in PS5.1- performance affecting progress bar for web requests
$currentProgressPreference = $ProgressPreference
$ProgressPreference = [System.Management.Automation.ActionPreference]::SilentlyContinue
try {
# in our first GET call we should get a response from consent.youtube.com.
# we save the session including all cookies in variable $youtubeSession.
$response = Invoke-WebRequest -Uri $youtubeUrl -UseBasicParsing -SessionVariable 'youtubeSession' -UserAgent $webUserAgent -ErrorAction Stop
# using BaseResponse to figure out which host has responded
if ($PSVersionTable.PSVersion.Major -gt 5) {
# PS 6+ has other properties than PS5.1 and below
$responseRequestUri = $response.BaseResponse.RequestMessage.RequestUri
} else {
$responseRequestUri = $response.BaseResponse.ResponseUri
}
if ($responseRequestUri.Host -eq $consentDomain) {
# check if got redirected to "consent.youtube.com"
# unfortunately the response object from "Invoke-WebRequest" does not provide any "Form" data as property,
# so we have to parse it from the content. There are two <form..> nodes, but we only need the one for method "POST".
$formContent = [regex]::Match(
$response.Content,
# we use lazy match, even if it's expensive when it comes to performance.
('{0}.+?(?:{1}.+?{2}|{2}.+?{1}).+?{3}' -f
[regex]::Escape('<form'),
[regex]::Escape("action=`"https://$consentDomain"),
[regex]::Escape('method="POST"'),
[regex]::Escape('</form>')
)
)
# getting the POST URL using our parsed form data. As of now it should parse: "https://consent.youtube.com/s"
$postUrl = [regex]::Match($formContent, '(?<=action\=\")[^\"]+(?=\")').Value
# build POST body as hashtable using our parsed form data.
# only elements with a "name" attribute are relevant and we only need the plain names and values
$postBody = @{}
[regex]::Matches($formContent -replace '\r?\n', '<input[^>]+>').Value | ForEach-Object {
$name = [regex]::Match($_, '(?<=name\=\")[^\"]+(?=\")').Value
$value = [regex]::Match($_, '(?<=value\=\")[^\"]+(?=\")').Value
if (![string]::IsNullOrWhiteSpace($name)) {
$postBody[[string]$name] = [string]$value
}
}
# now let's try to get an accepted CONSENT cookie by POSTing our hashtable to the parsed URL and override the sessionVariable again.
# Using the previous session variable here would return a HTTP error 400 ("method not allowed")
$response = Invoke-WebRequest -Uri $postUrl -Method Post -UseBasicParsing -SessionVariable 'youtubeSession' -UserAgent $webUserAgent -Body $postBody -ErrorAction Stop
# get all the cookies for domain '.youtube.com'
$youtubeCookies = [object[]]$youtubeSession.Cookies.GetCookies('https://youtube.com')
# check if we got the relevant cookie "CONSENT" with a "yes+" prefix in its value
# if the value changes in future, we have to adapt the condition here accordingly
$consentCookie = [object[]]($youtubeCookies | Where-Object { $_.Name -eq 'CONSENT' })
if (!$consentCookie.Count) {
Write-Error -Message 'The cookie "CONSENT" is missing in our session after our POST! Please check.' -ErrorAction Stop
} elseif (!($consentCookie.Value -like 'YES+*').count) {
Write-Error -Message ("The value of cookie ""CONSENT"" (""$($consentCookie.Value -join '" OR "')"") does not start with ""YES+"", but maybe it's intended and the condition has to be adapted!") -ErrorAction Stop
}
}
} finally {
# set the progress preference back to the previous value
$ProgressPreference = $currentProgressPreference
}
# From here on use the argument '-WebSession $youtubeSession' with each 'Invoke-WebRequest'
# e.g.: Invoke-WebRequest $youtubeUrl -WebSession $youtubeSession -UseBasicParsing
The process is the same as above, but it includes verbose output. It just includes verbose output statements, so that you or any other can debug it more easily if something changes.
$youtubeUrl = 'https://www.youtube.com/user/YouTube/videos'
$consentDomain = 'consent.youtube.com'
$webUserAgent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.4976.0 Safari/537.36 Edg/102.0.1227.0'
# remove this verbose preference definition or set it to "SilentlyContinue" to suppress verbose output
$currentVerbosePreference = $VerbosePreference
$VerbosePreference = [System.Management.Automation.ActionPreference]::Continue
# at first we disable the annoying (at least for this process) and in PS5.1- performance affecting progress bar for web requests
$currentProgressPreference = $ProgressPreference
$ProgressPreference = [System.Management.Automation.ActionPreference]::SilentlyContinue
#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#
#region THIS REGION CAN BE REMOVED.
#------------------------------------------------------------------------------------------------------------------------
Write-Verbose "`r`n>> Let's start with a GET to:`r`n`t$youtubeUrl"
#------------------------------------------------------------------------------------------------------------------------
#endregion
#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#
try {
# in our first GET call we should get a response from consent.youtube.com.
# we save the session including all cookies in variable $youtubeSession.
$response = Invoke-WebRequest -Uri $youtubeUrl -UseBasicParsing -SessionVariable 'youtubeSession' -UserAgent $webUserAgent -ErrorAction Stop
# using BaseResponse to figure out which host has responded
if ($PSVersionTable.PSVersion.Major -gt 5) {
# PS 6+ has other properties than PS5.1 and below
$responseRequestUri = $response.BaseResponse.RequestMessage.RequestUri
} else {
$responseRequestUri = $response.BaseResponse.ResponseUri
}
#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#
#region THIS REGION CAN BE REMOVED.
#------------------------------------------------------------------------------------------------------------------------
Write-Verbose "`r`n>> We got a response from:`r`n`t$responseRequestUri"
Write-Verbose "`r`n>> Let''s check our cookies. We should see a cookie 'CONSENT' which is pending:"
Write-Verbose ($youtubeSession.Cookies.GetCookies('https://youtube.com') | Format-Table Domain, Name, Value | Out-String)
#------------------------------------------------------------------------------------------------------------------------
#endregion
#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#
# check if got redirected to consent domain
if ($responseRequestUri.Host -eq $consentDomain) {
#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#
#region THIS REGION CAN BE REMOVED.
#------------------------------------------------------------------------------------------------------------------------
Write-Verbose "`r`n>> Let's parse the required form data and post it to the correct URL"
#------------------------------------------------------------------------------------------------------------------------
#endregion
#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#
# unfortunately the response object from "Invoke-WebRequest" does not provide any "Form" data as property,
# so we have to parse it from the content. There are two <form..> nodes, but we only need the one for method "POST".
$formContent = [regex]::Match(
$response.Content,
# we use lazy match, even if it's expensive when it comes to performance.
('{0}.+?(?:{1}.+?{2}|{2}.+?{1}).+?{3}' -f
[regex]::Escape('<form'),
[regex]::Escape("action=`"https://$consentDomain"),
[regex]::Escape('method="POST"'),
[regex]::Escape('</form>')
)
)
# getting the POST URL using our parsed form data. As of now it should parse: "https://consent.youtube.com/s"
$postUrl = [regex]::Match($formContent, '(?<=action\=\")[^\"]+(?=\")').Value
# build POST body as hashtable using our parsed form data.
# only elements with a "name" attribute are relevant and we only need the plain names and values
$postBody = @{}
[regex]::Matches($formContent -replace '\r?\n', '<input[^>]+>').Value | ForEach-Object {
$name = [regex]::Match($_, '(?<=name\=\")[^\"]+(?=\")').Value
$value = [regex]::Match($_, '(?<=value\=\")[^\"]+(?=\")').Value
if (![string]::IsNullOrWhiteSpace($name)) {
$postBody[[string]$name] = [string]$value
}
}
#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#
#region THIS REGION CAN BE REMOVED.
#------------------------------------------------------------------------------------------------------------------------
Write-Verbose "`r`n>> Now let's post the following body to:`r`n`t$postUrl"
Write-Verbose ($postBody | Out-String)
#------------------------------------------------------------------------------------------------------------------------
#endregion
#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#
# now let's try to get an accepted CONSENT cookie by POSTing our hashtable to the parsed URL and override the sessionVariable again.
# Using the previous session variable here would return a HTTP error 400 ("method not allowed")
$response = Invoke-WebRequest -Uri $postUrl -Method Post -UseBasicParsing -SessionVariable 'youtubeSession' -UserAgent $webUserAgent -Body $postBody -ErrorAction Stop
# get all the cookies for domain '.youtube.com'
$youtubeCookies = [object[]]$youtubeSession.Cookies.GetCookies('https://youtube.com')
# check if we got the relevant cookie "CONSENT" with a "yes+" prefix in its value
# if the value changes in future, we have to adapt the condition here accordingly
$consentCookie = [object[]]($youtubeCookies | Where-Object { $_.Name -eq 'CONSENT' })
if (!$consentCookie.Count) {
Write-Error -Message 'The cookie "CONSENT" is missing in our session after our POST! Please check.' -ErrorAction Stop
} elseif (!($consentCookie.Value -like 'YES+*').count) {
Write-Error -Message ("The value of cookie ""CONSENT"" (""$($consentCookie.Value -join '" OR "')"") does not start with ""YES+"", but maybe it's intended and the condition has to be adapted!") -ErrorAction Stop
}
#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#
#region THIS REGION CAN BE REMOVED. Even the $responseRequestUri part. Just for Verbose output
#------------------------------------------------------------------------------------------------------------------------
# using BaseResponse to figure out which host has responded
if ($PSVersionTable.PSVersion.Major -gt 5) {
# PS 6+ has other properties than PS5.1 and below
$responseRequestUri = $response.BaseResponse.RequestMessage.RequestUri
} else {
$responseRequestUri = $response.BaseResponse.ResponseUri
}
Write-Verbose "`r`n>> This time we got a response from:`r`n`t$responseRequestUri"
#------------------------------------------------------------------------------------------------------------------------
#endregion
#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#
}
#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#↓#
#region THIS REGION CAN BE REMOVED. JUST A TEST. Always use: -WebSession $youtubeSession
#------------------------------------------------------------------------------------------------------------------------
Write-Verbose "`r`n>> Let's check our cookies again:"
Write-Verbose ($youtubeSession.Cookies.GetCookies('https://youtube.com') | Format-Table Domain, Name, Value | Out-String)
Write-Verbose "`r`n>> Let''s check a video from github using our new session variable.`r`n`tVideo: https://www.youtube.com/watch?v=w3jLJU7DT5"
$test = Invoke-WebRequest 'https://www.youtube.com/watch?v=w3jLJU7DT5E' -UseBasicParsing -WebSession $youtubeSession
Write-Verbose "`r`n>> And again, let''s check our cookies:"
Write-Verbose ($youtubeSession.Cookies.GetCookies('https://youtube.com') | Format-Table Domain, Name, Value | Out-String)
Write-Verbose "`r`n>> And our content. But please press [Enter] first."
if ($VerbosePreference -eq [System.Management.Automation.ActionPreference]::Continue) {
Pause
$test.Content
}
#------------------------------------------------------------------------------------------------------------------------
#endregion
#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#↑#
} finally {
# set the progress preference back to the previous value
$ProgressPreference = $currentProgressPreference
# set the verbose preference back to the previous value, in case it was used in ths script
# can be removed if not used
if ($currentVerbosePreference) {
$VerbosePreference = $currentVerbosePreference
}
}
# From here on use the argument '-WebSession $youtubeSession' with each 'Invoke-WebRequest'
# e.g.: Invoke-WebRequest $youtubeUrl -WebSession $youtubeSession -UseBasicParsing
After getting the correct CONSENT cookie you can use Invoke-WebRequest to GET any site using the argument -WebSession $youtubeSession
e.g.
Invoke-WebRequest 'https://www.youtube.com/watch?v=w3jLJU7DT5E' -WebSession $youtubeSession -UseBasicParsing