I had a script that can parse html table to powershell table. Script broke after specific website update, Now when i'm trying to parse the table I get false information (Same when I open the website from internet explorer 11). Here's the code:
Get-HtmlTable
Function Get-HTMLTables{
<#
.SYNOPSIS Gets all the text in HTML tables on a webpage and returns it as an object of arrays
.DESCRIPTION
This function treats
1. table cells as properties of row objects
2. Rows as custom objects with cells as properties
3. Tables as arrays of rows
4. Pages as custom objects with numbered tables as properties.
Returns
-Custom object with numbered properties
-Each numbered property represents a table number from the top of the page down
-Each numbered property contains an array of row custom objects
-Each object in the array has properties representing each cell in the row.
.PARAMETER URL
The URL to look for HTML tables in
.PARAMETER FirstRowHeader
Whether or not to treat the first row of the table as a header
.EXAMPLE
$objPage = Get-HTMLTables -URL 'https://support.microsoft.com/en-us/lifecycle/search?sort=PN&alpha=windows%2010&Filter=FilterNO' -FirstRowHeader $true
.EXAMPLE
$objPage.21 | select -first 1
Extended Support End Date : 10/14/2025
Lifecycle Start Date : 7/29/2015
Products Released : Visual Studio Express 2015 for Windows 10
Mainstream Support End Date : 10/13/2020
Service Pack Support End Date :
Notes :
.EXAMPLE
$objPage.21 | Where{$_.('Products Released') -match 'Education'}
Extended Support End Date : 10/14/2025
Lifecycle Start Date : 7/29/2015
Products Released : Windows 10 Education, released in July 2015
Mainstream Support End Date : 10/13/2020
Service Pack Support End Date :
Notes : • Updates are cumulative, with each update built upon all of the updates that preceded it. A device needs to
install the latest update to remain supported.
• Updates may include new features, fixes (security and/or non-security), or a combination of both. Not all
features in an update will work on all devices.
• A device may not be able to receive updates if the device hardware is incompatible, lacking current
drivers, or otherwise outside of the Original Equipment Manufacturer’s (“OEM”) support period.
• Update availability may vary, for example by country, region, network connectivity, mobile operator (e.g.,
for cellular-capable devices), or hardware capabilities (including, e.g., free disk space).
#>
Param(
[uri]$URL = 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads',
[boolean]$firstRowHeader = $false
)
$AllProtocols = [System.Net.SecurityProtocolType]'Ssl3,Tls,Tls11,Tls12'
[System.Net.ServicePointManager]::SecurityProtocol = $AllProtocols
#Useragent
$userAgent = [Microsoft.PowerShell.Commands.PSUserAgent]::Safari
#Get the webpage
$page = Invoke-WebRequest $URL -UserAgent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36'
#Filter out only the tables
$tables = $page.ParsedHtml.body.getElementsByTagName('Table')
#Get only the tables that have cells
$tableswithcells = $tables | Where{$_.cells}
$hashPage = @{}
$tablecount = 0
#ForEach table
ForEach($table in $tableswithcells){
$arrTable = @()
$rownum = 0
$arrTableHeader = @()
#Get all the rows in the tables
ForEach($row in $table.rows){
#Treat the first row as a header
if($rownum -eq 0 -and $firstRowHeader){
ForEach($cell in $row.cells){
$arrTableHeader += $cell.InnerText.Trim()
}
#If not the first row, but using headers, store the value by header name
}elseIf($firstRowHeader){
$cellnum = 0
$hashRow = @{}
ForEach($cell in $row.cells){
$strHeader = $arrTableHeader[$cellNum]
If($strHeader){
$hashRow.Add($strHeader,(($cell.innerhtml -replace "(?s)^.*$href=`"", '') -replace "`">Download</a>", ""))
#$hashRow.Add($strHeader,$cell.innertext)
}else{
#If the header is null store it by cell number instead
$hashRow.Add($cellnum,$cell.innertext)
}
$cellnum++
}
#Save the row as a custom ps object
$objRow = New-object -TypeName PSCustomObject -Property $hashRow
$arrTable += $objRow
#if not the first row and not using headers, store the value by cell index
}else{
$cellnum = 0
$hashRow = @{}
ForEach($cell in $row.cells){
$hashRow.Add($cellnum,$cell.innertext)
$cellnum++
}
#Store the row as a custom object
$objRow = New-object -TypeName PSCustomObject -Property $hashRow
#Add the row to the array of rows
$arrTable += $objRow
}
$rownum++
}
#Add the tables to the hashtable of tables
$hashPage.Add($tablecount,$arrTable)
$tablecount++
}
$objPage = New-object -TypeName PSCustomObject -Property $hashPage
Return $objPage
}
When I execute the following lines:
$test = Get-HTMLTables -firstRowHeader $false -url 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads'
$test.0
And these are the results:
5 : Windows x86-32
4 : Windows x86-64
3 : Mac OS X
2 : Linux x86-32
1 : Linux x86-64
0 : PostgreSQL Version
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 14.1
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 13.5
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 12.9
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 11.14
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 10.19
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.6.24*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.5.25*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.4.26*
5 : Not supported
4 : Not supported
3 : postgresql.org
2 : postgresql.org
1 : postgresql.org
0 : 9.3.25*
The website I'm trying to scrape: https://www.enterprisedb.com/downloads/postgres-postgresql-downloads as you can see there are available versions for Win64.
Also same happens when opening this website with Internet Explorer 11, Tried to change user agents but issue still exist.
As commented by @Mathias: the required (Json) data is in the (__NEXT_DATA__) script.
To pull that information you need to do something like this:
$Page = Invoke-WebRequest 'https://www.enterprisedb.com/downloads/postgres-postgresql-downloads'
$Unicode = [System.Text.Encoding]::Unicode.GetBytes($Page.Content)
$Document = New-Object -Com 'HTMLFile'
if ($Document.IHTMLDocument2_Write) { $Document.IHTMLDocument2_Write($Unicode) } else { $Document.write($Unicode) }
$Document.Close()
$Data = $Document.getElementById('__NEXT_DATA__').innerHTML |ConvertFrom-Json
$Data |ConvertTo-Json -Depth 10
Yields:
{
"props": {
"pageProps": {
"postgreSQLDownloads": [
{
"version": "14.1",
"products": [
{
"title": "PostgreSQL Download",
"field_installer_version": "14",
"field_sub_version": "1",
"field_os": "Mac OS X",
"field_installation_method": "Interactive Installer",
"field_supported_os": "MacOSX 10.12+",
"uuid": "f027c016-7c5b-43fd-beb7-59ee43135607",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001NhszQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "14",
"field_sub_version": "1",
"field_os": "Windows x86-64",
"field_installation_method": "Interactive Installer",
"field_supported_os": "Windows Server 2016, Windows Server 2019",
"uuid": "db55e32d-e9f0-4d7c-9aef-b17d01210704",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001NhszQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
}
]
},
{
"version": "13.5",
"products": [
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Linux x86-64",
"field_installation_method": "RPM",
"field_supported_os": "CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12",
"uuid": "514cbe6a-089c-4cb9-a6b1-ad9d6375842f",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Linux x86-64",
"field_installation_method": "DEB",
"field_supported_os": "Debian 10, Debian 9.6.x, Ubuntu 18.04, Ubuntu 20.04",
"uuid": "e3c7350d-e18c-4eae-8817-c61cb83e09b9",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Mac OS X",
"field_installation_method": "Interactive Installer",
"field_supported_os": "MacOSX 10.12+",
"uuid": "76003af2-5f27-4b85-9804-c3b2237debf5",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
},
{
"title": "PostgreSQL Download",
"field_installer_version": "13",
"field_sub_version": "5",
"field_os": "Windows x86-64",
"field_installation_method": "Interactive Installer",
"field_supported_os": "Windows Server 2012 R2, Windows Server 2016, Windows Server 2019",
"uuid": "7c756686-90b4-4909-89ed-043e0705a76e",
"field_show_this_version": "1",
"field_campaign_id": "7012J000001BfmaQAC",
"field_product_category": "PostgreSQL",
"field_asset_details": ""
}
]
}
// ...
]
},
"__N_SSP": true
},
"page": "/downloads/postgres-postgresql-downloads",
"query": {},
"buildId": "u3IIRJVKDxUWWWroinlsJ",
"runtimeConfig": {},
"isFallback": false,
"gssp": true,
"scriptLoader": []
}
And
$Data.props.pageProps.postgreSQLDownloads.products |Format-Table
Yields:
title field_installer_version field_sub_version field_os field_installation_method field_supported_os uuid field_show_this_version field_campaign_id field_product_category
----- ----------------------- ----------------- -------- ------------------------- ------------------ ---- ----------------------- ----------------- ----------------------
PostgreSQL Download 14 1 Mac OS X Interactive Installer MacOSX 10.12+ f027c016-7c5b-43fd-beb7-59ee43135607 1 7012J000001NhszQAC PostgreSQL
PostgreSQL Download 14 1 Windows x86-64 Interactive Installer Windows Server 2016, Windows Server 2019 db55e32d-e9f0-4d7c-9aef-b17d01210704 1 7012J000001NhszQAC PostgreSQL
PostgreSQL Download 13 5 Linux x86-64 RPM CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12 514cbe6a-089c-4cb9-a6b1-ad9d6375842f 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Linux x86-64 DEB Debian 10, Debian 9.6.x, Ubuntu 18.04, Ubuntu 20.04 e3c7350d-e18c-4eae-8817-c61cb83e09b9 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Mac OS X Interactive Installer MacOSX 10.12+ 76003af2-5f27-4b85-9804-c3b2237debf5 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 13 5 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 7c756686-90b4-4909-89ed-043e0705a76e 1 7012J000001BfmaQAC PostgreSQL
PostgreSQL Download 12 9 Linux x86-64 RPM CentOS 8 / 7, OL 8 / 7, RHEL 8 / 7, SLES 12 9a8d19d8-980c-4475-af6c-31aa964d2a30 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Linux x86-64 DEB Debian 9.6.x, Ubuntu 18.04 0753b404-99a7-4c0d-867a-8e9534a566c7 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Mac OS X Interactive Installer MacOSX 10.12+ 942ce37d-8817-42be-8466-51f653728eda 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 12 9 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 03b13357-9281-4985-baea-17b72a0febc3 1 7012J000001F5IIQA0 PostgreSQL
PostgreSQL Download 11 14 Linux x86-64 RPM ababa720-d9ac-46fa-88a0-161c27bdc4cf 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Linux x86-64 & 32 DEB befdfd14-abe0-4614-b22b-0b4a577dfd7f 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Mac OS X Interactive Installer MacOSX 10.12+ dcf30630-6453-4c50-9de1-add7a8bf431c 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 11 14 Windows x86-64 Interactive Installer Windows Server 2012 R2, Windows Server 2016, Windows Server 2019 995d0688-d8cf-48b8-8afe-13c777b0ef8e 1 70138000001U9DfAAK PostgreSQL
PostgreSQL Download 10 19 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 f6607ede-6698-4116-bf56-bd25284203aa 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 Interactive Installer CentOS 7, Debian 8 / 7, OL 7, RHEL 7, Ubuntu 14.04 beb6b65f-6aeb-466f-8b9f-c27a7e327ccf 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 RPM 03f51b70-24ee-4c75-88a7-de2e488c59da 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Linux x86-64 & 32 DEB a92ccaab-7c3f-4a5b-9a5e-e632b79b33d7 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Mac OS X Interactive Installer MacOSX 10.12+ c1b50eb7-9ad6-4e6c-a094-9a7f70ee86f9 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Windows x86-32 Interactive Installer Windows 7 / 8 / 10 e9a60157-955a-4f24-a738-07cff51ccb43 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 10 19 Windows x86-64 Windows 2012 R2 / R1, Windows 7 / 8 / 10, Windows Server 2016 ea5c8104-3940-4ed1-b427-81cf19781581 1 70138000000rYFmAAM PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 a9b88ba3-839b-4883-b2c5-378d4a7ee3da 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 Interactive Installer CentOS 7, Debian 8 / 7, OL 7, RHEL 7, SLES 12, Ubuntu 14.04 c539f198-0912-46a9-b4e6-545166574893 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 RPM ceb7a789-4a0c-4c17-8d48-abb1cdfd9cff 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Linux x86-64 & 32 DEB b976efbe-41d1-4d3f-9b14-95b7b03e651e 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Mac OS X Interactive Installer MacOSX 10.12+ b40e8d78-4a3b-458a-85f5-f0b7e0bbfa31 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Windows x86-32 Interactive Installer Windows 2008 R1, Windows 7 / 8 / 10 5011320c-34df-4c8f-9b6f-21e53c4a4930 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 6.24 Windows x86-64 Interactive Installer Windows 2008 R2, Windows 2012 R2 / R1, Windows 7 / 8 / 10 91f8264f-8a2d-434a-9559-4dfa9b3b80b1 0 701380000017oAXAAY PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-32 Interactive Installer SLES 12, Ubuntu 14.04 25ec7179-be39-4e0f-a429-bc140ccfab00 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 Interactive Installer Debian 7, SLES 12, Ubuntu 14.04 147a6582-849c-44d1-8a51-e90fcece30a6 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 RPM 5310449a-13b3-4ca1-922c-f0baef05c3ab 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Linux x86-64 & 32 DEB 654437d3-3c63-449c-bd22-1deda40dad16 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Mac OS X Interactive Installer MacOSX 10.10, MacOSX 10.8, MacOSX 10.9 3a9e2080-a11f-4230-8974-78c2ef7234d9 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Windows x86-32 Interactive Installer Windows 2008 R1, Windows 7 / 8 / 10 aa5aa60c-2030-4e06-a6ba-06ba0eb79383 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 5.25 Windows x86-64 Interactive Installer Windows 2008 R2, Windows 2012 R2 / R1 7869632c-134f-4284-8fba-feb67010b372 0 70138000000fFsrAAE PostgreSQL
PostgreSQL Download 9 4.26 Linux x86-32 Interactive Installer ed2f791f-ca94-4872-ac52-d3221e1c1950 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Linux x86-64 Interactive Installer 5273bfc5-9382-4a55-ab9a-d42c86296dac 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Mac OS X Interactive Installer 58b175c3-0c86-425a-a3fa-5bbdb10abf08 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Windows x86-32 Interactive Installer 33e923ed-0d9e-40b1-8972-96c245d239e4 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 4.26 Windows x86-64 Interactive Installer cba59404-da4d-4009-8edb-6655b72464dd 0 70150000000vxMKAAY PostgreSQL
PostgreSQL Download 9 3.25 Linux x86-32 Interactive Installer 0e139bf2-f0da-4ba9-bb40-9c0cadb4087e 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Linux x86-64 Interactive Installer 566673e2-7d5e-46aa-82dc-7400d296400e 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Mac OS X Interactive Installer 72c6d5f4-68bb-4080-a8cf-769f5fc60f41 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Windows x86-32 Interactive Installer 46c086f2-944c-49a2-bac3-204ccf14ac17 0 70150000000rxXPAAY PostgreSQL
PostgreSQL Download 9 3.25 Windows x86-64 Interactive Installer dbb5a7ac-4ecb-4a11-be2b-d56e8908b2a4 0 70150000000rxXPAAY PostgreSQL