Search code examples
htmlexcelpowerbipowerqueryscreen-scraping

PowerBI query caps the text to 1,024 characters


I am sort of new to PowerBI and really struggling to solve the truncation of characters. A quick count of rows shows that the maximum limit of characters of a scraped text is 1,024.

Here's my function:

    let GetData=(URL) =>

    let
        Source = Web.BrowserContents(URL),
        #"Extracted Table From Html" = Html.Table(Source, {{"Column1", ".fld"}, {"Column2", ".fld + *"}, {"Column3", "DIV:nth-child(8) .title"}, {"Column4", "BR + BR + A"}, {"Column5", "TR:nth-child(3) A"}, {"Column6", ".less *"}, {"Column7", ".more P:nth-child(1)"}, {"Column8", "TR:nth-child(7) BR + *"}, {"Column9", "TR:nth-child(3) P"}, {"Column10", "TR:nth-child(4) P"}, {"Column11", "TR:nth-child(5) P"}, {"Column12", "TR:nth-child(6) P"}, {"Column13", ".seemore"}, {"Column14", ".horipane STRONG"}, {"Column15", "TR:nth-child(4) .refanch"}, {"Column16", "TR:nth-child(5) .refanch"}, {"Column17", "TR:nth-child(6) .refanch"}}, [RowSelector=".fld"]),
        #"Changed Type" = Table.TransformColumnTypes(#"Extracted Table From Html",{{"Column1", type text}, {"Column2", type text}, {"Column3", type text}, {"Column4", type text}, {"Column5", type text}, {"Column6", type text}, {"Column7", type text}, {"Column8", type text}, {"Column9", type text}, {"Column10", type text}, {"Column11", type text}, {"Column12", type text}, {"Column13", type text}, {"Column14", type text}, {"Column15", type text}, {"Column16", type text}, {"Column17", type text}}),
        #"Removed Other Columns" = Table.SelectColumns(#"Changed Type",{"Column1", "Column2"})

    in
        #"Removed Other Columns"
    in GetData

I tried to use the solution in this link but I couldn't apply the solution, as I need more clarification on how to implement it. I am hoping to remove the 1,024 character limit by doing so. Grateful if someone could help me with questions below:

  1. The output of using this function creates more than 40 rows per URL. I am particularly trying to solve the truncation problem in the first row. Not sure if this is indicated as 'Column1' in the function code above?

  2. Seems like all of the column values are text type in the function. If I can use the solution suggested here, how and where should I write the code string to replace values to text?


Solution

  • Turns out truncation only applies to the preview of the Power Editor! Once you save and close the editor and get back to the main PowerBI window, you can create your table and then, the truncated cells appear in full! You can save the final table as .csv file and no truncation there, either. Many thanks to Ron Rosenfeld for confirming the revelation, I think I got to the same place about the same time!