Search code examples
azuredata-warehouseazure-data-lakepolybase

Benchmark test for polybase with azure data lake


Have anyone performed benchmark test using polybase with adl, I want to know if I am having a data file which is having 4milion rows, will polybase be helpful in fetching those rows to the data warehouse. Can anyone post any articles where I Can learn about these things.


Solution

  • Yes Microsoft have conducted some trials, for example:

    Load 1 TB into Azure SQL Data Warehouse under 15 minutes with Data Factory https://learn.microsoft.com/en-us/azure/data-factory/data-factory-load-sql-data-warehouse

    This is using Data Factory but it's really Polybase under the hood doing the heavy lifting. Now, it was using Polybase with Blob Storage (not Data Lake) but you get the idea. As an experiment, why don't you set this up, run it, then convert it to use Data Lake and report back?