Search code examples
azurepowerbidatabase-administration

DBA/ BI workflows with Azure operations


I have a fair few databases migrated across to Azure but I am finding queries are taking significantly longer. Obviously this is caused by latency between customer and data centre - but I am curious to see what others have done to accommodate. As background - the company I work for performs transformations on SQL databases and then comes Loading and visualising in PowerBI (traditional ETL).

For table manipulation and creating, I was thinking of using databricks with the database tables spread on clusters, then again I am yet to come across an SQL friendly method of manipulation using spark via notebooks.

Throwing it out to the community to see what has worked for them :-) TIA


Solution

  • @dale - it isn't really possible to compare like for like between on-premises and Azure SQL DB. A few things I would try to work out where the slow down is:

    1. In SQL Management Studio, use the Include Client Statistics option and do separate tests between your on-premises and your Azure databases to see if the latency on the connection is the issue or whether processing times are also slower.
    2. Check whether you can use REDIRECT instead of PROXY on the Azure connection to save connection time on your queries - especially if your application make frequent new connections.
    3. Are you hitting the per-database resource limit in your Elastic Pool? The full pool resources do not need to be hit before resource governance kicks in. Look in sys.dm_db_resource_stats to get an idea on DB resource usage