Which Spark version is being used by Mapping Data Flows?
I ran into some issues recently which lead me to believe that the Mapping Data Flows used in my Synapse Analytics Workspace are still running Spark 2.4, which has reached EOL. I tried to find out which Spark version is being used and how to change it, without any success.
Is there a way to migrate to a newer version or to at least verify which Spark version is being used under the covers?
The concrete reasons for me caring revolve around the breaking changes regarding date/time handling in Spark 3.x.
You could figure out the version of spark through Monitoring output. Migration to Spark 3.x is already underway. And your subscription should be migrated to 3.x soon (if its not already migrated)
You can find the information in the output details of a data flow activity.
- WebJob is stopping due to website shutting down
- Is it possible to change the day returned from startofweek() and endofweek()?
- Handling Audio blobs sent through mediarecorder in JS through python server script for azure service
- Export app registrations with expiring secrets and certificates and send alert in teams
- SQL Azure import slow, hangs, but local import takes 3 minutes
- Issue with Group email address domain when create a new Microsoft 365 Group
- What is the URL for New Page in Azure DevOps Wiki? / How to bookmark New Page?
- Is there a dynamic way to get variables from Variable Group? Even secret ones? and insert in Azure Key Vault
- How do I configure my Bicep scripts to allow a container app to pull an image from an ACR using managed identity across subscriptions
- Azure DevOps. Can't trigger a pipeline in Repo A when a commit is done in repo B
- How to change the default Azure AI Search scoring to be increased when the whole search term is found exactly?
- List and download files in a folder in a blob with blob level SAS token
- How to change the Lease state of Azure Blob to Available
- unable to delete or move first step from logic app?
- Azure Terraform - Encrypt VM OS Disk
- How to temporary stop Azure API Management Service
- Unable to retrieve key vault certificates in azure runbook
- C#: 'A change set cannot include changes to more than '1' source resources' when adding a user To Group using Azure Graph API 2.0
- Set size limt to direct upload to azure blob storage via SAS URL
- Using AddAzureKeyVault makes my application 10 seconds slower
- PostGreSQL pg_dump psql and pg_restore errors with Azure Flexible SQL Server
- How to check signature of a token coming from Azure SSO
- How can I get error details from an if condition activity in Azure Data Factory?
- Getting a 404 for GET /robots933456.txt
- Given an Applications Insight Instrumentation key, get the name of the service in Azure
- How to specify rewrite url for query string parameters in Azure API Management
- Can a normal Azure Function and a Durable Azure Function coexist in the same Function App?
- TenantGuidNotFound when trying to send email using Microsfot GraphAPI
- Can't signin into microsoft account
- Azure Function App Kudu Functions API with empty response