Copy Data from Azure Data Lake to SnowFlake without stage using Azure Data Factory
All the Azure Data Factory examples of copying data from Azure Data Lake Gen 2 to SnowFlake use a storage account as stage. If the stage is not configured (as shown in picture), I get this error in Data Factory even when my source is a csv file in Azure data lake - "Direct copying data to Snowflake is only supported when source dataset is DelimitedText, Parquet, JSON with Azure Blob Storage or Amazon S3 linked service, for other dataset or linked service, please enable staging".
At the same time, SnowFlake documentation says the the external stage is optional. How can I copy data from Azure Data Lake to SnowFlake using Data Factory's Copy Data Activity without having an external storage account as stage? If staging storage is needed to make it work, we shouldn't say that data copy from Data Lake to SnowFlake is supported. It works only when, Data Lake data is is first copied in a storage blob and then to SnowFlake.
Though Snowflake supports blob storage, Data Lake storage Gen2, General purpose v1 & v2 storages, loading data into snowflake is supported- through blob storage only.
The source linked service is Azure Blob storage with shared access signature authentication. If you want to directly copy data from Azure Data Lake Storage Gen2 in the following supported format, you can create an Azure Blob linked service with SAS authentication against your ADLS Gen2 account, to avoid using staged copy to Snowflake.
Select Azure blob storage in linked service, provide SAS URI details of Azure data lake gen2 source file.
Blob storage linked service with data lake gen2 file:
- Azure Service Plan - Convert Consumption App to Premium App
- No PK or FK when exporting SQL Server database
- Assigning a Token Lifetime Policy to an application in Microsoft EntraID seems to have no effect
- How to create a dataset for Azure custom speech using spx (speechCLI)
- Azure App Service Autoscale Fails to Scale In
- Azure Scale Out on Memory keeps flapping (not scaling in)
- Setting a server minimum for Azure Web App when using automatic scaling, from Bicep
- A keyvault created with access policy using BICEP creates compound identity
- Azure Loadbalancer with public IP forward traffic to Container Apps
- How to grant a Managed Identity permissions to an Azure SQL Database using IaC?
- What means skipNegotiation in SignalR HubConnection?
- Azure ClientCertificateCredential - Using SNI
- Azure AI Search MultiIndex / Conditional Semantic search
- MSAL Node service & The Partner Center API
- Load Registered Component in Azure ML for Pipeline using Python sdk v2
- Azure Blob:- How to automate Azure Archive Storage to Cool/Hot tier conversion, send download link once it's avaible , and re-archive after 72 hours?
- Azure App Service Custom Backup to Firewall Enabled Storage Account not working as expected
- How to take max value and replace it in drived column in data factory
- Synapse/ ADF - How to Truncate table if dynamic config column is True in pre-copy script
- Azure Cost Management - track costs associated with Databricks job
- Error: "OrganizationFromTenantGuidNotFound" (even with Microsoft 365 subscription)
- How to check Debug.Writeline() output in VS code?
- C# API - Provide download of files from Azure Blobstorage
- localhost:300/api is not working : "This page isn’t working"
- Creating an Azure Linux VM with Ubuntu 20.04 with Terraform
- Failed to deploy path that does not exist
- Using Azure WAF for my server(not in Azure)
- How and where to define an environment variable on Azure
- Azure Data Factory - Unable to preview data
- Azure yaml trigger pipeline every 14 days on a Saturday