azureazure-data-factoryexists
how to resolve "Field 'exists' failed with error in ADF" error?
There is a pipeline to read files from FTP and i have used 'Get Metadata' Activity to achieve this. I have variable of type array that holds all the expected files. I have looped through this variable and using a if condition, I am checking if the file exists. Sometimes, the code runs without any issue but sometimes it throws the below error.
Field 'exists' failed with error: 'Type=System.Net.WebException,Message=System error.,Source=System,''Type=System.Net.InternalException,Message=System error.,Source=System,'.
JSON:
{
"name": "test1",
"properties": {
"activities": [
{
"name": "File availability summary",
"description": "Send File not found alert mail",
"type": "WebActivity",
"dependsOn": [
{
"activity": "avlFiles_final",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": "<Logic app url>",
"method": "POST",
"body": {
"value": "@variables('avlfiles_final')",
"type": "Expression"
}
}
},
{
"name": "ForEach Expected Files",
"type": "ForEach",
"dependsOn": [
{
"activity": "Intialize cntr to 1 for SNo",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "@variables('expectedFiles')",
"type": "Expression"
},
"activities": [
{
"name": "Metadata of FTP",
"description": "get metadata of folder in FTP",
"type": "GetMetadata",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "PartnerHub_FTP2_CopyActivity_prr_filename",
"type": "DatasetReference",
"parameters": {
"filename": {
"value": "@item()",
"type": "Expression"
}
}
},
"fieldList": [
"exists"
],
"storeSettings": {
"type": "FtpReadSettings",
"recursive": true,
"enablePartitionDiscovery": false,
"useBinaryTransfer": true,
"disableChunking": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
}
},
{
"name": "Check if file exists",
"description": "if exists, append to variable 'avlFiles' as available along with file name, otherwise 'Unavailable'",
"type": "IfCondition",
"dependsOn": [
{
"activity": "Metadata of FTP",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"expression": {
"value": "@activity('Metadata of FTP').output.exists",
"type": "Expression"
},
"ifFalseActivities": [
{
"name": "Append avlFiles with Unavailable files",
"type": "AppendVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "avlFiles",
"value": {
"value": "@json(concat('{','\"S.NO\":',variables('cntr'),',\"File Name\":\"',item(),'\",\"File Availability in FTP\":\"Unavailable\"}'))",
"type": "Expression"
}
}
}
],
"ifTrueActivities": [
{
"name": "Append avlFiles with available files",
"type": "AppendVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "avlFiles",
"value": {
"value": "@json(concat('{','\"S.NO\":',variables('cntr'),',\"File Name\":\"',item(),'\",\"File Availability in FTP\":\"Available\"}'))",
"type": "Expression"
}
}
}
]
}
},
{
"name": "increment and store in temp",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Check if file exists",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"variableName": "tmp",
"value": {
"value": "@string(add(int(variables('cntr')),1))",
"type": "Expression"
}
}
},
{
"name": "store temp to counter",
"type": "SetVariable",
"dependsOn": [
{
"activity": "increment and store in temp",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"variableName": "cntr",
"value": {
"value": "@variables('tmp')",
"type": "Expression"
}
}
}
]
}
},
{
"name": "avlFiles_final",
"description": "",
"type": "SetVariable",
"dependsOn": [
{
"activity": "ForEach Expected Files",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"variableName": "avlfiles_final",
"value": {
"value": "@{variables('avlFiles')}",
"type": "Expression"
}
}
},
{
"name": "Initialize cntr to 1 for SNo",
"type": "SetVariable",
"dependsOn": [],
"policy": {
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"variableName": "cntr",
"value": "1"
}
}
],
"variables": {
"expectedFiles": {
"type": "Array",
"defaultValue": [
"@{concat('FILE1.', '20230713', '.1.csv')}",
"@{concat('FILE2.', '20230713', '.1.csv')}",
"@{concat('FILE3.', '20230713', '.1.csv')}",
"@{concat('FILE4.', '20230713', '.1.csv')}"
]
},
"avlFiles": {
"type": "Array"
},
"avlfiles_final": {
"type": "String"
},
"cntr": {
"type": "String"
},
"tmp": {
"type": "String"
}
},
"annotations": [],
"lastPublishTime": "2023-07-25T19:45:49Z"
},
"type": "Microsoft.DataFactory/factories/pipelines"
}
Can someone tell me why this happens?
Solution
I tried the same JSON in my environment, and it is working fine for me.
As per your requirement, you want to generate serial number along with other keys. To get the incrementing serial number, make sure you checked the Sequential in the Foreach.
And coming to your error at Get Meta data activity, try the below trouble shooting steps to resolve it.
- Check the connection of linked service and try the same pipeline JSON in another pipeline after publishing.
- Check whether it results the same for the other Get meta data activity properties like
size,structure(Pass only the existing file names for these while checking).
It the error still persists, then it's better to raise a Support ticket for deeper investigation on this issue.
- What is a difference between elastic computing and serverless computing?
- Access Sharepoint Online Files from .NET Worker Service via Microsoft Graph/OAuth - Unauthorized or Access Denied error (401)
- How do I fix SerializationNotSupportedParentType and ThrowNotSupportedException_ConstructorContainsNullParameterNames exception in System.Text.Json?
- Update VCore by database in power shell
- While scanning a simple key, could not find expected ':'. Syntax error in azure-pipeline.yaml
- I'm not able to call the Azure document intelligence api using the latest api version "2024-02-29-preview"
- AzureWebJobsStorage Managed Identity not working
- Get-RemoteDomain in powershell exchange online error : The term 'Get-RemoteDomain' is not recognized as the name of a cmdlet
- How can I Get Started Using Certificates to Authenticate my APIs Using Azure Key Vault?
- Azure Storage blob getting the io.netty.channel.StacklessClosedChannelException while generating /downloading the from SasUrl
- How to set up a site-to-site connection between Azure VNet and an on-premises network with a single IP address for traffic selectors?
- Summarization of docx or pdf document
- Unable to use Azure Developer CLI (azd) in vsCode even after extensions installed
- How to configure backend application on Container Apps
- WebJob is stopping due to website shutting down
- Azure SQL trigger for Functions configuration problem
- Does azure blob storage support http2?
- Azure Pronunciation Assessment Could not deserialize speech context error
- Is there a query to run so I can get a list of users who has NOT logged in, in the past 30 days?
- How do I Parse FormData containing a audio file in AZ function app in 2025
- Not able to delete Public IP address in azure
- Azure Devops Apple App Store Release.Missing value for the attribute 'whatsNew'
- Azure Storage accounts container public access level has dissabled
- Issue with adding delegated Graph API permission to Enterprise app with Terraform
- Getting an error when using the Flexible File Destination Task in Visual Studio 2022 SSIS
- Problem with Azure Email Communication Service Custom Domain
- How to refresh client assertion in ConfidentialClientApplication?
- Using Microsoft Graph Explorer (we have code too) we can GET All Subscriptions, but GET one by ID gives access error
- Invalid operands found for operator -eq
- nginx - php-fpm - azure container app returns truncated pages