I Want To create a pipeline that contains a U-SQL script which combines multiple log files in Azure DataLake Store into one file, I tried to do that by creating a text file in my store and then add it in the pipeline scriptpath but I get an error and I searched for that and I found that the ADL isn't supported in the pipeline so I want to write the U-SQL script in the pipeline using script property. I tried to do that using this script but I get an error and I can't deploy it so anyone can help to do that? Here's My Pipeline script:
{
"name": "RG-GatherData",
"properties": {
"description": "description",
"activities": [
{
"name": "DataLakeAnalyticsUSqlActivityTemplate",
"type": "DataLakeAnalyticsU-SQL",
"linkedServiceName": "AzureDataLakeAnalyticsLinkedService",
"typeProperties": {
"script": "
@log = EXTRACT ["VersionID"] int,
["NodeName"] string,
["UpdateIng Area"] string,
["ActionDate"] string,
["UserName"] string,
["Code part Type"] string,
["DocCode"] string,
["Header Entity Id"] string,
["Common Entity Id"] string,
["Attribute Name"] string,
["Latest Update Value"] string,
["Previous Update Value"] string
FROM @in
USING Extractors.Csv(skipFirstNRows: 1);
OUTPUT @log
TO @out
USING Outputters.Csv();
,
"degreeOfParallelism": 3,
"priority": 100,
"parameters": {
"in": "/RowLogs/InPut/RoyalGardens/{*}.csv",
"out": "/RowLogs/OutPut/RoyalGardens/Alllog.csv"
}
},
"policy": {
"concurrency": 1,
"executionPriorityOrder": "OldestFirst",
"retry": 3,
"timeout": "10:00:00"
},
"scheduler": {
"frequency": "Day",
"interval": 1
}
}
],
"start": "2018-09-20T00:06:00Z",
"end": "2099-12-30T22:00:00Z"
}
}
Store the U-SQL script in Blob Storage and reference it via a Blob Storage linked service.