U-SQL + Python return dataframe with unknown number of columns

Question

If my python script is pivoting and i can no predict how many columns will be outputed, can this be done with the U-SQL REDUCE statement?

e.g.

@pythonOutput  =
    REDUCE @filteredBets ON [BetDetailID]
    PRODUCE [BetDetailID] string, EventID float
    USING new Extension.Python.Reducer(pyScript:@myScript);

There could be multiple columns, so i can't hard set the names in the Produce part.

Any ideas?

Answer 1

If you have a way to produce a SqlMap<string,string> value from within Python (I am not sure if that is supported right now, you can do it with a C# reducer :)), then you could use the map for the dynamic schema part.

If it is not supported in Python, please file a feature request at http://aka.ms/adlfeedback.