Fill the first missing null elements after shifting and rolling window

Question

I'm recreating a strategy made in python with pandas. I think my code works, even tho I haven't compared the values yet, because I'm getting an exception. Basically, the problem is that .Shift(20) removes the first 20 elements and .Window(12 * 60 / 15) removes 47 elements. The typical prices are 10180 by default. They become 10113 after the shifting and rolling window. I tried using .FillMissing(), but it doesn't seem to append the first null values to the series.

def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
    if not {'buy', 'sell'}.issubset(dataframe.columns):
        dataframe.loc[:, 'buy'] = 0
        dataframe.loc[:, 'sell'] = 0
    dataframe['typical'] = qtpylib.typical_price(dataframe)
    dataframe['typical_sma'] = qtpylib.sma(dataframe['typical'], window=10)
    min = dataframe['typical'].shift(20).rolling(int(12 * 60 / 15)).min()
    max = dataframe['typical'].shift(20).rolling(int(12 * 60 / 15)).max()
    dataframe['daily_mean'] = (max+min)/2
    return dataframe

My code (C#)

public override List<TradeAdvice> Prepare(List<OHLCV> candles)
{
    var result = new List<TradeAdvice>();

    var typicalPrice = candles.TypPrice().Select(e => e ?? 0).ToList();
    var typicalSma = typicalPrice.Sma(10);

    var series = typicalPrice.ToOrdinalSeries();

    var min = series.Shift(20).Window(12 * 60 / 15).Select(kvp => kvp.Value.Min()).FillMissing(); // 10113 elements / 10180 expected
    var max = series.Shift(20).Window(12 * 60 / 15).Select(kvp => kvp.Value.Max()).FillMissing(); // 10113 elements / 10180 expected

    var dailyMean = (max + min) / 2;

    var asd = dailyMean.SelectValues(e => Convert.ToDecimal(e)).Values.ToList();
    var crossedBelow = asd.CrossedBelow(typicalPrice);
    var crossedAbove = asd.CrossedAbove(typicalPrice);

    for (int i = 0; i < candles.Count; i++)
    {
        if (i < StartupCandleCount - 1)
            result.Add(TradeAdvice.WarmupData);
        else if (crossedBelow[i]) // crossBelow is 10113 elements instead of 10180...
            result.Add(TradeAdvice.Buy);
        else if (crossedAbove[i]) // crossBelow is 10113 elements instead of 10180...
            result.Add(TradeAdvice.Sell);
        else
            result.Add(TradeAdvice.NoAction);
    }

    return result;
}

public class OHLCV
{
    public DateTime Timestamp { get; set; }
    public decimal Open { get; set; }
    public decimal High { get; set; }
    public decimal Low { get; set; }
    public decimal Close { get; set; }
    public decimal Volume { get; set; }
}

Answer 1

If you have an ordinal series that you create with ToOrdinalSeries, it means that the index of the series will be automatically generated numerical value from 0 to length of your series - 1. However, this is still a real index and Deedle keeps the mapping when you use operations like Shift.

If your index was a date, say 01/01 => a, 02/01 => b, 03/01 => c, then Shift would shift the values and drop the keys that are no longer needed, i.e. you may get 02/01 => a, 03/01 => b.

It works the same with ordinal indices, so if you have 0 => a, 1 => b, 2 => c and shift the data, you will get something like 1 => a, 2 => b.

If you then want to get 0 => <default>, 1 => a, 2 => b, then you can do this using Realign which takes the new list of keys that you want to have followed by FillMissing. For example:

var ts = new[] { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }.ToOrdinalSeries();

var mins = ts.Shift(2).Window(2).Select(kvp => kvp.Value.Min());
var realigned = mins.Realign(Enumerable.Range(0, 10)).FillMissing(-1); 

ts.Print();        // Starts from key '0' 
mins.Print();      // Starts from key '3' because of Shift & Window
realigned.Print(); // Starts from key '0' with three -1 values at the start